Toolshed reset all repository metadata
We've noticed a funny about resetting repository metadata from the toolshed web interface. My understanding is that this is used to get the toolshed back in sync with the underlying mercurial repo, for example after pushing updates from hg on the command line. It seems that only the person who created the repository can reset metadata. I granted authority to make changes to a colleague, which enables him to push updates, but he doesn't see the reset metadata entry in the menu. This seems like a bug to me. Or is it by design? Also, is there a reason why this function is necessary at all? Wouldn't it be better if the toolshed just noticed updates to the underlying mercurial repo, and reset its own metadata automatically? cheers, Simon
Hello Simon, On Oct 23, 2013, at 5:01 PM, Simon Guest <simon.guest@agresearch.co.nz> wrote:
We've noticed a funny about resetting repository metadata from the toolshed web interface.
My understanding is that this is used to get the toolshed back in sync with the underlying mercurial repo, for example after pushing updates from hg on the command line.
This is not quite correct. When committing a new changeset to a repository (either via hg push from the command line or by using any feature in the Tool Shed's upload component), metadata is reset on the repository automatically, but only back to the previous installable changeset revision if one exists. So for those repositories that have multiple changeset revisions, not all of the repository changelog is inspected with a new commit. The "Reset all repository metadata" feature, however, does inspect the complete repository changelog and resets metadata from revision 0 through the repository tip. In addition to being useful for the current repository itself, it can also be useful for all other repositories that have a dependency relationship to it. See http://wiki.galaxyproject.org/RepositoryRevisions#Resetting_metadata_on_your.... In general, it is not necessary to use this feature.
It seems that only the person who created the repository can reset metadata. I granted authority to make changes to a colleague, which enables him to push updates, but he doesn't see the reset metadata entry in the menu. This seems like a bug to me. Or is it by design?
This was the original desing, but a more recent oversight as this feature has evolved. This has been corrected in 11114:71c35dbde130, which i committed to the next-stable branch and is now running on both the test and main Galaxy tol sheds.
Also, is there a reason why this function is necessary at all?
Yes, but it is not generally necessary for it to be used. It used to be restricted to only a Tool Shed admin, but recently it was exposed to repository owners (and noew anyone tha thas write access) because it became very useful for developers like Bjoern Gruening to be able to reset metadata on repositories (especially in the test tool shed) as new repository and tools dependency relationship features were being introduced at the same time as repository types (TOOL_DEPENDENCY_DEFINITION) and he was helping to flesh out the support for the new features.
Wouldn't it be better if the toolshed just noticed updates to the underlying mercurial repo, and reset its own metadata automatically?
It does, but hopefully my explanations above, as well as the details in the referenced tool shed wiki page will clarify this feature for you.
cheers, Simon ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
At Thu, 24 Oct 2013 10:03:14 -0400, Greg Von Kuster wrote:
Hello Simon,
On Oct 23, 2013, at 5:01 PM, Simon Guest <simon.guest@agresearch.co.nz> wrote:
We've noticed a funny about resetting repository metadata from the toolshed web interface.
My understanding is that this is used to get the toolshed back in sync with the underlying mercurial repo, for example after pushing updates from hg on the command line.
This is not quite correct. When committing a new changeset to a repository (either via hg push from the command line or by using any feature in the Tool Shed's upload component), metadata is reset on the repository automatically, but only back to the previous installable changeset revision if one exists. So for those repositories that have multiple changeset revisions, not all of the repository changelog is inspected with a new commit.
The "Reset all repository metadata" feature, however, does inspect the complete repository changelog and resets metadata from revision 0 through the repository tip. In addition to being useful for the current repository itself, it can also be useful for all other repositories that have a dependency relationship to it. See http://wiki.galaxyproject.org/RepositoryRevisions#Resetting_metadata_on_your....
In general, it is not necessary to use this feature.
Hi Greg, Thanks for the explanation, and the link to that helpful Wiki page. And also that code update 11114:71c35dbde130 which will help our tool test/development cycle. I have a remaining question, though, about the recommended working practices for developing a toolshed tool. I didn't find this described on the Wiki. There is obviously a cycle of development to go through, making updates, installing to a test galaxy instance, etc. It seems to us that the only sensible way to handle this is within a local test toolshed, essentially using the repo in the test toolshed as a development repo (despite the warnings not use the toolshed as a source code repo on the Wiki!). During such an iterative process, we don't want to keep bumping the tool version. Therefore it seems necessary to reset the repository meta data after every push, in order to install the just-updated-for-testing tool. Once the tool is working, the files can be uploaded into a clean repo on a production toolshed, so we don't keep all the intermediate development versions. Or is there a better way to do this? cheers, Simon
Hello Simon, On Oct 24, 2013, at 4:06 PM, "Guest, Simon" <Simon.Guest@agresearch.co.nz> wrote:
Hi Greg,
Thanks for the explanation, and the link to that helpful Wiki page. And also that code update 11114:71c35dbde130 which will help our tool test/development cycle.
I have a remaining question, though, about the recommended working practices for developing a toolshed tool. I didn't find this described on the Wiki. There is obviously a cycle of development to go through, making updates, installing to a test galaxy instance, etc. It seems to us that the only sensible way to handle this is within a local test toolshed, essentially using the repo in the test toolshed as a development repo (despite the warnings not use the toolshed as a source code repo on the Wiki!).
This is certainly a good approach, although using a local tool shed as a component of the devlopment process when developing tools is probably useful only when your repository includes repository dependency definitions or tool dependency definitions. In these cases, the tool shed encironment becomes useful for working out the details of the dependencies and making sure that everything ( all repository and tool dependencies ) installs correctly into your local Galaxy instance. In cases where you are developing simpler tools ( e.g., Text Manipulation tools like filter and sort ) that are completely self contained ( they have no dependencies ), using a local tool shed during the development process is probably not necessary.
During such an iterative process, we don't want to keep bumping the tool version.
It may ot be necessary to do so - the only time you should be bumping the tool version ( which is a manual process of changing the version string in the tool config ) is when the tool behavior changes such that the same input dataset produces defferent resulting output datasets(s).
Therefore it seems necessary to reset the repository meta data after every push, in order to install the just-updated-for-testing tool.
If you are using mercurial ersion 2.2.3 or later in your shell environment, then pushing to your local tool shed from the command line should be resetting metadata back to the last installable changeset revsion. If you are not seeing this behavior, make sure you are using that version or later - see http://wiki.galaxyproject.org/ToolShedRepositoryFeatures#Pushing_changes_to_...
Once the tool is working, the files can be uploaded into a clean repo on a production toolshed, so we don't keep all the intermediate development versions.
Yes, this is definitely the correct approach if you do find that using a local tool shed during the development process is beneficial.
Or is there a better way to do this?
I think you have things pretty well figured out. Don't hesitate to continue to ask questions as they arise.
cheers, Simon ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Greg Von Kuster -
Guest, Simon -
Simon Guest