Re: [galaxy-dev] Automatic installation of third party dependancies
Hi Lance, What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions: 1:14e18dc9ed13 2:f5d08224af89 4:14bec14f4290 Since you could not have installed 3:f7a5b54a8d4f, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates? I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem Greg Von Kuster On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks.
URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '4'
Lance Parsons wrote:
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc.
Glad I can be of help.
Lance
Greg Von Kuster wrote:
Hello Lance,
See my inline comments.
On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote:
I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into.
1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separate<action type="set_environment"> tags, then the second overwrites the first in the env.sh file. If instead, I put two<environment_variable> tags within the action, only one (the second) gets executed during installation.
The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now.
I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new "make_directory" tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version.
2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed).
Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation.
Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable.
Thanks for your help on this,
Greg Von Kuster
Any thoughts on how to address these issues would be appreciated.
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it. I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?) 4) I attempted to get updates and ran into the issue I described. I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool. Let me know if there is anything I can do to help sort this out. Lance Greg Von Kuster wrote:
Hi Lance,
What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions:
1:14e18dc9ed13 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13&id=2df7e24ce6c1f224> 2:f5d08224af89 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89&id=2df7e24ce6c1f224> 4:14bec14f4290 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290&id=2df7e24ce6c1f224>
Since you could not have installed 3:f7a5b54a8d4f <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4f&id=2df7e24ce6c1f224>, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates?
I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem
Greg Von Kuster
On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks.
URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 <http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4> File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '4'
Lance Parsons wrote:
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc.
Glad I can be of help.
Lance
Greg Von Kuster wrote:
Hello Lance,
See my inline comments.
On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote:
I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into.
1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separate<action type="set_environment"> tags, then the second overwrites the first in the env.sh file. If instead, I put two<environment_variable> tags within the action, only one (the second) gets executed during installation.
The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now.
I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new "make_directory" tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version.
2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed).
Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation.
Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable.
Thanks for your help on this,
Greg Von Kuster
Any thoughts on how to address these issues would be appreciated.
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
Hello Lance, I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments... On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it.
I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?)
You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing.
4) I attempted to get updates and ran into the issue I described.
I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool.
The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository. http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposito...
Let me know if there is anything I can do to help sort this out.
Lance
Greg Von Kuster wrote:
Hi Lance,
What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions:
1:14e18dc9ed13 2:f5d08224af89 4:14bec14f4290
Since you could not have installed 3:f7a5b54a8d4f, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates?
I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem
Greg Von Kuster
On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks.
URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '4'
Lance Parsons wrote:
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc.
Glad I can be of help.
Lance
Greg Von Kuster wrote:
Hello Lance,
See my inline comments.
On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote:
I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into.
1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separate<action type="set_environment"> tags, then the second overwrites the first in the env.sh file. If instead, I put two<environment_variable> tags within the action, only one (the second) gets executed during installation.
The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now.
I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new "make_directory" tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version.
2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed).
Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation.
Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable.
Thanks for your help on this,
Greg Von Kuster
Any thoughts on how to address these issues would be appreciated.
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will "invalidate" previously valid revisions and thus prevent users from installing or updating the tool at all. For example, push version 0.1 of the tool and create a valid revision 1:xxxxxx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxxxxxx. The tool shed will show 2:xxxxxx as the only valid revision to install, but the galaxy system with revision 1:xxxxxx will be stuck, unable to get upgrades (Server Error described previously). I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks. Lance Greg Von Kuster wrote:
Hello Lance,
I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments...
On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it.
I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?)
You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing.
4) I attempted to get updates and ran into the issue I described.
I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool.
The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository.
http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposito...
Let me know if there is anything I can do to help sort this out.
Lance
Greg Von Kuster wrote:
Hi Lance,
What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions:
1:14e18dc9ed13 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13&id=2df7e24ce6c1f224> 2:f5d08224af89 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89&id=2df7e24ce6c1f224> 4:14bec14f4290 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290&id=2df7e24ce6c1f224>
Since you could not have installed 3:f7a5b54a8d4f <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4f&id=2df7e24ce6c1f224>, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates?
I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem
Greg Von Kuster
On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks.
URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 <http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4> File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '4'
Lance Parsons wrote:
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc.
Glad I can be of help.
Lance
Greg Von Kuster wrote:
Hello Lance,
See my inline comments.
On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote:
> > I've put together a tool wrapper for the htseq-count script that > is part of the HTSeq python package and uploaded that to the > tool shed. However, I have discovered that the tool > dependancies do not install properly. There are a couple of > issues that I've run into. > > 1) The biggest issue is the inability to set both the PYTHONPATH > and PATH environment variables as part of installation. If I > put two separate<action type="set_environment"> tags, then the > second overwrites the first in the env.sh file. If instead, I > put two<environment_variable> tags within the action, only one > (the second) gets executed during installation.
The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now.
I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new "make_directory" tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version.
> > 2) Another issue is the inability to install Numpy as a > dependancy to HTSeq. I can get Numpy to install, but it won't > be in the PYTHONPATH when HTSeq is installed (or executed).
Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation.
Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable.
Thanks for your help on this,
Greg Von Kuster
> > Any thoughts on how to address these issues would be appreciated. > > -- > Lance Parsons - Scientific Programmer > 134 Carl C. Icahn Laboratory > Lewis-Sigler Institute for Integrative Genomics > Princeton University > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
OK, I was able to get a new version installed. It seems there are two issues: 1) New revisions with the same version "ionvalidate" previous revisions. This means that Galaxy servers with the old, and now invalid, revisions are not able to update the tool (nor install it again). 2) Pushes from Mercurial (even version 2.3.3) do not seem to trigger metadata refreshes in the tool shed, however, uploads of tar.gz files do. Hope this helps. Lance Lance Parsons wrote:
I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will "invalidate" previously valid revisions and thus prevent users from installing or updating the tool at all.
For example, push version 0.1 of the tool and create a valid revision 1:xxxxxx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxxxxxx. The tool shed will show 2:xxxxxx as the only valid revision to install, but the galaxy system with revision 1:xxxxxx will be stuck, unable to get upgrades (Server Error described previously).
I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks.
Lance
Greg Von Kuster wrote:
Hello Lance,
I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments...
On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it.
I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?)
You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing.
4) I attempted to get updates and ran into the issue I described.
I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool.
The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository.
http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposito...
Let me know if there is anything I can do to help sort this out.
Lance
Greg Von Kuster wrote:
Hi Lance,
What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions:
1:14e18dc9ed13 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13&id=2df7e24ce6c1f224> 2:f5d08224af89 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89&id=2df7e24ce6c1f224> 4:14bec14f4290 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290&id=2df7e24ce6c1f224>
Since you could not have installed 3:f7a5b54a8d4f <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4f&id=2df7e24ce6c1f224>, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates?
I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem
Greg Von Kuster
On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks.
URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 <http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4> File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '4'
Lance Parsons wrote:
Thanks Greg. I used you're updated version and added numpy as a separate dependency. It seems to work in my development system. I haven't updated my qa or production versions yet, so I can't check there. Perhaps you could test it and let me know if there are issues, etc.
Glad I can be of help.
Lance
Greg Von Kuster wrote: > > Hello Lance, > > See my inline comments. > > > On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: > >> >> I've put together a tool wrapper for the htseq-count script >> that is part of the HTSeq python package and uploaded that to >> the tool shed. However, I have discovered that the tool >> dependancies do not install properly. There are a couple of >> issues that I've run into. >> >> 1) The biggest issue is the inability to set both the >> PYTHONPATH and PATH environment variables as part of >> installation. If I put two separate<action >> type="set_environment"> tags, then the second overwrites the >> first in the env.sh file. If instead, I put >> two<environment_variable> tags within the action, only one >> (the second) gets executed during installation. > > > The above issues have been corrected in changeset > 7621:108cda898646, which is currently available only in the > Galaxy central repository. There is a new Galaxy distribution > scheduled to go out today, but unfortunately this changeset will > not be included. From here on, Galaxy releases are tentatively > scheduled for every 2 weeks, so if you don't want to pull from > the Galaxy central repository, the fix should be available in > the next release 2 weeks from now. > > I've slightly altered your tool_dependencies.xml file (mostly > because I've implemented support for a new "make_directory" > tag. It is attached below - you should updated your > tool_dependencies.xml file in your htseq_count repository with > this updated version. > > >> >> 2) Another issue is the inability to install Numpy as a >> dependancy to HTSeq. I can get Numpy to install, but it won't >> be in the PYTHONPATH when HTSeq is installed (or executed). > > > Were you attempting to install numpy as another package defined > in your tool-dependencies.xml file? If so, the fixes in the > above changeset should allow you to do that now since you can > install it before htseq and set PYTHONPATH to point to numpy as > part of the installation. > > Let me know if you need additional help or bump into further > problems. The tool dependency installation components are > fairly immature, so adding enhancements for new tools like yours > is very valuable. > > Thanks for your help on this, > > Greg Von Kuster > > >> >> Any thoughts on how to address these issues would be appreciated. >> >> -- >> Lance Parsons - Scientific Programmer >> 134 Carl C. Icahn Laboratory >> Lewis-Sigler Institute for Integrative Genomics >> Princeton University >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > >
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
Hi Lance, On Sep 21, 2012, at 6:04 PM, Lance Parsons wrote:
OK, I was able to get a new version installed. It seems there are two issues:
1) New revisions with the same version "ionvalidate" previous revisions. This means that Galaxy servers with the old, and now invalid, revisions are not able to update the tool (nor install it again).
I'm not quite sure what you're stating here. Do the following tool shed wiki page clarify the behavior you are seeing? http://wiki.g2.bx.psu.edu/ToolShedRepositoryFeatures#Pushing_changes_to_a_re... http://wiki.g2.bx.psu.edu/RepositoryRevisions#Installable_repository_changes...
2) Pushes from Mercurial (even version 2.3.3) do not seem to trigger metadata refreshes in the tool shed, however, uploads of tar.gz files do.
I am not able to reproduce this behavior. In my environment, metadata is always automatically generated for new changesets I push to my local tool shed (or the test tool shed) from the command line. What is the result of typing the following in the environment from which you are pushing changes to the tool shed? $hg --version You should see something like the following, showing that you are running at least hg version 2.2.3. gvk:/tmp/repos/convert_chars gvk$ hg --version Mercurial Distributed SCM (version 2.2.3) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Hope this helps.
Lance
Lance Parsons wrote:
I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will "invalidate" previously valid revisions and thus prevent users from installing or updating the tool at all.
For example, push version 0.1 of the tool and create a valid revision 1:xxxxxx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxxxxxx. The tool shed will show 2:xxxxxx as the only valid revision to install, but the galaxy system with revision 1:xxxxxx will be stuck, unable to get upgrades (Server Error described previously).
I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks.
Lance
Greg Von Kuster wrote:
Hello Lance,
I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments...
On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it.
I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?)
You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing.
4) I attempted to get updates and ran into the issue I described.
I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool.
The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository.
http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposito...
Let me know if there is anything I can do to help sort this out.
Lance
Greg Von Kuster wrote:
Hi Lance,
What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions:
1:14e18dc9ed13 2:f5d08224af89 4:14bec14f4290
Since you could not have installed 3:f7a5b54a8d4f, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates?
I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem
Greg Von Kuster
On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks.
URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '4'
Lance Parsons wrote: > > Thanks Greg. I used you're updated version and added numpy as a > separate dependency. It seems to work in my development system. I > haven't updated my qa or production versions yet, so I can't check > there. Perhaps you could test it and let me know if there are issues, etc. > > Glad I can be of help. > > Lance > > Greg Von Kuster wrote: >> >> Hello Lance, >> >> See my inline comments. >> >> >> On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: >> >>> >>> I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into. >>> >>> 1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separate<action type="set_environment"> tags, then the second overwrites the first in the env.sh file. If instead, I put two<environment_variable> tags within the action, only one (the second) gets executed during installation. >> >> >> The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now. >> >> I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new "make_directory" tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version. >> >> >>> >>> 2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed). >> >> >> Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation. >> >> Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable. >> >> Thanks for your help on this, >> >> Greg Von Kuster >> >> >>> >>> Any thoughts on how to address these issues would be appreciated. >>> >>> -- >>> Lance Parsons - Scientific Programmer >>> 134 Carl C. Icahn Laboratory >>> Lewis-Sigler Institute for Integrative Genomics >>> Princeton University >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> >> > > > -- > Lance Parsons - Scientific Programmer > 134 Carl C. Icahn Laboratory > Lewis-Sigler Institute for Integrative Genomics > Princeton University > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario: 1) Upload version 0.1 of MYTOOL at revision 0:XXXXXXXX to the tool shed. 2) Install version 0.1 of MYTOOL at revision 0:XXXXXXX to my local Galaxy instance. 3) Modify MYTOOL, leaving the version at 0.1, but changing the revision to 1:YYYYYYYY. These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.) 4) Attempt to update MYTOOL from my local Galaxy instance. This now results in an exception: "RepoLookupError: unknown revision '1' ". Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool. My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance. As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Let me know if I can help track things down any further, etc. Thanks. Lance Greg Von Kuster wrote:
Hi Lance,
On Sep 21, 2012, at 6:04 PM, Lance Parsons wrote:
OK, I was able to get a new version installed. It seems there are two issues:
1) New revisions with the same version "ionvalidate" previous revisions. This means that Galaxy servers with the old, and now invalid, revisions are not able to update the tool (nor install it again).
I'm not quite sure what you're stating here. Do the following tool shed wiki page clarify the behavior you are seeing?
http://wiki.g2.bx.psu.edu/ToolShedRepositoryFeatures#Pushing_changes_to_a_re... http://wiki.g2.bx.psu.edu/RepositoryRevisions#Installable_repository_changes...
2) Pushes from Mercurial (even version 2.3.3) do not seem to trigger metadata refreshes in the tool shed, however, uploads of tar.gz files do.
I am not able to reproduce this behavior. In my environment, metadata is always automatically generated for new changesets I push to my local tool shed (or the test tool shed) from the command line.
What is the result of typing the following in the environment from which you are pushing changes to the tool shed?
$hg --version
You should see something like the following, showing that you are running at least hg version 2.2.3.
gvk:/tmp/repos/convert_chars gvk$ hg --version Mercurial Distributed SCM (version 2.2.3) (see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Hope this helps.
Lance
Lance Parsons wrote:
I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will "invalidate" previously valid revisions and thus prevent users from installing or updating the tool at all.
For example, push version 0.1 of the tool and create a valid revision 1:xxxxxx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxxxxxx. The tool shed will show 2:xxxxxx as the only valid revision to install, but the galaxy system with revision 1:xxxxxx will be stuck, unable to get upgrades (Server Error described previously).
I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks.
Lance
Greg Von Kuster wrote:
Hello Lance,
I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments...
On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it.
I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?)
You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing.
4) I attempted to get updates and ran into the issue I described.
I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool.
The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository.
http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposito...
Let me know if there is anything I can do to help sort this out.
Lance
Greg Von Kuster wrote:
Hi Lance,
What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions:
1:14e18dc9ed13 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14e18dc9ed13&id=2df7e24ce6c1f224> 2:f5d08224af89 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f5d08224af89&id=2df7e24ce6c1f224> 4:14bec14f4290 <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=14bec14f4290&id=2df7e24ce6c1f224>
Since you could not have installed 3:f7a5b54a8d4f <http://toolshed.g2.bx.psu.edu/repository/view_changeset?ctx_str=f7a5b54a8d4f&id=2df7e24ce6c1f224>, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates?
I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem
Greg Von Kuster
On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote:
> I've updated my development system now, and when I try to get > updates for that particular tool (htseq_count) I run into the > following error. Any ideas on how I can/should fix this? Thanks. > > URL: > http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 > <http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4> > File > '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', > line 364 in respond > app_iter = self.application(environ, detect_start_response) > File > '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', > line 98 in __call__ > environ, self.app) > File > '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', > line 539 in intercept_output > app_iter = application(environ, replacement_start_response) > File > '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', > line 80 in __call__ > return self.application(environ, start_response) > File > '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', > line 91 in __call__ > return self.app( environ, start_response ) > File > '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', > line 632 in __call__ > return self.application(environ, start_response) > File > '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', > line 160 in __call__ > body = method( trans, **kwargs ) > File > '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', > line 184 in decorator > return func( self, trans, *args, **kwargs ) > File > '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', > line 1469 in update_to_changeset_revision > update_repository( repo, latest_ctx_rev ) > File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', > line 1655 in update_repository > rev=ctx_rev ) > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', > line 5661 in update > rev = scmutil.revsingle(repo, rev, rev).rev() > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', > line 515 in revsingle > l = revrange(repo, [revspec]) > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', > line 588 in revrange > for r in m(repo, range(len(repo))): > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', > line 1414 in mfunc > return getset(repo, subset, tree) > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', > line 156 in getset > return methods[x[0]](repo, subset, *x[1:]) > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', > line 171 in symbolset > return stringset(repo, subset, x) > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', > line 161 in stringset > x = repo[x].rev() > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', > line 235 in __getitem__ > return context.changectx(self, changeid) > File > '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', > line 103 in __init__ > _("unknown revision '%s'") % changeid) > RepoLookupError: unknown revision '4' > > > Lance Parsons wrote: >> >> Thanks Greg. I used you're updated version and added numpy as a >> separate dependency. It seems to work in my development system. I >> haven't updated my qa or production versions yet, so I can't check >> there. Perhaps you could test it and let me know if there are >> issues, etc. >> >> Glad I can be of help. >> >> Lance >> >> Greg Von Kuster wrote: >>> >>> Hello Lance, >>> >>> See my inline comments. >>> >>> >>> On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: >>> >>>> >>>> I've put together a tool wrapper for the htseq-count script >>>> that is part of the HTSeq python package and uploaded that to >>>> the tool shed. However, I have discovered that the tool >>>> dependancies do not install properly. There are a couple of >>>> issues that I've run into. >>>> >>>> 1) The biggest issue is the inability to set both the >>>> PYTHONPATH and PATH environment variables as part of >>>> installation. If I put two separate<action >>>> type="set_environment"> tags, then the second overwrites the >>>> first in the env.sh file. If instead, I put >>>> two<environment_variable> tags within the action, only one >>>> (the second) gets executed during installation. >>> >>> >>> The above issues have been corrected in changeset >>> 7621:108cda898646, which is currently available only in the >>> Galaxy central repository. There is a new Galaxy distribution >>> scheduled to go out today, but unfortunately this changeset >>> will not be included. From here on, Galaxy releases are >>> tentatively scheduled for every 2 weeks, so if you don't want >>> to pull from the Galaxy central repository, the fix should be >>> available in the next release 2 weeks from now. >>> >>> I've slightly altered your tool_dependencies.xml file (mostly >>> because I've implemented support for a new "make_directory" >>> tag. It is attached below - you should updated your >>> tool_dependencies.xml file in your htseq_count repository with >>> this updated version. >>> >>> >>>> >>>> 2) Another issue is the inability to install Numpy as a >>>> dependancy to HTSeq. I can get Numpy to install, but it >>>> won't be in the PYTHONPATH when HTSeq is installed (or executed). >>> >>> >>> Were you attempting to install numpy as another package >>> defined in your tool-dependencies.xml file? If so, the fixes >>> in the above changeset should allow you to do that now since >>> you can install it before htseq and set PYTHONPATH to point to >>> numpy as part of the installation. >>> >>> Let me know if you need additional help or bump into further >>> problems. The tool dependency installation components are >>> fairly immature, so adding enhancements for new tools like >>> yours is very valuable. >>> >>> Thanks for your help on this, >>> >>> Greg Von Kuster >>> >>> >>>> >>>> Any thoughts on how to address these issues would be appreciated. >>>> >>>> -- >>>> Lance Parsons - Scientific Programmer >>>> 134 Carl C. Icahn Laboratory >>>> Lewis-Sigler Institute for Integrative Genomics >>>> Princeton University >>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> >>>> http://lists.bx.psu.edu/ >>> >>> >> >> >> -- >> Lance Parsons - Scientific Programmer >> 134 Carl C. Icahn Laboratory >> Lewis-Sigler Institute for Integrative Genomics >> Princeton University >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
Hi Lance, I need to figure out precisely what steps you are taking to produce this behavior, as I have not been able to do so. Please see my inline comments, and let me know more information about each step you are taking to produce this behavior. On Sep 25, 2012, at 10:24 AM, Lance Parsons wrote:
I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario:
1) Upload version 0.1 of MYTOOL at revision 0:XXXXXXXX to the tool shed.
I assume you are uploading the initial tarball or separate files from a certain local directory. Let's call it TOOL_DEV_LOCAL_DIR.
2) Install version 0.1 of MYTOOL at revision 0:XXXXXXX to my local Galaxy instance.
This now creates a new mercurial repository at a specified location on your Galaxy server, let's call it REPO_INSTALL_DIR.
3) Modify MYTOOL, leaving the version at 0.1,
You are doing the above step in TOOL_DEV_LOCAL_DIR, correct?
but changing the revision to 1:YYYYYYYY.
The tool shed is creating the new change set hash for you when you push the changes from TOOL_DEV_LOCAL_DIR to the repository in the tool shed, correct?
These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.)
OK, this make sense.
4) Attempt to update MYTOOL from my local Galaxy instance.
You are using the Galaxy UI to "Get updates" to your cloned repository, which is located in REPO_INSTALL_DIR, correct?
This now results in an exception: " RepoLookupError: unknown revision '1' ".
If answers to all of my above questions are "Yes", then this is where I cannot reproduce the behavior you are seeing. If you are actually adding changsets to your cloned repository located in REPO_INSTALL_DIR, then pushing them to the repository in the tool shed, then attempting to get updates from the tool shed to your cloned repository in REPO_INSTALL_DIR, then this process may be causing the exception you are seeing. You should not be adding changesets to the cloned repository, but to your local development environment in TOOL_DEV_LOCAL_DIR.
Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool.
My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance.
As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I'm running version 2.2.3, so I'll upgrade to version 2.3.1 to see if the newer version breaks my environment.
Let me know if I can help track things down any further, etc. Thanks.
Lance
Greg Von Kuster wrote:
Hi Lance,
On Sep 21, 2012, at 6:04 PM, Lance Parsons wrote:
OK, I was able to get a new version installed. It seems there are two issues:
1) New revisions with the same version "ionvalidate" previous revisions. This means that Galaxy servers with the old, and now invalid, revisions are not able to update the tool (nor install it again).
I'm not quite sure what you're stating here. Do the following tool shed wiki page clarify the behavior you are seeing?
http://wiki.g2.bx.psu.edu/ToolShedRepositoryFeatures#Pushing_changes_to_a_re... http://wiki.g2.bx.psu.edu/RepositoryRevisions#Installable_repository_changes...
2) Pushes from Mercurial (even version 2.3.3) do not seem to trigger metadata refreshes in the tool shed, however, uploads of tar.gz files do.
I am not able to reproduce this behavior. In my environment, metadata is always automatically generated for new changesets I push to my local tool shed (or the test tool shed) from the command line.
What is the result of typing the following in the environment from which you are pushing changes to the tool shed?
$hg --version
You should see something like the following, showing that you are running at least hg version 2.2.3.
gvk:/tmp/repos/convert_chars gvk$ hg --version Mercurial Distributed SCM (version 2.2.3) (see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Hope this helps.
Lance
Lance Parsons wrote:
I've run into this issue again, and I'm having a hard time working around it. However, I have confirmed that at least some updates to a tool in the tool shed will "invalidate" previously valid revisions and thus prevent users from installing or updating the tool at all.
For example, push version 0.1 of the tool and create a valid revision 1:xxxxxx. Then install the tool in galaxy. Make a small change (say to tool_dependencies.xml) and push a new revision (but keep the tool version the same), now at revision 2:xxxxxxx. The tool shed will show 2:xxxxxx as the only valid revision to install, but the galaxy system with revision 1:xxxxxx will be stuck, unable to get upgrades (Server Error described previously).
I'm trying to work around this now with my htseq-count tool, but so far no luck. I've created a few spurious revisions in the attempt, and I think now I may just try bumping the version (already did to no avail, toolshed still thinks it's the same) and uploading a tar file. That seems to more reliably parse metadata. Will let you know what, if anything, works. Thanks.
Lance
Greg Von Kuster wrote:
Hello Lance,
I've just committed a fix for getting updates to installed tool shed repositories in change set 7713:23107188eab8, which is currently available only in the Galaxy central repository. However, my fix will probably not correct the issue you're describing, and I'm still not able to reproduce this behavior. See my inline comments...
On Sep 13, 2012, at 4:41 PM, Lance Parsons wrote:
Actually, I think that is exactly the issue. I DO have 3:f7a5b54a8d4f installed. I've run into a related issue before, but didn't fully understand it.
I believe what happened was: 1) I pushed revision 3:f7a5b54a8d4f to the tool shed which contained the first revision of version 0.2 of the htseq-count tool. 2) I installed the htseq-count tool from the tool shed, getting revision 3:f7a5b54a8d4f 3) I pushed an update to version 0.2 of the htseq-count tool. The only changes were to tool-dependencies so I thought it would be safe to leave the version number alone (perhaps this is problem?)
You are correct in stating that the tool version number should not change just because you've added a tool_dependencies.xml file. This is definitely not causing the behavior you're describing.
4) I attempted to get updates and ran into the issue I described.
I also ran into this (I believe it was with freebayes, but not sure) when I removed (uninstalled) a particular revision of a tool. Then the tool was updated. I went to install and and it said that I already had a previous revision installed and should install that. However, I couldn't since the tool shed won't allow installation of old revisions of the same version of a tool.
The following section of the tool shed wiki should provide the details about why you are seeing this behavior. Keep in mind that you will only get certain updates to installed repositories from the tool shed. This behavior enables updates to installed tool versions. To get a completely new version of an installed tool (if one exists), you need to install a new (different) changeset revision from the tool shed repository.
http://wiki.g2.bx.psu.edu/Tool%20Shed#Getting_updates_for_tool_shed_reposito...
Let me know if there is anything I can do to help sort this out.
Lance
Greg Von Kuster wrote: > > Hi Lance, > > What is the changeset revision that you installed? It looks like you could only have installed one of the following 3 revisions: > > 1:14e18dc9ed13 > 2:f5d08224af89 > 4:14bec14f4290 > > Since you could not have installed 3:f7a5b54a8d4f, I'm not quite sure how you could be trying to update to 4. Did you install 4 and are trying to get updates? > > I've tried several things but am not able to reproduce this behavior, so it's difficult to determine what may be causing the problem > > Greg Von Kuster > > On Sep 12, 2012, at 3:08 PM, Lance Parsons wrote: > >> I've updated my development system now, and when I try to get updates for that particular tool (htseq_count) I run into the following error. Any ideas on how I can/should fix this? Thanks. >> >> URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=f7a5b54a8d4f&latest_changeset_revision=14bec14f4290&latest_ctx_rev=4 >> File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond >> app_iter = self.application(environ, detect_start_response) >> File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ >> environ, self.app) >> File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output >> app_iter = application(environ, replacement_start_response) >> File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ >> return self.application(environ, start_response) >> File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ >> return self.app( environ, start_response ) >> File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ >> return self.application(environ, start_response) >> File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ >> body = method( trans, **kwargs ) >> File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 184 in decorator >> return func( self, trans, *args, **kwargs ) >> File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/controllers/admin_toolshed.py', line 1469 in update_to_changeset_revision >> update_repository( repo, latest_ctx_rev ) >> File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util.py', line 1655 in update_repository >> rev=ctx_rev ) >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update >> rev = scmutil.revsingle(repo, rev, rev).rev() >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle >> l = revrange(repo, [revspec]) >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange >> for r in m(repo, range(len(repo))): >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc >> return getset(repo, subset, tree) >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset >> return methods[x[0]](repo, subset, *x[1:]) >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset >> return stringset(repo, subset, x) >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset >> x = repo[x].rev() >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ >> return context.changectx(self, changeid) >> File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ >> _("unknown revision '%s'") % changeid) >> RepoLookupError: unknown revision '4' >> >> >> Lance Parsons wrote: >>> >>> Thanks Greg. I used you're updated version and added numpy as a >>> separate dependency. It seems to work in my development system. I >>> haven't updated my qa or production versions yet, so I can't check >>> there. Perhaps you could test it and let me know if there are issues, etc. >>> >>> Glad I can be of help. >>> >>> Lance >>> >>> Greg Von Kuster wrote: >>>> >>>> Hello Lance, >>>> >>>> See my inline comments. >>>> >>>> >>>> On Sep 4, 2012, at 3:15 PM, Lance Parsons wrote: >>>> >>>>> >>>>> I've put together a tool wrapper for the htseq-count script that is part of the HTSeq python package and uploaded that to the tool shed. However, I have discovered that the tool dependancies do not install properly. There are a couple of issues that I've run into. >>>>> >>>>> 1) The biggest issue is the inability to set both the PYTHONPATH and PATH environment variables as part of installation. If I put two separate<action type="set_environment"> tags, then the second overwrites the first in the env.sh file. If instead, I put two<environment_variable> tags within the action, only one (the second) gets executed during installation. >>>> >>>> >>>> The above issues have been corrected in changeset 7621:108cda898646, which is currently available only in the Galaxy central repository. There is a new Galaxy distribution scheduled to go out today, but unfortunately this changeset will not be included. From here on, Galaxy releases are tentatively scheduled for every 2 weeks, so if you don't want to pull from the Galaxy central repository, the fix should be available in the next release 2 weeks from now. >>>> >>>> I've slightly altered your tool_dependencies.xml file (mostly because I've implemented support for a new "make_directory" tag. It is attached below - you should updated your tool_dependencies.xml file in your htseq_count repository with this updated version. >>>> >>>> >>>>> >>>>> 2) Another issue is the inability to install Numpy as a dependancy to HTSeq. I can get Numpy to install, but it won't be in the PYTHONPATH when HTSeq is installed (or executed). >>>> >>>> >>>> Were you attempting to install numpy as another package defined in your tool-dependencies.xml file? If so, the fixes in the above changeset should allow you to do that now since you can install it before htseq and set PYTHONPATH to point to numpy as part of the installation. >>>> >>>> Let me know if you need additional help or bump into further problems. The tool dependency installation components are fairly immature, so adding enhancements for new tools like yours is very valuable. >>>> >>>> Thanks for your help on this, >>>> >>>> Greg Von Kuster >>>> >>>> >>>>> >>>>> Any thoughts on how to address these issues would be appreciated. >>>>> >>>>> -- >>>>> Lance Parsons - Scientific Programmer >>>>> 134 Carl C. Icahn Laboratory >>>>> Lewis-Sigler Institute for Integrative Genomics >>>>> Princeton University >>>>> >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> >>>>> http://lists.bx.psu.edu/ >>>> >>>> >>> >>> >>> -- >>> Lance Parsons - Scientific Programmer >>> 134 Carl C. Icahn Laboratory >>> Lewis-Sigler Institute for Integrative Genomics >>> Princeton University >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
I'm actually still a bit confused as to what the expected workflow is. Should I be able to clone the tool shed repo to my local development workstation (TOOL_DEV_LOCAL_DIR), commit a change, and then push that change back to the Tool Shed? I've added some comments inline to clarify. Greg Von Kuster wrote:
Hi Lance,
I need to figure out precisely what steps you are taking to produce this behavior, as I have not been able to do so. Please see my inline comments, and let me know more information about each step you are taking to produce this behavior.
On Sep 25, 2012, at 10:24 AM, Lance Parsons wrote:
I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario:
1) Upload version 0.1 of MYTOOL at revision 0:XXXXXXXX to the tool shed.
I assume you are uploading the initial tarball or separate files from a certain local directory. Let's call it TOOL_DEV_LOCAL_DIR.
Not quite. I would typically create the repository using the Tool Shed web interface. Then, on my local workstation, I would create the TOOL_DEV_LOCAL_DIR by using the 'hg clone http://toolshedurl/repo' command.
2) Install version 0.1 of MYTOOL at revision 0:XXXXXXX to my local Galaxy instance.
This now creates a new mercurial repository at a specified location on your Galaxy server, let's call it REPO_INSTALL_DIR.
Yes, I use the Galaxy Admin interface to install the tool from a tool shed to my Galaxy instance.
3) Modify MYTOOL, leaving the version at 0.1,
You are doing the above step in TOOL_DEV_LOCAL_DIR, correct?
Yes, I modify the files in TOOL_DEV_LOCAL_DIR, leaving the version the same.
but changing the revision to 1:YYYYYYYY.
The tool shed is creating the new change set hash for you when you push the changes from TOOL_DEV_LOCAL_DIR to the repository in the tool shed, correct?
I don't think so. I would typically commit the changes by issuing an 'hg commit' while in my TOOL_DEV_LOCAL_DIR and then push them to the tool shed using the 'hg push' command.
These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.)
OK, this make sense.
Right, I think we're on the same page here, though this should probably be spelled out somewhere in a "Best Practices" type document for tool development (which could also help to clarify the steps to setup a sane development environment, test tools, and push them to the public tool shed. I've not even quite gotten a solid workflow for myself, so publishing one might be useful.
4) Attempt to update MYTOOL from my local Galaxy instance.
You are using the Galaxy UI to "Get updates" to your cloned repository, which is located in REPO_INSTALL_DIR, correct?
Yes, then I use the Galaxy web admin interface to "Get updates".
This now results in an exception: "RepoLookupError: unknown revision '1' ".
If answers to all of my above questions are "Yes", then this is where I cannot reproduce the behavior you are seeing. If you are actually adding changsets to your cloned repository located in REPO_INSTALL_DIR, then pushing them to the repository in the tool shed, then attempting to get updates from the tool shed to your cloned repository in REPO_INSTALL_DIR, then this process may be causing the exception you are seeing. You should not be adding changesets to the cloned repository, but to your local development environment in TOOL_DEV_LOCAL_DIR.
I don't add changesets to the REPO_INSTALL_DIR, but instead make changes in TOOL_DEV_LOCAL_DIR, commit them using mercurial, and then push them to the tool shed using hg push.
Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool.
My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance.
As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I'm running version 2.2.3, so I'll upgrade to version 2.3.1 to see if the newer version breaks my environment.
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
Hi Lance, On Sep 27, 2012, at 4:37 PM, Lance Parsons wrote:
I'm actually still a bit confused as to what the expected workflow is. Should I be able to clone the tool shed repo to my local development workstation (TOOL_DEV_LOCAL_DIR), commit a change, and then push that change back to the Tool Shed?
Assuming that you eliminate everything in your TOOL_DEV_LOCAL_DIR (you have a clean directory, eliminating any .hg directories that may exist), pull a fresh clone from the tool shed, make changes to the clone you've just pull into TOOL_DEV_LOCAL_DIR, push the changes to the tool shed, this should be fine. A better approach would probably be to simply keep your TOOL_DEV_LOCAL_DIR as the "master repository", never pulling to it from the tool shed (because no one except you is able to change the repository in the tool shed) and just pushing changes from TOOL_DEV_LOCAL_DIR to the tool shed repository. So, the path is something like: TOOL_DEV_LOCAL_DIR pushes new changes to your tool shed repository, and others in the Galaxy community clone from your tool shed repository, including updates you committed over time Greg Von Kuster
I've added some comments inline to clarify.
Greg Von Kuster wrote:
Hi Lance,
I need to figure out precisely what steps you are taking to produce this behavior, as I have not been able to do so. Please see my inline comments, and let me know more information about each step you are taking to produce this behavior.
On Sep 25, 2012, at 10:24 AM, Lance Parsons wrote:
I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario:
1) Upload version 0.1 of MYTOOL at revision 0:XXXXXXXX to the tool shed.
I assume you are uploading the initial tarball or separate files from a certain local directory. Let's call it TOOL_DEV_LOCAL_DIR.
Not quite. I would typically create the repository using the Tool Shed web interface. Then, on my local workstation, I would create the TOOL_DEV_LOCAL_DIR by using the 'hg clone http://toolshedurl/repo' command.
2) Install version 0.1 of MYTOOL at revision 0:XXXXXXX to my local Galaxy instance.
This now creates a new mercurial repository at a specified location on your Galaxy server, let's call it REPO_INSTALL_DIR.
Yes, I use the Galaxy Admin interface to install the tool from a tool shed to my Galaxy instance.
3) Modify MYTOOL, leaving the version at 0.1,
You are doing the above step in TOOL_DEV_LOCAL_DIR, correct?
Yes, I modify the files in TOOL_DEV_LOCAL_DIR, leaving the version the same.
but changing the revision to 1:YYYYYYYY.
The tool shed is creating the new change set hash for you when you push the changes from TOOL_DEV_LOCAL_DIR to the repository in the tool shed, correct?
I don't think so. I would typically commit the changes by issuing an 'hg commit' while in my TOOL_DEV_LOCAL_DIR and then push them to the tool shed using the 'hg push' command.
These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.)
OK, this make sense.
Right, I think we're on the same page here, though this should probably be spelled out somewhere in a "Best Practices" type document for tool development (which could also help to clarify the steps to setup a sane development environment, test tools, and push them to the public tool shed. I've not even quite gotten a solid workflow for myself, so publishing one might be useful.
4) Attempt to update MYTOOL from my local Galaxy instance.
You are using the Galaxy UI to "Get updates" to your cloned repository, which is located in REPO_INSTALL_DIR, correct?
Yes, then I use the Galaxy web admin interface to "Get updates".
This now results in an exception: " RepoLookupError: unknown revision '1' ".
If answers to all of my above questions are "Yes", then this is where I cannot reproduce the behavior you are seeing. If you are actually adding changsets to your cloned repository located in REPO_INSTALL_DIR, then pushing them to the repository in the tool shed, then attempting to get updates from the tool shed to your cloned repository in REPO_INSTALL_DIR, then this process may be causing the exception you are seeing. You should not be adding changesets to the cloned repository, but to your local development environment in TOOL_DEV_LOCAL_DIR.
I don't add changesets to the REPO_INSTALL_DIR, but instead make changes in TOOL_DEV_LOCAL_DIR, commit them using mercurial, and then push them to the tool shed using hg push.
Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool.
My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance.
As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I'm running version 2.2.3, so I'll upgrade to version 2.3.1 to see if the newer version breaks my environment.
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
Hi lance, Another "best practice" I should have mentioned: if you are using your TOOL_DEV_LOCAL_DIR as a source code repository for developing your tools (where you are committing tool development change sets into your hg repository in TOOL_DEV_LOCAL_DIR), you should not push the change log history to the tool shed when you have your developed tools functional in TOOL_DEV_LOCAL_DIR The tool shed generates metadata for the contents of each change set pushed to its repository, so you want a single, functionally correct change set for each push to the tool shed, not the history of small development changes you made in your development repository in TOOL_DEV_LOCAL_DIR. A good process to use for this is to create a new tarball of your TOOL_DEV_LOCAL_DIR repository when you are ready to push it to the tool shed and upload the tarball. The tool shed will strip out unwanted items from your tarball like .hg subdirectories. You could also choose to not use a source code revision system in your TOOL_DEV_LOCAL_DIR if that is more appropriate. Greg Von Kuster On Sep 27, 2012, at 5:02 PM, Greg Von Kuster wrote:
Hi Lance,
On Sep 27, 2012, at 4:37 PM, Lance Parsons wrote:
I'm actually still a bit confused as to what the expected workflow is. Should I be able to clone the tool shed repo to my local development workstation (TOOL_DEV_LOCAL_DIR), commit a change, and then push that change back to the Tool Shed?
Assuming that you eliminate everything in your TOOL_DEV_LOCAL_DIR (you have a clean directory, eliminating any .hg directories that may exist), pull a fresh clone from the tool shed, make changes to the clone you've just pull into TOOL_DEV_LOCAL_DIR, push the changes to the tool shed, this should be fine.
A better approach would probably be to simply keep your TOOL_DEV_LOCAL_DIR as the "master repository", never pulling to it from the tool shed (because no one except you is able to change the repository in the tool shed) and just pushing changes from TOOL_DEV_LOCAL_DIR to the tool shed repository.
So, the path is something like:
TOOL_DEV_LOCAL_DIR pushes new changes to your tool shed repository, and others in the Galaxy community clone from your tool shed repository, including updates you committed over time
Greg Von Kuster
I've added some comments inline to clarify.
Greg Von Kuster wrote:
Hi Lance,
I need to figure out precisely what steps you are taking to produce this behavior, as I have not been able to do so. Please see my inline comments, and let me know more information about each step you are taking to produce this behavior.
On Sep 25, 2012, at 10:24 AM, Lance Parsons wrote:
I'm sorry I wasn't more clear. I do believe that those links explain the behavior I am seeing. However, let me try to describe it a different way. It seems that there will, at most, be one installable revision of a given version of a tool. Here I use revision to denote a mercurial revision and version to denote the version described in the tool's config xml. However, while it seems this is the desired behavior, it seems to lead to an undesirable situation in some circumstances. Consider this scenario:
1) Upload version 0.1 of MYTOOL at revision 0:XXXXXXXX to the tool shed.
I assume you are uploading the initial tarball or separate files from a certain local directory. Let's call it TOOL_DEV_LOCAL_DIR.
Not quite. I would typically create the repository using the Tool Shed web interface. Then, on my local workstation, I would create the TOOL_DEV_LOCAL_DIR by using the 'hg clone http://toolshedurl/repo' command.
2) Install version 0.1 of MYTOOL at revision 0:XXXXXXX to my local Galaxy instance.
This now creates a new mercurial repository at a specified location on your Galaxy server, let's call it REPO_INSTALL_DIR.
Yes, I use the Galaxy Admin interface to install the tool from a tool shed to my Galaxy instance.
3) Modify MYTOOL, leaving the version at 0.1,
You are doing the above step in TOOL_DEV_LOCAL_DIR, correct?
Yes, I modify the files in TOOL_DEV_LOCAL_DIR, leaving the version the same.
but changing the revision to 1:YYYYYYYY.
The tool shed is creating the new change set hash for you when you push the changes from TOOL_DEV_LOCAL_DIR to the repository in the tool shed, correct?
I don't think so. I would typically commit the changes by issuing an 'hg commit' while in my TOOL_DEV_LOCAL_DIR and then push them to the tool shed using the 'hg push' command.
These updates would be for things that don't change the input or output of a tool (such as updates to documentation, the addition of tool dependencies, etc.)
OK, this make sense.
Right, I think we're on the same page here, though this should probably be spelled out somewhere in a "Best Practices" type document for tool development (which could also help to clarify the steps to setup a sane development environment, test tools, and push them to the public tool shed. I've not even quite gotten a solid workflow for myself, so publishing one might be useful.
4) Attempt to update MYTOOL from my local Galaxy instance.
You are using the Galaxy UI to "Get updates" to your cloned repository, which is located in REPO_INSTALL_DIR, correct?
Yes, then I use the Galaxy web admin interface to "Get updates".
This now results in an exception: " RepoLookupError: unknown revision '1' ".
If answers to all of my above questions are "Yes", then this is where I cannot reproduce the behavior you are seeing. If you are actually adding changsets to your cloned repository located in REPO_INSTALL_DIR, then pushing them to the repository in the tool shed, then attempting to get updates from the tool shed to your cloned repository in REPO_INSTALL_DIR, then this process may be causing the exception you are seeing. You should not be adding changesets to the cloned repository, but to your local development environment in TOOL_DEV_LOCAL_DIR.
I don't add changesets to the REPO_INSTALL_DIR, but instead make changes in TOOL_DEV_LOCAL_DIR, commit them using mercurial, and then push them to the tool shed using hg push.
Deactivating and/or uninstalling MYTOOL works, but any attempt to reinstall the latest version results in an error stating that an older revision of the tool was installed. I now have to way to (easily) update my local Galaxy instance to this updated revision of the tool.
My only workaround to this was to increment the version of MYTOOL and push another revision. This allowed me to install the new version and deactivate the old version in my local Galaxy instance.
As for the mercurial issue, I'm running OS X, using the Homebrew version of Mercurial: hg --version Mercurial Distributed SCM (version 2.3.1) (see http://mercurial.selenic.com for more information)
Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I'm running version 2.2.3, so I'll upgrade to version 2.3.1 to see if the newer version breaks my environment.
-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University
participants (2)
-
Greg Von Kuster -
Lance Parsons