Since I've had issues with workflow and the toolshed in the past, I have changed my workflow so that my only interaction with the toolshed is either via uploading a tarball via the web interface or installing a tool into my local Galaxy via the web interface (or occasionally just downloading tarballs). In this specific case, I uploaded a tar.gz file of the entire tool wrapper directory. I did NOT increment the tool version. In the past, incrementing the tool version has generally worked fine. However, when I leave the tool version the same, but change something (such as the help text for the tool), and then upload a tarball to the toolshed, I am unable to pull updates into my local galaxy instance. I get an error similar to what I have described. I hope this is enough information, but let me know if I can provide more detail, etc. I would very much like to get to the root of this issue, since it has plagued my attempts to use the toolshed smoothly when (seemingly inevitable) minor changes need to be made. In the past I have just had to wait until I update the tool version, as no changes can be made to the installed revision in my production Galaxy instance. Greg Von Kuster wrote:

Hi Lance,

What process did you use for updating the tool shed repository and how many separate repositories did you use? You should not be updating a local repository, pushing the changes to the tool shed repository, and then attempting to pull the changes back to the same repository. Your repository environment should be something like this:

DEV REPO ( all development updates are performed here, so this is where the tool version is incremented ) | V <pushed changeset> | V EXISTING TOOL SHED REPO | V <pulled changeset to some previously cloned repo> | V PREVIOUSLY CLONED (possibly production) REPO

In other words, nothing should ever get pushed and pulled from the tool shed repo into the same repo. If this is the process you used, it would undoubtedly result in the behavior you saw. If you used separate repos as I describe above, we'll need more details as to the process you used in order to determine the cause of the problem.

Greg Von Kuster

On Dec 10, 2012, at 1:45 PM, Lance Parsons wrote:

...

I recently updated the htseq-count tool in the Galaxy toolshed to version 0.3. This version has some functional changes and thus the tool version was incremented. I installed this new version of the tool in my local Galaxy instance.

However, I realized later that I missed updating one part of the help text. Today I fixed that text (but left the tool version the same), tarred up the file, and uploaded it to the toolshed. I then attempted to "Get Updates" from my local Galaxy instances for that tool got the following error:

URL: http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=5d969cb56112&latest_changeset_revision=f320093f1e8e&latest_ctx_rev=11 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 206 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1677 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util_common.py', line 1428 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '11'

I believe this is because the revision I have installed (10) is not longer considered a valid revision by the toolshed (I presume since it is an older revision of with the same tool version).

Any help would be appreciated. Thanks.

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

Thanks for looking into this Greg. The environment where I'm having this problem is our testing and production systems. Both environments are CentOS release 5.8 (Final) with Mercurial version 2.4.1. Interestingly, when I tried the same thing with a development setup on my OSX box (10.6.8) with Mercurial 2.4 I did not have the same issue. One difference is that on my testing/production boxes I do have an older version of the tool installed, so perhaps that is the issue. I also tried uninstalling, but that led to the (expected) error that that a previous changeset is installed: The tool shed repository htseq_count with owner lparsons and changeset revision f320093f1e8e was previously installed using changeset revision 5d969cb56112. The repository has been uninstalled, however, so reinstall the original repository instead of installing it again. You can get the latest updates for the repository using the Get updates option from the repository's Repository Actions pop-up menu. Click here to manage the repository. I then attempted to reinstall and I got a javascript message the cloning failed, and the repo is now "stuck" in the cloning state. I did not see anything in the paster logs regarding this at all. Here are the steps I took on my local system: 1. Upload htseq-count v0.3 to local toolshed 2. Install on local galaxy 3. Upload htseq-count v0.3-release2 to toolshed [Errno 2] No such file or directory: '/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/database/community_files/000/repo_3/./tool_data_table_conf.xml.sample' STDOUT from paster htseq-count.xml does not exist! sam_fa_indices.loc.sample does not exist! static/images/count_modes.png does not exist! test-data/htseq-test-paired.bam does not exist! test-data/htseq-test-paired_counts.tsv does not exist! test-data/htseq-test-paired_othercounts.tsv does not exist! test-data/htseq-test.bam does not exist! test-data/htseq-test.gff does not exist! test-data/htseq-test.sam does not exist! test-data/htseq-test_counts.tsv does not exist! test-data/htseq-test_othercounts.tsv does not exist! tool_data_table_conf.xml.sample does not exist! 4. Second attempt to upload The file 'htseq-count_0.3.release2.tar.gz' has been successfully uncompressed and uploaded to the repository. 1 undesirable items (.hg .svn .git directories, .DS_Store, hgrc files, etc) were removed from the archive. 13 files were removed from the repository root. 5. Get repository updates The installed repository named 'htseq_count' has been updated to change set revision '74a6a8221337'. Greg Von Kuster wrote:

Hi Lance,

I tried several scenarios today in an attempt to reproduce this but I was not able to. Have you tried uninstalling the repository (check the checkbox to completely uninstall it) and then reinstall it? I'm trying to reproduce this on a mac running mercurial version 2.2.3 and Python 2.5. What's your environment?

Greg Von Kuster

On Dec 13, 2012, at 2:30 PM, Lance Parsons wrote:

...

Since I've had issues with workflow and the toolshed in the past, I have changed my workflow so that my only interaction with the toolshed is either via uploading a tarball via the web interface or installing a tool into my local Galaxy via the web interface (or occasionally just downloading tarballs).

In this specific case, I uploaded a tar.gz file of the entire tool wrapper directory. I did NOT increment the tool version. In the past, incrementing the tool version has generally worked fine. However, when I leave the tool version the same, but change something (such as the help text for the tool), and then upload a tarball to the toolshed, I am unable to pull updates into my local galaxy instance. I get an error similar to what I have described.

I hope this is enough information, but let me know if I can provide more detail, etc. I would very much like to get to the root of this issue, since it has plagued my attempts to use the toolshed smoothly when (seemingly inevitable) minor changes need to be made. In the past I have just had to wait until I update the tool version, as no changes can be made to the installed revision in my production Galaxy instance.

Greg Von Kuster wrote:

...

Hi Lance,

What process did you use for updating the tool shed repository and how many separate repositories did you use? You should not be updating a local repository, pushing the changes to the tool shed repository, and then attempting to pull the changes back to the same repository. Your repository environment should be something like this:

DEV REPO ( all development updates are performed here, so this is where the tool version is incremented ) | V <pushed changeset> | V EXISTING TOOL SHED REPO | V <pulled changeset to some previously cloned repo> | V PREVIOUSLY CLONED (possibly production) REPO

In other words, nothing should ever get pushed and pulled from the tool shed repo into the same repo. If this is the process you used, it would undoubtedly result in the behavior you saw. If you used separate repos as I describe above, we'll need more details as to the process you used in order to determine the cause of the problem.

Greg Von Kuster

On Dec 10, 2012, at 1:45 PM, Lance Parsons wrote:

...

I recently updated the htseq-count tool in the Galaxy toolshed to version 0.3. This version has some functional changes and thus the tool version was incremented. I installed this new version of the tool in my local Galaxy instance.

However, I realized later that I missed updating one part of the help text. Today I fixed that text (but left the tool version the same), tarred up the file, and uploaded it to the toolshed. I then attempted to "Get Updates" from my local Galaxy instances for that tool got the following error:

URL:http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=5d969cb56112&latest_changeset_revision=f320093f1e8e&latest_ctx_rev=11 File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 206 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1677 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util_common.py', line 1428 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '11'

I believe this is because the revision I have installed (10) is not longer considered a valid revision by the toolshed (I presume since it is an older revision of with the same tool version).

Any help would be appreciated. Thanks.

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

Hi all, Reply to my own post... So we nailed it down, well almost. The problem comes from the fact that we are using the 'run job as real user'. When we switched back to normal 'all jobs run as galaxy', it is just working fine. When run as real user, the error we are getting from LSF is "can t open lsf.sudoers" (well you need to hack in the drmaa C code to log this out). We traced the runner operations with strace and the process really tries to read this file (and it fails as LSF imposes that this file is owned by root and as 600 rights). The strange thing is that when we disable the 'run as user' feature, this goes away. It might be the setuid() executed during the 'run as real user' procedure that somehow forces the process to access this file (ie imposed by LSF in some way) but we are lost. So we just run as galaxy now... Comments welcome! Does anybody run LSF with the "run as real user" feature on? Thx C On 18 Dec 2012, at 10:47, Charles Girardot wrote:

Hi all,

We are currently changing how cluster management from PBSPro to LSF (LSF 7 Update 6). We have a running Galaxy using drmaa with PBSPro (with the "job are submitted as real users" option). We expected an easy transition to LSF i.e. simply changing the drmaa implementation but of course, life is not that simple. So basically it is not working. We have tried with drmaa 1.0.4 and 1.0.3 (downloaded from http://sourceforge.net/projects/lsf-drmaa/ ).

Before getting to the symptoms: does anybody successfully run Galaxy with drmaa and LSF 7 Update 6 ?

Now the symptoms: - first we had an error saying something like "queued as Job <5160> is submitted to default queue <medium_priority>" is not an idea - we traced this in the drmaa C code and added a regex to actually extract the job id (if you are successfully running Galaxy with drmaa and LSF 7 Update 6; did you also have to do this??);

but then a new error came:

- jobs are successfully sent to the LSF queue and submitted to a node - after few ms we get an error : galaxy.jobs.runners.drmaa DEBUG 2012-12-17 11:14:29,227 (1699) submitting with credentials: sauer [uid: 8483] galaxy.jobs.runners.drmaa DEBUG 2012-12-17 11:14:29,229 (1699) Job script for external submission is: /g/galaxy/galaxy-dev_data/pbs/1699.jt_json galaxy.jobs.runners.drmaa INFO 2012-12-17 11:14:29,464 (1699) queued as Job <5160> is submitted to default queue <medium_priority>. E #2bae [ 0.00] * call to lsb_openjobinfo returned with error 1:No matching job found mapped to 1040:Job does not exist in DRMs queue. galaxy.jobs.runners.drmaa DEBUG 2012-12-17 11:14:30,275 (1699/Job <5160> is submitted to default queue <medium_priority>. 5160) job left DRM queue with following message: code 18: lsb_openjobinfo: XDR operation error

We are lost and the PBSPro license runs out on January 1 so we badly need to fix this...

PS: Note that if we simply switch back to PBSPro, it is all working fine; which tells us that the Galaxy setup is ok.

Thx for your help

bw

Charles

===================================== Charles Girardot European Molecular Biology Laboratory E. Furlong Group http://furlonglab.embl.de Tel: +49 6221 387 -8585 (V205) or 8433 (V320) Fax: +49-(0)6221-387-8166 Email: charles.girardot@embl.de Room V205/V320 Meyerhofstraße 1, 69117 Heidelberg, Germany =====================================

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

===================================== Charles Girardot European Molecular Biology Laboratory E. Furlong Group http://furlonglab.embl.de Tel: +49 6221 387 -8585 (V205) or 8433 (V320) Fax: +49-(0)6221-387-8166 Email: charles.girardot@embl.de Room V205/V320 Meyerhofstraße 1, 69117 Heidelberg, Germany =====================================

Hi Joachim, It seems that the error is because your system is trying to install the library files to lib64/python instead of just lib/python. For reasons I don't quite understand, the setup.py installation fails when that directory doesn't exist (I would think it would just create it, but that doesn't appear to be the case). As a quick fix, you might try creating the directory /apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/. Create it as the user that galaxy runs under so make sure galaxy has permissions later. Then try reinstalling the dependency. I'm not sure that will work, but that is the gist of the problem. I noticed there is an update to the htseq-count package, so I'll try to put out a new version of the wrapper with a fix for that as well as the new version. Let me know if you still have trouble or if you find some sort of workaround or fix. I'd like to make the installation more robust. Lance Joachim Jacob | VIB | wrote:

Hi Lance,

[apologies for directly emailing to you and not through the galaxy-dev list. I will post it on the dev list later on]

I am trying to install htseq-count, but it fails during the dependency installation of htseq.

The error I get is:

error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the installation directory:

[Errno 2] No such file or directory: '/apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/test-easy-install-30482.pth'

The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was:

/apps/htseq/0.5.3p9/lparsons/htseq_count/f320093f1e8e/lib64/python/

This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option).

Any idea whether it is related to my setup (and what I can do about it)? Note: only the htseq_count dependency is not being installed. Thanks for providing the htseq_count wrapper! Now getting it through the toolshed will make it again a little closer to a perfect world :-)

Cheers, Joachim

Joachim Jacob

Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

On 12/17/2012 05:28 PM, Lance Parsons wrote:

...

Thanks for looking into this Greg. The environment where I'm having this problem is our testing and production systems.

Both environments are CentOS release 5.8 (Final) with Mercurial version 2.4.1.

Interestingly, when I tried the same thing with a development setup on my OSX box (10.6.8) with Mercurial 2.4 I did not have the same issue. One difference is that on my testing/production boxes I do have an older version of the tool installed, so perhaps that is the issue.

I also tried uninstalling, but that led to the (expected) error that that a previous changeset is installed: The tool shed repository htseq_count with owner lparsons and changeset revision f320093f1e8e was previously installed using changeset revision 5d969cb56112. The repository has been uninstalled, however, so reinstall the original repository instead of installing it again. You can get the latest updates for the repository using the Get updates option from the repository's Repository Actions pop-up menu. Click here to manage the repository.

I then attempted to reinstall and I got a javascript message the cloning failed, and the repo is now "stuck" in the cloning state. I did not see anything in the paster logs regarding this at all.

Here are the steps I took on my local system:

1. Upload htseq-count v0.3 to local toolshed

2. Install on local galaxy

3. Upload htseq-count v0.3-release2 to toolshed

[Errno 2] No such file or directory: '/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/database/community_files/000/repo_3/./tool_data_table_conf.xml.sample'

STDOUT from paster htseq-count.xml does not exist! sam_fa_indices.loc.sample does not exist! static/images/count_modes.png does not exist! test-data/htseq-test-paired.bam does not exist! test-data/htseq-test-paired_counts.tsv does not exist! test-data/htseq-test-paired_othercounts.tsv does not exist! test-data/htseq-test.bam does not exist! test-data/htseq-test.gff does not exist! test-data/htseq-test.sam does not exist! test-data/htseq-test_counts.tsv does not exist! test-data/htseq-test_othercounts.tsv does not exist! tool_data_table_conf.xml.sample does not exist! 4. Second attempt to upload The file 'htseq-count_0.3.release2.tar.gz' has been successfully uncompressed and uploaded to the repository. 1 undesirable items (.hg .svn .git directories, .DS_Store, hgrc files, etc) were removed from the archive. 13 files were removed from the repository root.

5. Get repository updates The installed repository named 'htseq_count' has been updated to change set revision '74a6a8221337'.

Greg Von Kuster wrote:

...

Hi Lance,

I tried several scenarios today in an attempt to reproduce this but I was not able to. Have you tried uninstalling the repository (check the checkbox to completely uninstall it) and then reinstall it? I'm trying to reproduce this on a mac running mercurial version 2.2.3 and Python 2.5. What's your environment?

Greg Von Kuster

On Dec 13, 2012, at 2:30 PM, Lance Parsons wrote:

...

Since I've had issues with workflow and the toolshed in the past, I have changed my workflow so that my only interaction with the toolshed is either via uploading a tarball via the web interface or installing a tool into my local Galaxy via the web interface (or occasionally just downloading tarballs).

In this specific case, I uploaded a tar.gz file of the entire tool wrapper directory. I did NOT increment the tool version. In the past, incrementing the tool version has generally worked fine. However, when I leave the tool version the same, but change something (such as the help text for the tool), and then upload a tarball to the toolshed, I am unable to pull updates into my local galaxy instance. I get an error similar to what I have described.

I hope this is enough information, but let me know if I can provide more detail, etc. I would very much like to get to the root of this issue, since it has plagued my attempts to use the toolshed smoothly when (seemingly inevitable) minor changes need to be made. In the past I have just had to wait until I update the tool version, as no changes can be made to the installed revision in my production Galaxy instance.

Greg Von Kuster wrote:

...

Hi Lance,

What process did you use for updating the tool shed repository and how many separate repositories did you use? You should not be updating a local repository, pushing the changes to the tool shed repository, and then attempting to pull the changes back to the same repository. Your repository environment should be something like this:

DEV REPO ( all development updates are performed here, so this is where the tool version is incremented ) | V <pushed changeset> | V EXISTING TOOL SHED REPO | V <pulled changeset to some previously cloned repo> | V PREVIOUSLY CLONED (possibly production) REPO

In other words, nothing should ever get pushed and pulled from the tool shed repo into the same repo. If this is the process you used, it would undoubtedly result in the behavior you saw. If you used separate repos as I describe above, we'll need more details as to the process you used in order to determine the cause of the problem.

Greg Von Kuster

On Dec 10, 2012, at 1:45 PM, Lance Parsons wrote:

...

I recently updated the htseq-count tool in the Galaxy toolshed to version 0.3. This version has some functional changes and thus the tool version was incremented. I installed this new version of the tool in my local Galaxy instance.

However, I realized later that I missed updating one part of the help text. Today I fixed that text (but left the tool version the same), tarred up the file, and uploaded it to the toolshed. I then attempted to "Get Updates" from my local Galaxy instances for that tool got the following error:

URL:http://galaxy-dev.princeton.edu/admin_toolshed/update_to_changeset_revision?tool_shed_url=http://toolshed.g2.bx.psu.edu/&name=htseq_count&owner=lparsons&changeset_revision=5d969cb56112&latest_changeset_revision=f320093f1e8e&latest_ctx_rev=11

File '/data/galaxy-dev/galaxy-dev/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/data/galaxy-dev/galaxy-dev/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/web/framework/__init__.py', line 206 in decorator return func( self, trans, *args, **kwargs ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1677 in update_to_changeset_revision update_repository( repo, latest_ctx_rev ) File '/data/galaxy-dev/galaxy-dev/lib/galaxy/util/shed_util_common.py', line 1428 in update_repository rev=ctx_rev ) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/commands.py', line 5661 in update rev = scmutil.revsingle(repo, rev, rev).rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 515 in revsingle l = revrange(repo, [revspec]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/scmutil.py', line 588 in revrange for r in m(repo, range(len(repo))): File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 1414 in mfunc return getset(repo, subset, tree) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 156 in getset return methods[x[0]](repo, subset, *x[1:]) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 171 in symbolset return stringset(repo, subset, x) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/revset.py', line 161 in stringset x = repo[x].rev() File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/localrepo.py', line 235 in __getitem__ return context.changectx(self, changeid) File '/data/galaxy-dev/galaxy-dev/eggs/mercurial-2.2.3-py2.6-linux-x86_64-ucs2.egg/mercurial/context.py', line 103 in __init__ _("unknown revision '%s'") % changeid) RepoLookupError: unknown revision '11'

I believe this is because the revision I have installed (10) is not longer considered a valid revision by the toolshed (I presume since it is an older revision of with the same tool version).

Any help would be appreciated. Thanks.

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

-- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University