Strange repository dependency behavior with toolshed-installed naive_variant_caller
First, I'm at a bit of a loss to even describe what's going on, so I apologize in advance for a rambling bug report. Please bear with me! The naive_variant_caller tool was failing on our instance with "Failed to resolve dependency" errors on numpy, pyBamParser, and pyBamTools, despite the tool dependencies and their respective dependency packages being installed. I attempted to use the "repair repository" action on naive_variant_caller with no success. I then uninstalled naive_variant_caller, package_numpy_1_7, package_pybamparser_0_0_1, and package_pybamtools_0_0_1 and attempted to reinstall naive_variant_caller from scratch, again allowing Galaxy to manage the dependencies. This time, the dependencies resolved (at least as far as the "Building dependency shell command" messages were concerned), but naive_variant_caller still failed, this time with an import error: ImportError: No module named pyBamTools.genotyping.naive Digging around a little I found that the env.sh that was sourced for pyBamTools was looking for an env.sh in another pyBamTools revision directory, but the revision it was looking for didn't exist. Then I looked in the admin interface and saw that package_pybamtools_0_0_1 and package_pybamparser_0_0_1 were each installed twice; each had both the most recent revision and an older revision with the "updates are available" icon. At this point I uninstalled naive_variant_caller and all of its dependency packages again. I then installed naive_variant_caller from the toolshed *without* dependency management, and manually installed the required revisions of numpy, pybamparser, and pybamtools. Finally naive_variant_caller started working. However, naive_variant_caller is now shown in the admin interface as "Installed, missing repository dependencies", and the installed repository page shows that the dependencies are a very strange state (see attached screenshot). That is, the same revision of numpy is listed as installed twice, and the pybamparser and pybamtools dependencies are each listed as simultaneously installed and uninstalled. Can anyone shed some light on what could be happening and what to do about it? Is there some external way to verify the consistency of a tool and its dependencies? Thanks! Cheers, Brian -- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
Hello Brian, I have just tried to install and uninstall NVC on my development machine multiple times and was not able to replicate the problem. However I have seen similar things before. Let me ask you few questions: Are you using Admin UI for installation or API scripts? Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these? If you check on the filesystem in the {{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication? How big is your Galaxy in terms of installed tools, complexity of setup and traffic? Sorry for the many questions but I hope it might help us track down the problems. Thank you Martin, Galaxy Team On Mon Feb 23 2015 at 10:38:08 PM Brian Claywell <bclaywel@fredhutch.org> wrote:
First, I'm at a bit of a loss to even describe what's going on, so I apologize in advance for a rambling bug report. Please bear with me!
The naive_variant_caller tool was failing on our instance with "Failed to resolve dependency" errors on numpy, pyBamParser, and pyBamTools, despite the tool dependencies and their respective dependency packages being installed.
I attempted to use the "repair repository" action on naive_variant_caller with no success. I then uninstalled naive_variant_caller, package_numpy_1_7, package_pybamparser_0_0_1, and package_pybamtools_0_0_1 and attempted to reinstall naive_variant_caller from scratch, again allowing Galaxy to manage the dependencies.
This time, the dependencies resolved (at least as far as the "Building dependency shell command" messages were concerned), but naive_variant_caller still failed, this time with an import error:
ImportError: No module named pyBamTools.genotyping.naive
Digging around a little I found that the env.sh that was sourced for pyBamTools was looking for an env.sh in another pyBamTools revision directory, but the revision it was looking for didn't exist.
Then I looked in the admin interface and saw that package_pybamtools_0_0_1 and package_pybamparser_0_0_1 were each installed twice; each had both the most recent revision and an older revision with the "updates are available" icon.
At this point I uninstalled naive_variant_caller and all of its dependency packages again. I then installed naive_variant_caller from the toolshed *without* dependency management, and manually installed the required revisions of numpy, pybamparser, and pybamtools. Finally naive_variant_caller started working.
However, naive_variant_caller is now shown in the admin interface as "Installed, missing repository dependencies", and the installed repository page shows that the dependencies are a very strange state (see attached screenshot). That is, the same revision of numpy is listed as installed twice, and the pybamparser and pybamtools dependencies are each listed as simultaneously installed and uninstalled.
Can anyone shed some light on what could be happening and what to do about it? Is there some external way to verify the consistency of a tool and its dependencies?
Thanks!
Cheers,
Brian
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Martin, Thanks for the prompt reply! My responses are inline below. On Tue, Feb 24, 2015 at 8:26 AM, Martin Čech <marten@bx.psu.edu> wrote:
Are you using Admin UI for installation or API scripts?
The admin UI exclusively.
Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these?
Yes -- the iuc package_numpy_1_7 revision a4bdc17eed4a is listed twice, with both showing the green "Installed" status. Interestingly the installed repo page shows that they're installed in different locations: (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/ef12a3a11d5b/package_numpy_1_7 (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/a4bdc17eed4a/package_numpy_1_7 At one point during the troubleshooting steps I described in my original email, package_pybamparser_0_0_1 and package_pybamtools_0_0_1 showed up in the same way, with two records each having the same revision and a green "Installed" status. If you check on the filesystem in the
{{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication?
I didn't see anything that struck me as out of the ordinary. Here's a gist of the directory structures for the affected packages under both shed_tools and tool_deps: https://gist.github.com/bcclaywell/23c0b274f2d4bd0d659f How big is your Galaxy in terms of installed tools, complexity of setup and
traffic?
I don't think it's terribly big -- we have a couple of large tool suites like BEDTools, the gops interval operations suites, GATK 1 and 2, etc, and then maybe 15-20 other tools and their dependencies? Our setup isn't very complex, either; Galaxy is running with one web process and one handler process behind an nginx reverse proxy, all running on a single machine. Jobs are submitted to our slurm cluster via the SSH CLI runner. Traffic is pretty low -- we're just now rolling out of beta. Cheers, Brian -- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
Hello Brian, we have made some changes in the past month that might address the issue you are having. Are you still encountering problems? If so could you please try re-installing the tools again? thank you Martin On Tue, Feb 24, 2015 at 5:15 PM Brian Claywell <bclaywel@fredhutch.org> wrote:
Hi Martin,
Thanks for the prompt reply! My responses are inline below.
On Tue, Feb 24, 2015 at 8:26 AM, Martin Čech <marten@bx.psu.edu> wrote:
Are you using Admin UI for installation or API scripts?
The admin UI exclusively.
Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these?
Yes -- the iuc package_numpy_1_7 revision a4bdc17eed4a is listed twice, with both showing the green "Installed" status. Interestingly the installed repo page shows that they're installed in different locations:
(...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/ef12a3a11d5b/package_numpy_1_7 (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/a4bdc17eed4a/package_numpy_1_7
At one point during the troubleshooting steps I described in my original email, package_pybamparser_0_0_1 and package_pybamtools_0_0_1 showed up in the same way, with two records each having the same revision and a green "Installed" status.
If you check on the filesystem in the
{{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication?
I didn't see anything that struck me as out of the ordinary. Here's a gist of the directory structures for the affected packages under both shed_tools and tool_deps:
https://gist.github.com/bcclaywell/23c0b274f2d4bd0d659f
How big is your Galaxy in terms of installed tools, complexity of setup
and traffic?
I don't think it's terribly big -- we have a couple of large tool suites like BEDTools, the gops interval operations suites, GATK 1 and 2, etc, and then maybe 15-20 other tools and their dependencies?
Our setup isn't very complex, either; Galaxy is running with one web process and one handler process behind an nginx reverse proxy, all running on a single machine. Jobs are submitted to our slurm cluster via the SSH CLI runner.
Traffic is pretty low -- we're just now rolling out of beta.
Cheers,
Brian
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
Hi Martin, We haven't upgraded to the March release yet, and I haven't really touched the tools I referred to in my original email since I got them working (despite the duplicate installation records in the admin interface, etc). I've started testing the new release, though, and will take another look after we complete the upgrade. Thanks for following up! Cheers, Brian On Mon, Apr 13, 2015 at 7:27 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hello Brian,
we have made some changes in the past month that might address the issue you are having. Are you still encountering problems? If so could you please try re-installing the tools again?
thank you
Martin
On Tue, Feb 24, 2015 at 5:15 PM Brian Claywell <bclaywel@fredhutch.org> wrote:
Hi Martin,
Thanks for the prompt reply! My responses are inline below.
On Tue, Feb 24, 2015 at 8:26 AM, Martin Čech <marten@bx.psu.edu> wrote:
Are you using Admin UI for installation or API scripts?
The admin UI exclusively.
Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these?
Yes -- the iuc package_numpy_1_7 revision a4bdc17eed4a is listed twice, with both showing the green "Installed" status. Interestingly the installed repo page shows that they're installed in different locations:
(...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/ef12a3a11d5b/package_numpy_1_7 (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/a4bdc17eed4a/package_numpy_1_7
At one point during the troubleshooting steps I described in my original email, package_pybamparser_0_0_1 and package_pybamtools_0_0_1 showed up in the same way, with two records each having the same revision and a green "Installed" status.
If you check on the filesystem in the
{{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication?
I didn't see anything that struck me as out of the ordinary. Here's a gist of the directory structures for the affected packages under both shed_tools and tool_deps:
https://gist.github.com/bcclaywell/23c0b274f2d4bd0d659f
How big is your Galaxy in terms of installed tools, complexity of setup
and traffic?
I don't think it's terribly big -- we have a couple of large tool suites like BEDTools, the gops interval operations suites, GATK 1 and 2, etc, and then maybe 15-20 other tools and their dependencies?
Our setup isn't very complex, either; Galaxy is running with one web process and one handler process behind an nginx reverse proxy, all running on a single machine. Jobs are submitted to our slurm cluster via the SSH CLI runner.
Traffic is pretty low -- we're just now rolling out of beta.
Cheers,
Brian
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
Hi Martin, After we upgraded to the March release I was still seeing behavior similar to that I originally described back in February, even after uninstalling and reinstalling the affected repositories. On a separate instance using the 15.05 release and the new `install_database_connection` support, I've been working to duplicate all of our installed repositories from production using scripted API calls for reproducibility, and have again run into similar problems. What I've learned, though, is that the problem does not appear to be caused by installing a *single* repository, but rather some kind of interaction between multiple repositories. For example, against a brand new 15.05 instance, I ran: ``` $ python ./scripts/api/install_tool_shed_repositories.py --local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps --url https://toolshed.g2.bx.psu.edu --owner iuc --name samtools_sort --revision 38ea74bd4054 ``` followed by: ``` $ python ./scripts/api/install_tool_shed_repositories.py --local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps --url https://toolshed.g2.bx.psu.edu --owner lparsons --name htseq_count --revision 6f920f33c5eb ``` The first repository, samtools_sort, installed fine: ``` Response -------- /api/tool_shed_repositories/f2db41e1fa331b3e name: samtools_sort status: Installed tool_shed_status: {u'latest_installable_revision': u'True', u'revision_update': u'False', u'revision_upgrade': u'False', u'repository_deprecated': u'False'} deleted: False ctx_rev: 2 error_message: dist_to_shed: False tool_shed: toolshed.g2.bx.psu.edu installed_changeset_revision: 38ea74bd4054 uninstalled: False owner: iuc changeset_revision: 38ea74bd4054 id: f2db41e1fa331b3e includes_datatypes: False /api/tool_shed_repositories/f597429621d6eb2b name: package_samtools_0_1_19 status: Installed tool_shed_status: {u'latest_installable_revision': u'True', u'revision_update': u'False', u'revision_upgrade': u'False', u'repository_deprecated': u'False'} deleted: False ctx_rev: 1 error_message: dist_to_shed: False tool_shed: toolshed.g2.bx.psu.edu installed_changeset_revision: 95d2c4aefb5f uninstalled: False owner: devteam changeset_revision: 95d2c4aefb5f id: f597429621d6eb2b includes_datatypes: False ``` However, installation of htseq_count next failed and logged the following: ``` tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2015-06-20 02:31:33,466 Creating repository dependency objects... tool_shed.util.shed_util_common DEBUG 2015-06-20 02:31:34,552 Adding new row for repository 'htseq_count' in the tool_shed_repository table, status set to 'New'. tool_shed.util.shed_util_common DEBUG 2015-06-20 02:31:34,880 Adding new row for repository 'package_numpy_1_7' in the tool_shed_repository table, status set to 'New'. tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2015-06-20 02:31:34,892 Skipping installation of revision 95d2c4aefb5f of repository 'package_samtools_0_1_19' because it was installed with the (possibly updated) revision 95d2c4aefb5f and its current installation status is 'Installed'. tool_shed.util.shed_util_common DEBUG 2015-06-20 02:31:35,242 Adding new row for repository 'package_pysam_0_7_7' in the tool_shed_repository table, status set to 'New'. tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2015-06-20 02:31:35,255 Building repository dependency relationships... tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2015-06-20 02:31:35,266 Creating new repository_dependency record for installed revision 0c288abd2a1e of repository: package_numpy_1_7 owned by devteam. tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2015-06-20 02:31:35,332 Creating new repository_dependency record for installed revision b62538c8c664 of repository: package_pysam_0_7_7 owned by iuc. galaxy.web.framework.decorators ERROR 2015-06-20 02:31:35,387 Uncaught exception in exposed API method: Traceback (most recent call last): File "/mnt/scdata/scdata_03/galaxy/containers/galaxy-builder/stable/lib/galaxy/web/framework/decorators.py", line 251, in decorator rval = func( self, trans, *args, **kwargs) File "/mnt/scdata/scdata_03/galaxy/containers/galaxy-builder/stable/lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py", line 246, in install_repository_revision payload ) File "/mnt/scdata/scdata_03/galaxy/containers/galaxy-builder/stable/lib/tool_shed/galaxy_install/install_manager.py", line 709, in install install_options File "/mnt/scdata/scdata_03/galaxy/containers/galaxy-builder/stable/lib/tool_shed/galaxy_install/install_manager.py", line 801, in __initiate_and_install_repositories return self.install_repositories(tsr_ids, decoded_kwd, reinstalling=False) File "/mnt/scdata/scdata_03/galaxy/containers/galaxy-builder/stable/lib/tool_shed/galaxy_install/install_manager.py", line 844, in install_repositories reinstalling=reinstalling ) File "/mnt/scdata/scdata_03/galaxy/containers/galaxy-builder/stable/lib/tool_shed/galaxy_install/install_manager.py", line 864, in install_tool_shed_repository repo_info_tuple = repo_info_dict[ tool_shed_repository.name ] KeyError: u'package_pysam_0_7_7' ``` This left some of the dependency repositories in strange states, even after a restart: [image: Inline image 1] Attempting to `purge` the package_numpy_1_7 repository still listed as 'New' resulted in an error traceback: [image: Inline image 2] However, without installing samtools_sort first, htseq_count and its dependencies install fine. This is the only failing case I've reduced to the bare minimum from the ~110 tools I was trying to install. Before trying to reduce to a test case I provoked similar errors from at least three other repositories, so the problem doesn't seem to be limited to this combination. Cheers, Brian On Wed, Apr 15, 2015 at 2:48 PM, Brian Claywell <bclaywel@fredhutch.org> wrote:
Hi Martin,
We haven't upgraded to the March release yet, and I haven't really touched the tools I referred to in my original email since I got them working (despite the duplicate installation records in the admin interface, etc). I've started testing the new release, though, and will take another look after we complete the upgrade. Thanks for following up!
Cheers,
Brian
On Mon, Apr 13, 2015 at 7:27 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hello Brian,
we have made some changes in the past month that might address the issue you are having. Are you still encountering problems? If so could you please try re-installing the tools again?
thank you
Martin
On Tue, Feb 24, 2015 at 5:15 PM Brian Claywell <bclaywel@fredhutch.org> wrote:
Hi Martin,
Thanks for the prompt reply! My responses are inline below.
On Tue, Feb 24, 2015 at 8:26 AM, Martin Čech <marten@bx.psu.edu> wrote:
Are you using Admin UI for installation or API scripts?
The admin UI exclusively.
Does the 'installed tool shed repos' page show multiple installations of the same package (with the same changesets)? If so what are the states of these?
Yes -- the iuc package_numpy_1_7 revision a4bdc17eed4a is listed twice, with both showing the green "Installed" status. Interestingly the installed repo page shows that they're installed in different locations:
(...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/ef12a3a11d5b/package_numpy_1_7 (...)/shed_tools/ toolshed.g2.bx.psu.edu/repos/iuc/package_numpy_1_7/a4bdc17eed4a/package_numpy_1_7
At one point during the troubleshooting steps I described in my original email, package_pybamparser_0_0_1 and package_pybamtools_0_0_1 showed up in the same way, with two records each having the same revision and a green "Installed" status.
If you check on the filesystem in the
{{configured_dependency_dir}}/path/to/duplicated/package do you see any strange stuff/duplication?
I didn't see anything that struck me as out of the ordinary. Here's a gist of the directory structures for the affected packages under both shed_tools and tool_deps:
https://gist.github.com/bcclaywell/23c0b274f2d4bd0d659f
How big is your Galaxy in terms of installed tools, complexity of setup
and traffic?
I don't think it's terribly big -- we have a couple of large tool suites like BEDTools, the gops interval operations suites, GATK 1 and 2, etc, and then maybe 15-20 other tools and their dependencies?
Our setup isn't very complex, either; Galaxy is running with one web process and one handler process behind an nginx reverse proxy, all running on a single machine. Jobs are submitted to our slurm cluster via the SSH CLI runner.
Traffic is pretty low -- we're just now rolling out of beta.
Cheers,
Brian
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
-- Brian Claywell | programmer/analyst Matsen Group | http://matsen.fredhutch.org <http://matsen.fhcrc.org/> Fred Hutchinson Cancer Research Center
participants (2)
-
Brian Claywell -
Martin Čech