recurring issues with revisions of packages from toolshed
Dear all, I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency. Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation. I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside. Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed". Best regards, Sarah ---- Sarah Diehl HPC System Administrator UNIVERSITÉ DU LUXEMBOURG LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
Note these are personal views of another admin not official Galaxy views. The version issues is a hard one to solve, as automatically updating upstream project is also dangerous as that leaves them in a state that has not be verified by the developer. Best solution could be for the toolshed to warn if a tool upload would pull in multiple versions. As for the R install issue. 1. I too find it extremely frustrating how often Galaxy just silently fails. 2. It could be due to a missing R package. The galaxy tool to determine required R packages appears to miss ones only listed as linked in. For example I had one which depends on lme4. This failed to build because RcppEigen was missing Yet it RccpEigen is only listed as "LinkingTo:" Christian ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Sarah DIEHL [sarah.diehl@uni.lu] Sent: Wednesday, October 14, 2015 2:05 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed Dear all, I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency. Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation. I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside. Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed". Best regards, Sarah ---- Sarah Diehl HPC System Administrator UNIVERSITÉ DU LUXEMBOURG LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
Am 15.10.2015 um 11:43 schrieb Christian Brenninkmeijer:
Note these are personal views of another admin not official Galaxy views.
The version issues is a hard one to solve, as automatically updating upstream project is also dangerous as that leaves them in a state that has not be verified by the developer.
Best solution could be for the toolshed to warn if a tool upload would pull in multiple versions.
Can you elaborate there a little bit more?
As for the R install issue. 1. I too find it extremely frustrating how often Galaxy just silently fails.
We are working on this. https://github.com/bgruening/galaxy/pull/8 But it is really hard to convince R to provide us with a real error code. I spend several days, but different R versions, fail differently or not at all .. I will keep you posted on this and hopefully the latest idea from us works out. Sorry for any inconvenience, Bjoern
2. It could be due to a missing R package. The galaxy tool to determine required R packages appears to miss ones only listed as linked in. For example I had one which depends on lme4. This failed to build because RcppEigen was missing Yet it RccpEigen is only listed as "LinkingTo:"
Christian ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Sarah DIEHL [sarah.diehl@uni.lu] Sent: Wednesday, October 14, 2015 2:05 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Regarding package dependency. Assume there packages A, B and C B depends on A and C on A and B At the Start there is A0, B0 and C0 B0 depends on A0 and C0 on A0 and B0 Now if A is updated there is a A1 But B0 still depends on A0 and C0 still on A0 and B0 Now if C is updated you have C1 C1 detects the latest versions of A and B so depends on A1 and B0 But B0 still depends on A0 The result is that C1 directly depends on A1 and indirectly (via B0) on A0 Could the tool shed detect this double dependency and refuse to load C1 unless it specifically references A0 and B0 === Regard R I agree that as a wrapper it Galaxy is highly dependent on what the underlying tools report. And if they change what they report that is a disaster. What I do in my testing is after all the R is loaded try to run an R script which loads the packages as a test. Something like: library(Cairo) args<-commandArgs(TRUE) writeLines(capture.output(sessionInfo()), args[1]) sessionInfo() But even then the tool goes green with an R error, Christian ________________________________________ From: Björn Grüning [bjoern.gruening@gmail.com] Sent: Thursday, October 15, 2015 10:48 AM To: Christian Brenninkmeijer; Sarah DIEHL; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] recurring issues with revisions of packages from toolshed Am 15.10.2015 um 11:43 schrieb Christian Brenninkmeijer:
Note these are personal views of another admin not official Galaxy views.
The version issues is a hard one to solve, as automatically updating upstream project is also dangerous as that leaves them in a state that has not be verified by the developer.
Best solution could be for the toolshed to warn if a tool upload would pull in multiple versions.
Can you elaborate there a little bit more?
As for the R install issue. 1. I too find it extremely frustrating how often Galaxy just silently fails.
We are working on this. https://github.com/bgruening/galaxy/pull/8 But it is really hard to convince R to provide us with a real error code. I spend several days, but different R versions, fail differently or not at all .. I will keep you posted on this and hopefully the latest idea from us works out. Sorry for any inconvenience, Bjoern
2. It could be due to a missing R package. The galaxy tool to determine required R packages appears to miss ones only listed as linked in. For example I had one which depends on lme4. This failed to build because RcppEigen was missing Yet it RccpEigen is only listed as "LinkingTo:"
Christian ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Sarah DIEHL [sarah.diehl@uni.lu] Sent: Wednesday, October 14, 2015 2:05 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Am 15.10.2015 um 13:20 schrieb Christian Brenninkmeijer:
Regarding package dependency. Assume there packages A, B and C B depends on A and C on A and B
At the Start there is A0, B0 and C0 B0 depends on A0 and C0 on A0 and B0
Now if A is updated there is a A1 But B0 still depends on A0 and C0 still on A0 and B0
Now if C is updated you have C1 C1 detects the latest versions of A and B so depends on A1 and B0 But B0 still depends on A0
The result is that C1 directly depends on A1 and indirectly (via B0) on A0
Could the tool shed detect this double dependency and refuse to load C1 unless it specifically references A0 and B0
As far as I know (@Dave, please correct me if I'm wrong) A0 does not exists for Galaxy as soon as you have A1 installed. Galaxy should see during runtime that A is a "package", a special repository only containing tool dependencies, and should only take the most recent version. Imho a different behaviour is a bug. And I actually don't know why A0 is not deleted as soon as A1 is installed successfully. Revisions as Sarah has described in her mail do not exist for tool-dependency packages, afaik.
=== Regard R I agree that as a wrapper it Galaxy is highly dependent on what the underlying tools report.
And if they change what they report that is a disaster.
What I do in my testing is after all the R is loaded try to run an R script which loads the packages as a test.
Something like: library(Cairo) args<-commandArgs(TRUE) writeLines(capture.output(sessionInfo()), args[1]) sessionInfo()
But even then the tool goes green with an R error,
Christian
________________________________________ From: Björn Grüning [bjoern.gruening@gmail.com] Sent: Thursday, October 15, 2015 10:48 AM To: Christian Brenninkmeijer; Sarah DIEHL; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] recurring issues with revisions of packages from toolshed
Am 15.10.2015 um 11:43 schrieb Christian Brenninkmeijer:
Note these are personal views of another admin not official Galaxy views.
The version issues is a hard one to solve, as automatically updating upstream project is also dangerous as that leaves them in a state that has not be verified by the developer.
Best solution could be for the toolshed to warn if a tool upload would pull in multiple versions.
Can you elaborate there a little bit more?
As for the R install issue. 1. I too find it extremely frustrating how often Galaxy just silently fails.
We are working on this.
https://github.com/bgruening/galaxy/pull/8
But it is really hard to convince R to provide us with a real error code. I spend several days, but different R versions, fail differently or not at all ..
I will keep you posted on this and hopefully the latest idea from us works out.
Sorry for any inconvenience, Bjoern
2. It could be due to a missing R package. The galaxy tool to determine required R packages appears to miss ones only listed as linked in. For example I had one which depends on lme4. This failed to build because RcppEigen was missing Yet it RccpEigen is only listed as "LinkingTo:"
Christian ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Sarah DIEHL [sarah.diehl@uni.lu] Sent: Wednesday, October 14, 2015 2:05 PM To: galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Just to avoid confusion, I would like to explain how I understand the meaning of "revisions" and "versions". Revisions are what shows up the in tool list as the orange triangle with a "!". You can simply update the tool and it's replaced by the newer revision. So revisions should be mostly bug fixes and such, which don't change the behaviour of the tool. Version changes show up with the red up arrow and if you install a new version of a tool it will not replace the older one, but both will be active. So a version change usually means an update of the underlying program and other changes that influence behaviour and/or output of the tool. What I was talking about in my mail were different revisions. Since those are not supposed to change the behaviour or the version of the program/package behind, I don't see why always using the latest revision should be an issue. ---- Sarah Diehl HPC System Administrator UNIVERSITÉ DU LUXEMBOURG LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. ----- From: Christian Brenninkmeijer <christian.brenninkmeijer@manchester.ac.uk<mailto:christian.brenninkmeijer@manchester.ac.uk>> Date: Thursday 15 October 2015 11:43 To: Sarah DIEHL <sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" <galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>> Subject: RE: [galaxy-dev] recurring issues with revisions of packages from toolshed Note these are personal views of another admin not official Galaxy views. The version issues is a hard one to solve, as automatically updating upstream project is also dangerous as that leaves them in a state that has not be verified by the developer. Best solution could be for the toolshed to warn if a tool upload would pull in multiple versions. As for the R install issue. 1. I too find it extremely frustrating how often Galaxy just silently fails. 2. It could be due to a missing R package. The galaxy tool to determine required R packages appears to miss ones only listed as linked in. For example I had one which depends on lme4. This failed to build because RcppEigen was missing Yet it RccpEigen is only listed as "LinkingTo:" Christian ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org<mailto:galaxy-dev-bounces@lists.galaxyproject.org>] on behalf of Sarah DIEHL [sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu>] Sent: Wednesday, October 14, 2015 2:05 PM To: galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed Dear all, I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency. Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation. I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside. Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed". Best regards, Sarah ---- Sarah Diehl HPC System Administrator UNIVERSITÉ DU LUXEMBOURG LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
Hi Sarah, as far as I can tell this is exactly how it should work. Thanks for clarifying! Bjoern
Just to avoid confusion, I would like to explain how I understand the meaning of "revisions" and "versions".
Revisions are what shows up the in tool list as the orange triangle with a "!". You can simply update the tool and it's replaced by the newer revision. So revisions should be mostly bug fixes and such, which don't change the behaviour of the tool. Version changes show up with the red up arrow and if you install a new version of a tool it will not replace the older one, but both will be active. So a version change usually means an update of the underlying program and other changes that influence behaviour and/or output of the tool.
What I was talking about in my mail were different revisions. Since those are not supposed to change the behaviour or the version of the program/package behind, I don't see why always using the latest revision should be an issue.
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
From: Christian Brenninkmeijer <christian.brenninkmeijer@manchester.ac.uk<mailto:christian.brenninkmeijer@manchester.ac.uk>>
Date: Thursday 15 October 2015 11:43
To: Sarah DIEHL <sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" <galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: RE: [galaxy-dev] recurring issues with revisions of packages from toolshed
Note these are personal views of another admin not official Galaxy views.
The version issues is a hard one to solve, as automatically updating upstream project is also dangerous as that leaves them in a state that has not be verified by the developer.
Best solution could be for the toolshed to warn if a tool upload would pull in multiple versions.
As for the R install issue. 1. I too find it extremely frustrating how often Galaxy just silently fails.
2. It could be due to a missing R package. The galaxy tool to determine required R packages appears to miss ones only listed as linked in. For example I had one which depends on lme4. This failed to build because RcppEigen was missing Yet it RccpEigen is only listed as "LinkingTo:"
Christian ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org<mailto:galaxy-dev-bounces@lists.galaxyproject.org>] on behalf of Sarah DIEHL [sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu>] Sent: Wednesday, October 14, 2015 2:05 PM To: galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>
Subject: [galaxy-dev] recurring issues with revisions of packages from toolshed
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Sarah, can you create a reproducible example in a Docker container for me? Together with a friend I will have a look at this. But we need to short reproducible example :) Thanks, Bjoern
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Sorry to hijack the thread, but I think the issue that Sarah has described could be related to https://github.com/galaxyproject/galaxy/issues/667. (I know it's not exactly the same, but chances are we fix this, we automatically fix Sarah's issue.) There is also an example to reproduce https://github.com/galaxyproject/galaxy/issues/667#issuecomment-138878174 And I made a bit of work on this in Pull Request: https://github.com/galaxyproject/galaxy/pull/707 But unfortunately at the time this was a bit over my head. Bjoern, if you want to work on this, let me know, maybe I can give you a hand. Hope that's helpful, Marius On 19 October 2015 at 23:17, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi Sarah,
can you create a reproducible example in a Docker container for me? Together with a friend I will have a look at this. But we need to short reproducible example :)
Thanks, Bjoern
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi, I'm afraid it won't be easy to reproduce my problem with a clean galaxy install. I think it has to do with the history of the instance, a lot of back and forth on the tool installations and also lot's of failed installations (due to my environment or network issues). Investigating the issue Marius mentioned also seems worthwhile to me, it sounds like it could be the underlying bug. What might also help is a way to really purge all traces of a tool or tool dependency (and all it's versions) from the disk and the database. Currently the only way to solve my numpy revision issue would be to delete the whole tool install database (which I learned to put in a different db and I already resorted to that once in the beginning). I had a look at the database and it looks messed up, but I don't know enough about it to feel comfortable changing entries. I'm happy to send a DB dump along, if this is helpful for anyone. Best regards, Sarah ---- Sarah Diehl HPC System Administrator UNIVERSITÉ DU LUXEMBOURG LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. ----- From: Marius van den Beek <m.vandenbeek@gmail.com<mailto:m.vandenbeek@gmail.com>> Date: Tuesday 20 October 2015 09:53 To: Björn Grüning <bjoern.gruening@gmail.com<mailto:bjoern.gruening@gmail.com>> Cc: Sarah DIEHL <sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" <galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>> Subject: Re: [galaxy-dev] recurring issues with revisions of packages from toolshed Sorry to hijack the thread, but I think the issue that Sarah has described could be related to https://github.com/galaxyproject/galaxy/issues/667. (I know it's not exactly the same, but chances are we fix this, we automatically fix Sarah's issue.) There is also an example to reproduce https://github.com/galaxyproject/galaxy/issues/667#issuecomment-138878174 And I made a bit of work on this in Pull Request: https://github.com/galaxyproject/galaxy/pull/707 But unfortunately at the time this was a bit over my head. Bjoern, if you want to work on this, let me know, maybe I can give you a hand. Hope that's helpful, Marius On 19 October 2015 at 23:17, Björn Grüning <bjoern.gruening@gmail.com<mailto:bjoern.gruening@gmail.com>> wrote: Hi Sarah, can you create a reproducible example in a Docker container for me? Together with a friend I will have a look at this. But we need to short reproducible example :) Thanks, Bjoern
Dear all,
I continuously have several issues with tool dependency packages from the toolshed and their revisions. It looks like the tools I install have requirements for different revisions of the same package_xy repository. What I end up with quite often is having the same package installed in multiple revisions (see attached "galaxytools_multiple_revisions.jpg" screenshot). When I then update the older revision, it's still listet twice and additionally the tool that had the older one as a dependency now says it's missing a repository dependency.
Although this is already quite annoying and creates a mess within the tool list, I could live with it. However, sometimes it completely messes up revisions. In the second attached screenshot (numpy_revision_issue.jpg) you can see that this numpy repository thinks it is revision "300877695495", but it's actual location has revision "8b3a5c7061d8". This causes tools that depend on revision "300877695495" to think it's there, but when the dependency is resolved at runtime of the tool, numpy is not found (since it's location is in a different revision's folder) and the run fails. As you can also see from the screenshot, resetting the metadata doesn't do anything about this. Also reinstalling doesn't work, since the database keeps the faulty record and uses it upon reinstallation.
I think it's a problem that when I install tools that have outdated revisions of a package repository as a dependency, they don't recognize that a newer version is available (and already installed), but instead insist on installing the older revision, too. I don't understand why they even have this strict requirement on a specific version of a package repo, and not just the version of the tool inside.
Btw. I also noticed that the installation of R packages (e.g. in package_dexseq_1_14) silently fails. Some of the requirements and also the dexseq R package itself failed the installation, but Galaxy didn't recognize any error and it still showed up as "Installed".
Best regards, Sarah
---- Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu><mailto:sarah.diehl@uni.lu<mailto:sarah.diehl@uni.lu>> http://lcsb.uni.lu<http://lcsb.uni.lu/> ----- This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies. -----
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (4)
-
Björn Grüning -
Christian Brenninkmeijer -
Marius van den Beek -
Sarah DIEHL