ImportError and Project life cycle
Dear galaxy-dev list, I'm currently trying to introduce myself to the galaxy server, i.e., learning how to use and administer a galaxy server. I have installed galaxy following this excellent tutorial: https://github.com/martenson/dagobah-training Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I get a python error (the Admin panel shows that all dependencies are installed): Traceback (most recent call last): File "/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py", line 3, in <module> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in <module> from six import Iterator, string_types ImportError: No module named six Do you have an idea what could be wrong? I guess that this is to little information, but I did not know which kind of additional information would be helpful. Another question: At my institution quite a lot of NGS projects are carried out. After a project is finished all methods and data are stored to a long term read only tape archive. For the future the idea is to use galaxy. Which options are there to implement such a project life cycle with galaxy? A last general question: would it be better to send a separate mail to the list when I have multiple questions? Best regards, Matthias Bernt
On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt <m.bernt@ufz.de> wrote:
Dear galaxy-dev list,
I'm currently trying to introduce myself to the galaxy server, i.e., learning how to use and administer a galaxy server.
I have installed galaxy following this excellent tutorial:
https://github.com/martenson/dagobah-training
Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I get a python error (the Admin panel shows that all dependencies are installed):
Traceback (most recent call last): File "/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py", line 3, in <module> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in <module> from six import Iterator, string_types ImportError: No module named six
Do you have an idea what could be wrong? I guess that this is to little information, but I did not know which kind of additional information would be helpful.
This appears to be a missing dependency on the Python module "six" which is used to help write code which will work under both Python 2 and Python 3. This is likely a recent regression - hopefully one of the Galaxy team can comment on this, likely the Tool Shed definition for this tool needs a minor tweak (and then you can apply the update via the Galaxy Admin controls within your browser).
Another question: At my institution quite a lot of NGS projects are carried out. After a project is finished all methods and data are stored to a long term read only tape archive. For the future the idea is to use galaxy. Which options are there to implement such a project life cycle with galaxy?
Not that I personally am aware of, no. There are optional disk quotas to encourage users to remove (delete) old data.
A last general question: would it be better to send a separate mail to the list when I have multiple questions?
Yes please, with useful email subjects would be best. Peter
Hi Matthias, what version of Galaxy are you running? The tool you have installed seems to support only new Galaxies that have Conda dependency resolution enabled. We are currently in the middle of the transition of the Galaxy's ecosystem to this new dependency manager so there are rough edges here and there - sorry about that. You can read details about Conda and dependency resolution here: https://docs.galaxyproject.org/en/master/admin/conda_faq.html Since Galaxy 16.07 you can switch the `conda_auto_init` in Galaxy's config to True and after restart Galaxy will install Conda itself. Therefore I recommend you (with Galaxy newer than 16.07) to uninstall the tool, install Conda (with conda_auto_init) and install the tool again. That will install the needed dependency for the tool. If you have older Galaxy please update it. Please let us know about any questions you have and thanks for using Galaxy. Martin On Fri, Feb 10, 2017 at 10:38 AM Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt <m.bernt@ufz.de> wrote:
Dear galaxy-dev list,
I'm currently trying to introduce myself to the galaxy server, i.e., learning how to use and administer a galaxy server.
I have installed galaxy following this excellent tutorial:
https://github.com/martenson/dagobah-training
Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I get a python error (the Admin panel shows that all dependencies are installed):
Traceback (most recent call last): File "/home/berntm/shed_tools/ toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py ", line 3, in <module> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in <module> from six import Iterator, string_types ImportError: No module named six
Do you have an idea what could be wrong? I guess that this is to little information, but I did not know which kind of additional information would be helpful.
This appears to be a missing dependency on the Python module "six" which is used to help write code which will work under both Python 2 and Python 3.
This is likely a recent regression - hopefully one of the Galaxy team can comment on this, likely the Tool Shed definition for this tool needs a minor tweak (and then you can apply the update via the Galaxy Admin controls within your browser).
Another question: At my institution quite a lot of NGS projects are carried out. After a project is finished all methods and data are stored to a long term read only tape archive. For the future the idea is to use galaxy. Which options are there to implement such a project life cycle with galaxy?
Not that I personally am aware of, no. There are optional disk quotas to encourage users to remove (delete) old data.
A last general question: would it be better to send a separate mail to the list when I have multiple questions?
Yes please, with useful email subjects would be best.
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Dear Martin, Thanks for your input. Before answering the raised questions I would like to add some more information that I just found: I've noticed something odd (to me) which is maybe the source of the problem. My galaxy installation has galaxy_utils installed twice: - lib/galaxy_utils/sequence/fastq.py - database/dependencies/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/8882f14715b5/lib/python/galaxy_sequence_utils-1.0.0-py2.7.egg/galaxy_utils/sequence/fastq.py The two files have different content. Only the former imports the six module and the former is used by fastq_trimmer_by_quality (as you see in the python stack trace). The former is already installed in a fresh clone of galaxy (I have checked for 16.07 and 16.10). The Dependency Resolver Details show that fastq_trimmer_by_quality requires galaxy_sequence_utils 1.0.1, but I can not find this version in my galaxy installation after installing fastq_trimmer_by_quality (find . -iname "galaxy_sequence_utils"). If I install galaxy_sequence_utils 1.0.1 manually (only available from TS) the 1.0.0 version in the database/dependencies/galaxy_sequence_utils is replaced by 1.0.1 (which seems also odd to me - shouldn't both versions be available?). Anyway, it makes no difference to the python import error if this version is installed or not, since still the version in lib/ is used.
what version of Galaxy are you running?
I am using galaxy 16.10.
The tool you have installed seems to support only new Galaxies that have Conda dependency resolution enabled. We are currently in the middle of the transition of the Galaxy's ecosystem to this new dependency manager so there are rough edges here and there - sorry about that. You can read details about Conda and dependency resolution here: https://docs.galaxyproject.org/en/master/admin/conda_faq.html
Since Galaxy 16.07 you can switch the `conda_auto_init` in Galaxy's config to True and after restart Galaxy will install Conda itself.
I'm using the default dependency_resolvers_conf.xml (ie I don't have the file). Furthermore I have set: conda_auto_install = True conda_auto_init = True conda_copy_dependencies = True
Therefore I recommend you (with Galaxy newer than 16.07) to uninstall the tool, install Conda (with conda_auto_init) and install the tool again. That will install the needed dependency for the tool. If you have older Galaxy please update it.
I did uncheck the checkbox for TS dependencies during the tool installation. I was wondering that galaxy_sequence_utils is not shown as installed in "Manage installed tools". It seems to make no difference if I install it manually or not. Cheers, Matthias
Please let us know about any questions you have and thanks for using Galaxy.
Martin
On Fri, Feb 10, 2017 at 10:38 AM Peter Cock <p.j.a.cock@googlemail.com <mailto:p.j.a.cock@googlemail.com>> wrote:
On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt <m.bernt@ufz.de <mailto:m.bernt@ufz.de>> wrote: > Dear galaxy-dev list, > > I'm currently trying to introduce myself to the galaxy server, i.e., > learning how to use and administer a galaxy server. > > I have installed galaxy following this excellent tutorial: > > https://github.com/martenson/dagobah-training > > Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is > mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I > get a python error (the Admin panel shows that all dependencies are > installed): > > Traceback (most recent call last): > File > "/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py <http://toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py>", > line 3, in <module> > from galaxy_utils.sequence.fastq import fastqReader, fastqWriter > File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in > <module> > from six import Iterator, string_types > ImportError: No module named six > > Do you have an idea what could be wrong? I guess that this is to little > information, but I did not know which kind of additional information would > be helpful.
This appears to be a missing dependency on the Python module "six" which is used to help write code which will work under both Python 2 and Python 3.
This is likely a recent regression - hopefully one of the Galaxy team can comment on this, likely the Tool Shed definition for this tool needs a minor tweak (and then you can apply the update via the Galaxy Admin controls within your browser).
> Another question: At my institution quite a lot of NGS projects are carried > out. After a project is finished all methods and data are stored to a long > term read only tape archive. For the future the idea is to use galaxy. Which > options are there to implement such a project life cycle with galaxy?
Not that I personally am aware of, no. There are optional disk quotas to encourage users to remove (delete) old data.
> A last general question: would it be better to send a separate mail to the > list when I have multiple questions?
Yes please, with useful email subjects would be best.
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Mon, Feb 13, 2017 at 9:17 AM, Matthias Bernt <m.bernt@ufz.de> wrote:
Dear Martin,
Thanks for your input. Before answering the raised questions I would like to add some more information that I just found:
I've noticed something odd (to me) which is maybe the source of the problem. My galaxy installation has galaxy_utils installed twice:
- lib/galaxy_utils/sequence/fastq.py
- database/dependencies/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/8882f14715b5/lib/python/galaxy_sequence_utils-1.0.0-py2.7.egg/galaxy_utils/sequence/fastq.py
Something like that is to be expected right now as this is in transition, see also: http://dev.list.galaxyproject.org/Tools-using-Galaxy-s-Python-library-td4670... In the next release of Galaxy, there will not be a copy of the code at lib/galaxy_utils/sequence/fastq.py - see this pull request for the change https://github.com/galaxyproject/galaxy/pull/3551 Originally tools were allowed to access Galaxy's internal Python code, here lib/galaxy_utils/sequence/fastq.py - but that is changing. In the next Galaxy release tools should explicitly depend on the new packaged version of Galaxy's sequence_utils - that's where the second copy you've seen comes from: https://github.com/galaxyproject/sequence_utils This does now import six as part of work to make sure it can be used on Python 2 and Python 3: https://github.com/galaxyproject/sequence_utils/blob/master/galaxy_utils/seq... Galaxy itself will depend on the now separate sequence_utils package. -- As to what is going wrong in your situation, I'm hoping one of the Galaxy developers working on this can help. Peter
Hi Matthias,
I did uncheck the checkbox for TS dependencies during the tool installation. I was wondering that galaxy_sequence_utils is not shown as installed in "Manage installed tools". It seems to make no difference if I install it manually or not.
I recommend leaving the box checked so Galaxy will use Conda to install the dependency. Is the tool still missing imports after you install it this way? Currently 'Managed installed repositories' does _not_ show Conda-installed dependencies - the UI is lacking behind the system's capabilities - our apologies. Martin On Mon, Feb 13, 2017 at 4:17 AM Matthias Bernt <m.bernt@ufz.de> wrote:
Dear Martin,
Thanks for your input. Before answering the raised questions I would like to add some more information that I just found:
I've noticed something odd (to me) which is maybe the source of the problem. My galaxy installation has galaxy_utils installed twice:
- lib/galaxy_utils/sequence/fastq.py
-
database/dependencies/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/8882f14715b5/lib/python/galaxy_sequence_utils-1.0.0-py2.7.egg/galaxy_utils/sequence/fastq.py
The two files have different content. Only the former imports the six module and the former is used by fastq_trimmer_by_quality (as you see in the python stack trace).
The former is already installed in a fresh clone of galaxy (I have checked for 16.07 and 16.10).
The Dependency Resolver Details show that fastq_trimmer_by_quality requires galaxy_sequence_utils 1.0.1, but I can not find this version in my galaxy installation after installing fastq_trimmer_by_quality (find . -iname "galaxy_sequence_utils").
If I install galaxy_sequence_utils 1.0.1 manually (only available from TS) the 1.0.0 version in the database/dependencies/galaxy_sequence_utils is replaced by 1.0.1 (which seems also odd to me - shouldn't both versions be available?).
Anyway, it makes no difference to the python import error if this version is installed or not, since still the version in lib/ is used.
what version of Galaxy are you running?
I am using galaxy 16.10.
The tool you have installed seems to support only new Galaxies that have Conda dependency resolution enabled. We are currently in the middle of the transition of the Galaxy's ecosystem to this new dependency manager so there are rough edges here and there - sorry about that. You can read details about Conda and dependency resolution here: https://docs.galaxyproject.org/en/master/admin/conda_faq.html
Since Galaxy 16.07 you can switch the `conda_auto_init` in Galaxy's config to True and after restart Galaxy will install Conda itself.
I'm using the default dependency_resolvers_conf.xml (ie I don't have the file). Furthermore I have set:
conda_auto_install = True conda_auto_init = True conda_copy_dependencies = True
Therefore I recommend you (with Galaxy newer than 16.07) to uninstall the tool, install Conda (with conda_auto_init) and install the tool again. That will install the needed dependency for the tool. If you have older Galaxy please update it.
I did uncheck the checkbox for TS dependencies during the tool installation. I was wondering that galaxy_sequence_utils is not shown as installed in "Manage installed tools". It seems to make no difference if I install it manually or not.
Cheers, Matthias
Please let us know about any questions you have and thanks for using Galaxy.
Martin
On Fri, Feb 10, 2017 at 10:38 AM Peter Cock <p.j.a.cock@googlemail.com <mailto:p.j.a.cock@googlemail.com>> wrote:
On Fri, Feb 10, 2017 at 3:26 PM, Matthias Bernt <m.bernt@ufz.de <mailto:m.bernt@ufz.de>> wrote: > Dear galaxy-dev list, > > I'm currently trying to introduce myself to the galaxy server, i.e., > learning how to use and administer a galaxy server. > > I have installed galaxy following this excellent tutorial: > > https://github.com/martenson/dagobah-training > > Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is > mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I > get a python error (the Admin panel shows that all dependencies are > installed): > > Traceback (most recent call last): > File > "/home/berntm/shed_tools/ toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py < http://toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c2437... ", > line 3, in <module> > from galaxy_utils.sequence.fastq import fastqReader, fastqWriter > File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in > <module> > from six import Iterator, string_types > ImportError: No module named six > > Do you have an idea what could be wrong? I guess that this is to little > information, but I did not know which kind of additional information would > be helpful.
This appears to be a missing dependency on the Python module "six" which is used to help write code which will work under both Python 2 and Python 3.
This is likely a recent regression - hopefully one of the Galaxy team can comment on this, likely the Tool Shed definition for this tool needs a minor tweak (and then you can apply the update via the Galaxy Admin controls within your browser).
> Another question: At my institution quite a lot of NGS projects are carried > out. After a project is finished all methods and data are stored to a long > term read only tape archive. For the future the idea is to use galaxy. Which > options are there to implement such a project life cycle with galaxy?
Not that I personally am aware of, no. There are optional disk quotas to encourage users to remove (delete) old data.
> A last general question: would it be better to send a separate mail to the > list when I have multiple questions?
Yes please, with useful email subjects would be best.
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Martin, First of all: Since I have upgraded from 16.10 to 17.10 the problem disappeared :)
I did uncheck the checkbox for TS dependencies during the tool installation. I was wondering that galaxy_sequence_utils is not shown as installed in "Manage installed tools". It seems to make no difference if I install it manually or not.
I recommend leaving the box checked so Galaxy will use Conda to install the dependency.
OK.
Is the tool still missing imports after you install it this way?
No, its shown in the list of installed tools in version 1 of the tool. Version 2 seems to solely rely on conda.
Currently 'Managed installed repositories' does _not_ show Conda-installed dependencies - the UI is lacking behind the system's capabilities - our apologies.
Thanks a lot for the explanations. Cheers, Matthias
Hi Matthias, to your life cycle question, do you have more information about your filesystem(s) and what you have specifically in mind? Everything is possible I guess, from a hack to a advanced solution like iRODS. My feeling is that this should be implemented on the FS level and not on the Galaxy level. Ciao, Bjoern
Dear galaxy-dev list,
I'm currently trying to introduce myself to the galaxy server, i.e., learning how to use and administer a galaxy server.
I have installed galaxy following this excellent tutorial:
https://github.com/martenson/dagobah-training
Then I tried the tool "FASTQ Quality Trimmer by sliding window" which is mentioned in NGS101-6 tutorial. I installed the tool from tool shed, but I get a python error (the Admin panel shows that all dependencies are installed):
Traceback (most recent call last): File "/home/berntm/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.py", line 3, in <module> from galaxy_utils.sequence.fastq import fastqReader, fastqWriter File "/home/berntm/galaxy/lib/galaxy_utils/sequence/fastq.py", line 7, in <module> from six import Iterator, string_types ImportError: No module named six
Do you have an idea what could be wrong? I guess that this is to little information, but I did not know which kind of additional information would be helpful.
Another question: At my institution quite a lot of NGS projects are carried out. After a project is finished all methods and data are stored to a long term read only tape archive. For the future the idea is to use galaxy. Which options are there to implement such a project life cycle with galaxy?
A last general question: would it be better to send a separate mail to the list when I have multiple questions?
Best regards, Matthias Bernt
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (4)
-
Björn Grüning -
Martin Čech -
Matthias Bernt -
Peter Cock