Nah, definitely baby steps…

so, in the repo you point to there seems to be an error in the Dockerfile, the ENV line should use the var=value syntax to have more than one setting on one line (maybe that’s changed recently in docker).

with this I built a new docker image, when I run my first tool it takes an age while it’s installing the deps, then it crashes with:

Traceback (most recent call last): File "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module> from sqlalchemy.orm import clear_mappers ImportError: No module named sqlalchemy.orm

and the output:

discarding /galaxy-central/tool_deps/_conda/bin from PATH prepending /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to PATH discarding /galaxy-central/tool_deps/_conda/bin from PATH prepending /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to PATH

I’m guessing this is some kind of conflict between python versions in and out of conda environments? Surely sqlalchemy would be installed for Galaxy to work?

I’ll try to dig around this in the morning but if it rings a bell…

Steve — Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

On 25 Aug 2016, at 8:43 PM, Björn Grüning bjoern.gruening@gmail.com wrote:

Hi Steve,

you call this baby-steps? I think this is huge! :)

All what you are missing is to enable conda in Galaxy. Look at Gregs new flavour which is entirely Conda/Galaxy based.

You need to enable these env vars to make Galaxy conda enabled:

https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9

Hope this helps, Bjoern

Am 25.08.2016 um 12:32 schrieb Steve Cassidy:

...

Hi all, I’m making baby steps towards having a repeatable installation for my tools. But I’m now stuck, so help would be appreciated.

I have a tool that works and is in the test toolshed (alveoimport in Data Sources).

It depends on my python package which is now part of bioconda (pyalveo, version 0.6).

I can run my tool via planemo, which works I think because I have pyalveo installed in a local venv.

If I try to run the docker image (derived from bgruening/galaxy-stable but with the testtoolshed added) I am able to install my tool, but it doesn’t pick up the dependency, so it doesn’t work.

I tried running with planemo turning on conda dependency resolution (following https://pypi.python.org/pypi/planemo/):

planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .

It seems to have a go, but fails:

galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency shell command for dependency 'pyalveo' galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo version 0.6 galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned <galaxy.tools.deps.resolvers.NullDependency object at 0x105bef5d0> (isnull? True) galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned <galaxy.tools.deps.resolvers.NullDependency object at 0x105bef5d0> (isnull? True) galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve dependency on 'pyalveo', ignoring

So, what’s the easiest route to a Galaxy deployment with my tool installed. The Docker route would be best I think, but what do I have to add to bgruening/galaxy-stable to have conda find my dependencies.

Thanks in advance,

Steve

— Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy http://web.science.mq.edu.au/%7Ecassidy

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Hi Steve,

Looks like you're running an older version of galaxy in the docker container, since newer galaxy will build the metadata environment in a separate environment called 'conda-metadata-env',

and we have also changed the way that the handlers receive their Python environment (that's why sqlalachemy couldn't be found).

Can you try updating the container or building from the dev branch? I think the necessary changes for conda should be in the dev branch, which you can pull with docker pull quay.io/bgruening/galaxy:dev.

Best,

Marius

On Aug 26, 2016 6:49 AM, "Steve Cassidy" steve.cassidy@mq.edu.au wrote:

Ok, probing further trying to understand this. It looks like the job runner is trying to call set_meta() from galaxy_ext/metadata/set_metadata.py. In there there’s a line that tries to set up the PATH:

# insert *this* galaxy before all others on sys.path sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ ), os.pardir, os.pardir ) ) )

In my case the result is:

['/export/galaxy-central/database/job_working_directory/000/3’, '/galaxy-central/lib’, '/galaxy-central/lib’, '/export/galaxy-central/database/job_working_directory/000/ 3/conda-env/lib/python27.zip’, '/export/galaxy-central/database/job_working_directory/000/ 3/conda-env/lib/python2.7’, '/export/galaxy-central/database/job_working_directory/000/ 3/conda-env/lib/python2.7/plat-linux2’, '/export/galaxy-central/database/job_working_directory/000/ 3/conda-env/lib/python2.7/lib-tk’, '/export/galaxy-central/database/job_working_directory/000/3 /conda-env/lib/python2.7/lib-old’, '/export/galaxy-central/database/job_working_directory/000/ 3/conda-env/lib/python2.7/lib-dynload’, '/export/galaxy-central/database/job_working_directory/000/3 /conda-env/lib/python2.7/site-packages’, '/export/galaxy-central/database/job_working_directory/000/3 /conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’ ]

Looking for sqlalchemy, I find it installed at:

/galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy

so clearly that line of code is not doing what it should. I’m guessing that the Conda resolver is running the script within a conda env and the BETA magic hasn’t quite made it to the right place yet…

BTW I was trying to understand the flags you mentioned but I can’t find GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in sample Dockerfiles - is this some kind of magic incantation that creates a rift in space-time…enquiring minds want to know!!!

Steve — Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

...

On 25 Aug 2016, at 9:54 PM, Steve Cassidy steve.cassidy@mq.edu.au

wrote:

...

Nah, definitely baby steps…

so, in the repo you point to there seems to be an error in the

Dockerfile, the ENV line should use the var=value syntax to have more than one setting on one line (maybe that’s changed recently in docker).

...

with this I built a new docker image, when I run my first tool it takes

an age while it’s installing the deps, then it crashes with:

...

Traceback (most recent call last): File "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",

line 1, in <module>

...

from galaxy_ext.metadata.set_metadata import set_metadata;

set_metadata()

...

File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line

23, in <module>

...

from sqlalchemy.orm import clear_mappers ImportError: No module named sqlalchemy.orm

and the output:

discarding /galaxy-central/tool_deps/_conda/bin from PATH prepending /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin

to PATH

...

discarding /galaxy-central/tool_deps/_conda/bin from PATH prepending /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin

to PATH

...

I’m guessing this is some kind of conflict between python versions in

and out of conda environments? Surely sqlalchemy would be installed for Galaxy to work?

...

I’ll try to dig around this in the morning but if it rings a bell…

Steve — Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

...

On 25 Aug 2016, at 8:43 PM, Björn Grüning bjoern.gruening@gmail.com

wrote:

...

...

Hi Steve,

you call this baby-steps? I think this is huge! :)

All what you are missing is to enable conda in Galaxy. Look at Gregs new flavour which is entirely Conda/Galaxy based.

You need to enable these env vars to make Galaxy conda enabled:

https://github.com/gregvonkuster/docker-galaxy-csg/blob/mast

er/Dockerfile#L9

...

...

Hope this helps, Bjoern

Am 25.08.2016 um 12:32 schrieb Steve Cassidy:

...

Hi all, I’m making baby steps towards having a repeatable installation for my tools. But I’m now stuck, so help would be appreciated.

I have a tool that works and is in the test toolshed (alveoimport in Data Sources).

It depends on my python package which is now part of bioconda (pyalveo, version 0.6).

I can run my tool via planemo, which works I think because I have pyalveo installed in a local venv.

If I try to run the docker image (derived from bgruening/galaxy-stable but with the testtoolshed added) I am able to install my tool, but it doesn’t pick up the dependency, so it doesn’t work.

I tried running with planemo turning on conda dependency resolution (following https://pypi.python.org/pypi/planemo/):

planemo serve --galaxy_branch release_16.01

--conda_dependency_resolution .

...

...

...

It seems to have a go, but fails:

galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency shell command for dependency 'pyalveo' galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo version 0.6 galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned <galaxy.tools.deps.resolvers.NullDependency object at 0x105bef5d0> (isnull? True) galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned <galaxy.tools.deps.resolvers.NullDependency object at 0x105bef5d0> (isnull? True) galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve dependency on 'pyalveo', ignoring

So, what’s the easiest route to a Galaxy deployment with my tool installed. The Docker route would be best I think, but what do I have

to

...

...

...

add to bgruening/galaxy-stable to have conda find my dependencies.

Thanks in advance,

Steve

— Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy http://web.science.mq.edu.au/%7Ecassidy

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

Looks like I’m hitting this issue:

https://github.com/galaxyproject/galaxy/issues/2797

Steve

— Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

On 29 Aug 2016, at 11:09 AM, Steve Cassidy steve.cassidy@mq.edu.au wrote:

Thanks Marius, I tried a build based on the image you mentioned and get the following error:

/export/galaxy-central/database/job_working_directory/000/1/tool_script.sh: line 9: /tool_deps/_conda/bin/activate: No such file or directory /export/galaxy-central/database/job_working_directory/000/1/galaxy_1.sh: line 66: /tool_deps/_conda/bin/activate: No such file or directory

I tried again with the 16.07 release image but got the same result.

Checking the /tool_deps/_conda/bin/ directory, indeed there is no activate script, in fact it contains:

root@db343fbf7372:/galaxy-central# ls /tool_deps/_conda/bin/ 2to3 conda c_rehash easy_install-3.5 idle3.5 pip pydoc3 python python3.5 python3.5m python3-config pyvenv-3.5 tclsh8.5 wheel xz 2to3-3.5 conda-env easy_install idle3 openssl pydoc pydoc3.5 python3 python3.5-config python3.5m-config pyvenv sqlite3 unxz wish8.5

Any pointers appreciated.

Thanks,

Steve

— Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

...

On 26 Aug 2016, at 6:16 PM, Marius van den Beek m.vandenbeek@gmail.com wrote:

Hi Steve,

Looks like you're running an older version of galaxy in the docker container, since newer galaxy will build the metadata environment in a separate environment called 'conda-metadata-env',

and we have also changed the way that the handlers receive their Python environment (that's why sqlalachemy couldn't be found).

Can you try updating the container or building from the dev branch? I think the necessary changes for conda should be in the dev branch, which you can pull with docker pull quay.io/bgruening/galaxy:dev.

Best,

Marius

On Aug 26, 2016 6:49 AM, "Steve Cassidy" steve.cassidy@mq.edu.au wrote: Ok, probing further trying to understand this. It looks like the job runner is trying to call set_meta() from galaxy_ext/metadata/set_metadata.py. In there there’s a line that tries to set up the PATH:

# insert *this* galaxy before all others on sys.path sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ ), os.pardir, os.pardir ) ) )

In my case the result is:

['/export/galaxy-central/database/job_working_directory/000/3’, '/galaxy-central/lib’, '/galaxy-central/lib’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python27.zip’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/plat-linux2’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-tk’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-old’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-dynload’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages’, '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’ ]

Looking for sqlalchemy, I find it installed at:

/galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy

so clearly that line of code is not doing what it should. I’m guessing that the Conda resolver is running the script within a conda env and the BETA magic hasn’t quite made it to the right place yet…

BTW I was trying to understand the flags you mentioned but I can’t find GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in sample Dockerfiles - is this some kind of magic incantation that creates a rift in space-time…enquiring minds want to know!!!

Steve — Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

...

On 25 Aug 2016, at 9:54 PM, Steve Cassidy steve.cassidy@mq.edu.au wrote:

Nah, definitely baby steps…

so, in the repo you point to there seems to be an error in the Dockerfile, the ENV line should use the var=value syntax to have more than one setting on one line (maybe that’s changed recently in docker).

with this I built a new docker image, when I run my first tool it takes an age while it’s installing the deps, then it crashes with:

Traceback (most recent call last): File "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module> from sqlalchemy.orm import clear_mappers ImportError: No module named sqlalchemy.orm

and the output:

discarding /galaxy-central/tool_deps/_conda/bin from PATH prepending /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to PATH discarding /galaxy-central/tool_deps/_conda/bin from PATH prepending /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to PATH

I’m guessing this is some kind of conflict between python versions in and out of conda environments? Surely sqlalchemy would be installed for Galaxy to work?

I’ll try to dig around this in the morning but if it rings a bell…

Steve — Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy

...

On 25 Aug 2016, at 8:43 PM, Björn Grüning bjoern.gruening@gmail.com wrote:

Hi Steve,

you call this baby-steps? I think this is huge! :)

All what you are missing is to enable conda in Galaxy. Look at Gregs new flavour which is entirely Conda/Galaxy based.

You need to enable these env vars to make Galaxy conda enabled:

https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9

Hope this helps, Bjoern

Am 25.08.2016 um 12:32 schrieb Steve Cassidy:

...

Hi all, I’m making baby steps towards having a repeatable installation for my tools. But I’m now stuck, so help would be appreciated.

I have a tool that works and is in the test toolshed (alveoimport in Data Sources).

It depends on my python package which is now part of bioconda (pyalveo, version 0.6).

I can run my tool via planemo, which works I think because I have pyalveo installed in a local venv.

If I try to run the docker image (derived from bgruening/galaxy-stable but with the testtoolshed added) I am able to install my tool, but it doesn’t pick up the dependency, so it doesn’t work.

I tried running with planemo turning on conda dependency resolution (following https://pypi.python.org/pypi/planemo/):

planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .

It seems to have a go, but fails:

galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency shell command for dependency 'pyalveo' galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo version 0.6 galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned <galaxy.tools.deps.resolvers.NullDependency object at 0x105bef5d0> (isnull? True) galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned <galaxy.tools.deps.resolvers.NullDependency object at 0x105bef5d0> (isnull? True) galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve dependency on 'pyalveo', ignoring

So, what’s the easiest route to a Galaxy deployment with my tool installed. The Docker route would be best I think, but what do I have to add to bgruening/galaxy-stable to have conda find my dependencies.

Thanks in advance,

Steve

— Department of Computing, Macquarie University http://web.science.mq.edu.au/~cassidy http://web.science.mq.edu.au/%7Ecassidy

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

---

Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/