Ahhh - I get it now. Galaxy only adds itself to the PYTHONPATH if it can infer it is running a Python script (i.e. the command block starts with python or a Python interpreter is used).

If you are building a tool for the tool shed - it would be best to setup a tool dependency on the galaxy_sequence_utils package - this would force every execution of the tool to have these utilities on your PYTHONPATH. But since you are hard-encoding paths - it looks like you are tailoring your Galaxy instance to your local environment. In this case an easy hack is to just setup that dependency globally for the system:

sudo easy_install http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_s...

If you are setting up Galaxy env.sh files this can be installed into a virtualenv instead - which is a better practice - let me know if you are interested and I can send details.

-John

On Mon, Jul 7, 2014 at 4:02 AM, julie dubois dubjulie@gmail.com wrote:

Hi John,

As I specified it in the "edit" of my first message, I have this error only if I run groomer with a command line in another xml file surrounding by other command. When I run groomer alone with the original xml, it works fine, so no problem with the PYTHONPATH. But when I use this :

<command> #if $fastq_format == "illumina" blabla_No_python_command #else python /data/galaxy-dist/tools/fastq/fastq_groomer.py 'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii' 'dont_summarize_input' #end if </command>

It doesn't work.

I've checked the files contained in job_working_directory. A shell script is generated for my job and it begin with these lines :

#!/bin/sh

GALAXY_SLOTS="1"; export GALAXY_SLOTS; export GALAXY_SLOTS GALAXY_LIB="None" if [ "$GALAXY_LIB" != "None" ]; then if [ -n "$PYTHONPATH" ]; then PYTHONPATH="$GALAXY_LIB:$PYTHONPATH" else PYTHONPATH="$GALAXY_LIB" fi export PYTHONPATH fi

So, with the run of groomer alone the line GALAXY_LIB is equal to "/data/galaxy/galaxy-dist/lib", but here with the run of groomer with a command line with another job the GALAXY_LIB is set to "None"!!! I don't understand why.

For the moment, to surround the problem, I've explicitly specified the GALAXY_LIB path int the xml command before the command line wich run python fastq_groomer.py and it works but I'm aware that is not beautiful.

Finally I'm using a local jobrunner.

Thanks for your help Julie

P.S : we use such command because we build a workflow where the quality format of the fastq must be invisible for the users.

2014-07-03 19:48 GMT+02:00 John Chilton jmchilton@gmail.com:

...

Hello Julie,

This is odd - sorry.

I might be worth uninstalling and re-installing this in the tool shed admin options - in particular ensure you install tool dependencies and that this installation process succeeds.

Otherwise - it might be worth trying to put some debug statements in the fastq groomer tool to try to track down the problem. Here (https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to have that tool write out the run-time value PYTHONPATH to Galaxy's home directory - it might be worth running a job, finding the output, setting PYTHONPATH to that value on the compute node and attempting to just import it directly.

cd $HOME; python -c "import galaxy.sequences.fastq"

Out of curiosity - what job runner are you using (local, DRMAA, etc...)?

-John

On Tue, Jul 1, 2014 at 4:50 AM, julie dubois dubjulie@gmail.com wrote:

...

Hi, I've this error message when I run groomer.

My local instance of galaxy runs on a python virtualenv (python 2.7) and I have make the latest update with : hg pull -u.

I've verified and this library is present in my galaxy-dist/lib/galaxy_utils/sequence directory.

I've found no other idea than the update on this forum to run groomer correctly, any other idea ?

Thanks. Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

If your tool defines a package requirement like:

<requirements> <requirement type="package" version="1.0.0">galaxy-sequence-utils</requirement> </requirements>

And your tool_dependency_dir setting universe_wsgi.ini is something like /data/galaxy_deps - then you should be able to create a file /data/galaxy_deps/galaxy-sequence-utils/1.0.0/env.sh with contents like

. /data/galaxy_deps/galaxy-sequence-utils/1.0.0/venv/bin/activate

This will cause that tool's jobs to source that isolated environment before execution. You should then be able to setup this virtual environment using the following commands:

virtualenv /data/galaxy_deps/galaxy-sequence-utils/1.0.0/venv . /data/galaxy_deps/galaxy-sequence-utils/1.0.0/venv/bin/activate easy_install http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_s...

This may require installing some system Python packages such as python-virtualenv on Ubuntu.

Hope this helps.

-John

On Mon, Jul 7, 2014 at 9:31 AM, julie dubois dubjulie@gmail.com wrote:

Thanks!

Actually it's not for the toolshed, for the moment... But I'm very intereseted by details to install it into a virtualenv. Thank you.

Julie

2014-07-07 16:16 GMT+02:00 John Chilton jmchilton@gmail.com:

...

Ahhh - I get it now. Galaxy only adds itself to the PYTHONPATH if it can infer it is running a Python script (i.e. the command block starts with python or a Python interpreter is used).

If you are building a tool for the tool shed - it would be best to setup a tool dependency on the galaxy_sequence_utils package - this would force every execution of the tool to have these utilities on your PYTHONPATH. But since you are hard-encoding paths - it looks like you are tailoring your Galaxy instance to your local environment. In this case an easy hack is to just setup that dependency globally for the system:

sudo easy_install http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_s...

If you are setting up Galaxy env.sh files this can be installed into a virtualenv instead - which is a better practice - let me know if you are interested and I can send details.

-John

On Mon, Jul 7, 2014 at 4:02 AM, julie dubois dubjulie@gmail.com wrote:

...

Hi John,

As I specified it in the "edit" of my first message, I have this error only if I run groomer with a command line in another xml file surrounding by other command. When I run groomer alone with the original xml, it works fine, so no problem with the PYTHONPATH. But when I use this :

<command> #if $fastq_format == "illumina" blabla_No_python_command #else python /data/galaxy-dist/tools/fastq/fastq_groomer.py 'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii' 'dont_summarize_input' #end if </command>

It doesn't work.

I've checked the files contained in job_working_directory. A shell script is generated for my job and it begin with these lines :

#!/bin/sh

GALAXY_SLOTS="1"; export GALAXY_SLOTS; export GALAXY_SLOTS GALAXY_LIB="None" if [ "$GALAXY_LIB" != "None" ]; then if [ -n "$PYTHONPATH" ]; then PYTHONPATH="$GALAXY_LIB:$PYTHONPATH" else PYTHONPATH="$GALAXY_LIB" fi export PYTHONPATH fi

So, with the run of groomer alone the line GALAXY_LIB is equal to "/data/galaxy/galaxy-dist/lib", but here with the run of groomer with a command line with another job the GALAXY_LIB is set to "None"!!! I don't understand why.

For the moment, to surround the problem, I've explicitly specified the GALAXY_LIB path int the xml command before the command line wich run python fastq_groomer.py and it works but I'm aware that is not beautiful.

Finally I'm using a local jobrunner.

Thanks for your help Julie

P.S : we use such command because we build a workflow where the quality format of the fastq must be invisible for the users.

2014-07-03 19:48 GMT+02:00 John Chilton jmchilton@gmail.com:

...

Hello Julie,

This is odd - sorry.

I might be worth uninstalling and re-installing this in the tool shed admin options - in particular ensure you install tool dependencies and that this installation process succeeds.

Otherwise - it might be worth trying to put some debug statements in the fastq groomer tool to try to track down the problem. Here (https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to have that tool write out the run-time value PYTHONPATH to Galaxy's home directory - it might be worth running a job, finding the output, setting PYTHONPATH to that value on the compute node and attempting to just import it directly.

cd $HOME; python -c "import galaxy.sequences.fastq"

Out of curiosity - what job runner are you using (local, DRMAA, etc...)?

-John

On Tue, Jul 1, 2014 at 4:50 AM, julie dubois dubjulie@gmail.com wrote:

...

Hi, I've this error message when I run groomer.

My local instance of galaxy runs on a python virtualenv (python 2.7) and I have make the latest update with : hg pull -u.

I've verified and this library is present in my galaxy-dist/lib/galaxy_utils/sequence directory.

I've found no other idea than the update on this forum to run groomer correctly, any other idea ?

Thanks. Julie ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/