Hi Bjoern, thanks for the pointer ... We are running the default dependency_resolvers_conf.xml now and that, along with the symlinks hint has helped with a lot of the tool dep. issues.  But there is still a pathing issue we are stuck on.

Take for example the trim_galore tool.

The error is 

/home/.galaxy/database/jobs_directory/007/7480/tool_script.sh: line 25: trim_galore: command not found

And the script is:

#!/bin/bash

# The following block can be used by the job system
# to ensure this script is runnable before actually attempting
# to run it.
if [ -n "$ABC_TEST_JOB_SCRIPT_INTEGRITY_XYZ" ]; then
    exit 42
fi
[ "$CONDA_DEFAULT_ENV" = "/home/.galaxy/tooldeps/_conda/envs/__trim-galore@0.4.3" ] ||
MAX_TRIES=3
COUNT=0
while [ $COUNT -lt $MAX_TRIES ]; do
    . /home/.galaxy/tooldeps/_conda/bin/activate '/home/.galaxy/tooldeps/_conda/envs/__trim-galore@0.4.3' > conda_activate.log 2>&1
    if [ $? -eq 0 ];then
        break
    else
        let COUNT=COUNT+1
        if [ $COUNT -eq $MAX_TRIES ];then
            echo "Failed to activate conda environment! Error was:"
            cat conda_activate.log
            exit 1
        fi
        sleep 10s
    fi
done ; trim_galore --version > /scratch2/galaxy/tmp/GALAXY_VERSION_STRING_7477 2>&1; ln -s '/home/.galaxy/database/files/013/dataset_13139.dat' input_1.fastq &&  trim_galore  --phred33    --output_dir ./      input_1.fastq  --dont_gzip  && if [ -f input_1_trimmed.fq.gz ] ; then mv input_1_trimmed.fq.gz input_1_trimmed.fq ; fi && if [ -f input_1_val_1.fq.gz ] ; then mv input_1_val_1.fq.gz input_1_val_1.fq ; fi && if [ -f input_2_val_2.fq.gz ] ; then mv input_2_val_2.fq.gz input_2_val_2.fq ; fi && if [ -f input_1_unpaired_1.fq.gz ] ; then mv input_1_unpaired_1.fq.gz input_1_unpaired_1.fq ; fi && if [ -f input_2_unpaired_2.fq.gz ] ; then mv input_2_unpaired_2.fq.gz input_2_unpaired_2.fq ; fi



In fact most of the tools do this post-upgrade - where they cannot find a file.

Here is the trace from hisat2 - it fails in a similar way so we conclude that there is a common underlying fault here that is affecting the pathing of the scripts.

Any ideas how we should go about finding this error?

Thanks, Gerhard

PS Here is the hisat2 output.

#!/bin/bash

 

# The following block can be used by the job system

# to ensure this script is runnable before actually attempting

# to run it.

if [ -n "$ABC_TEST_JOB_SCRIPT_INTEGRITY_XYZ" ]; then

    exit 42

fi

[ "$CONDA_DEFAULT_ENV" = "/home/.galaxy/tooldeps/_conda/envs/mulled-v1-3b104c294f65450b09ba89d24826c61eddd553d789c9f1ae48a29715de1b7426" ] ||

MAX_TRIES=3

COUNT=0

while [ $COUNT -lt $MAX_TRIES ]; do

    . /home/.galaxy/tooldeps/_conda/bin/activate '/home/.galaxy/tooldeps/_conda/envs/mulled-v1-3b104c294f65450b09ba89d24826c61eddd553d789c9f1ae48a29715de1b7426' > conda_activate.log 2>&1

    if [ $? -eq 0 ];then

        break

    else

        let COUNT=COUNT+1

        if [ $COUNT -eq $MAX_TRIES ];then

            echo "Failed to activate conda environment! Error was:"

            cat conda_activate.log

            exit 1

        fi

        sleep 10s

    fi

done ; [ "$CONDA_DEFAULT_ENV" = "/home/.galaxy/tooldeps/_conda/envs/mulled-v1-3b104c294f65450b09ba89d24826c61eddd553d789c9f1ae48a29715de1b7426" ] ||

MAX_TRIES=3

COUNT=0

while [ $COUNT -lt $MAX_TRIES ]; do

    . /home/.galaxy/tooldeps/_conda/bin/activate '/home/.galaxy/tooldeps/_conda/envs/mulled-v1-3b104c294f65450b09ba89d24826c61eddd553d789c9f1ae48a29715de1b7426' > conda_activate.log 2>&1

    if [ $? -eq 0 ];then

        break

    else

        let COUNT=COUNT+1

        if [ $COUNT -eq $MAX_TRIES ];then

            echo "Failed to activate conda environment! Error was:"

            cat conda_activate.log

            exit 1

        fi

        sleep 10s

    fi

done ; hisat2 --version > /scratch2/galaxy/tmp/GALAXY_VERSION_STRING_7484 2>&1; set -o | grep -q pipefail && set -o pipefail;   ln -s '/home/.galaxy/database/files/013/dataset_13248.dat' genome.fa && hisat2-build -p ${GALAXY_SLOTS:-1} genome.fa genome &&        ln -f -s '/home/.galaxy/database/files/013/dataset_13238.dat' input_f.fastq &&  ln -f -s '/home/.galaxy/database/files/013/dataset_13243.dat' input_r.fastq &&     hisat2  -p ${GALAXY_SLOTS:-1}  -x 'genome'     -1 'input_f.fastq' -2 'input_r.fastq'                               | samtools sort - -@ ${GALAXY_SLOTS:-1} -l 6 -o '/home/.galaxy/database/jobs_directory/007/7484/galaxy_dataset_19227.dat'

 

The job fails when it cannot find “hisat2-build”. 

 

Fatal error: Exit code 127 ()

/home/.galaxy/database/jobs_directory/007/7484/tool_script.sh: line 41: hisat2-build: command not found

 

The PATH variable should be exported with installation (/home/.galaxy/tooldeps/hisat/2.0.3/iuc/package_hisat_2_0_3/dc03603edd43/env.sh):

 

PATH=/home/.galaxy/tooldeps/hisat/2.0.3/iuc/package_hisat_2_0_3/dc03603edd43/bin:$PATH; export PATH

HISAT2_ROOT_DIR=/home/.galaxy/tooldeps/hisat/2.0.3/iuc/package_hisat_2_0_3/dc03603edd43; export HISAT2_ROOT_DIR

HISAT_ROOT_DIR=/home/.galaxy/tooldeps/hisat/2.0.3/iuc/package_hisat_2_0_3/dc03603edd43; export HISAT_ROOT_DIR

 

The conda environment with dependencies seems to be there:

 

/home/.galaxy/tooldeps/_conda/envs/mulled-v1-3b104c294f65450b09ba89d24826c61eddd553d789c9f1ae48a29715de1b7426/bin




Björn Grüning <bjoern.gruening@gmail.com>;
galaxy-dev@lists.galaxyproject.org

From: Björn Grüning <bjoern.gruening@gmail.com>
Sent: 05 July 2018 22:00:58
To: Van Wageningen, GERHARD [gerhardv@sun.ac.za]; galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] Conda problems: Failed to activate conda environment
 
Hi Gerhard,

it seems your tool is picking up the old tool_dependency version and not
the conda version of python.

Have a look at
https://github.com/galaxyproject/galaxy/blob/dev/config/dependency_resolvers_conf.xml.sample


to configure the order of dependency resolution in Galaxy.

Cheers,
Bjoern

> Hi All
>
> I have recently started learning the Galaxy setup starting with an upgrade.
> Made some headway but I was wondering if someone could comment on a
> problem that has emerged.
> The system is a Galaxy 18.05 installation (upgraded from 17.01) on an
> Altair PBS scheduler with drmaa setup (working)
>
> Conda problem: Failed to activate conda environment.
>
> Conda works during tool installation or reinstallation.
>
> Galaxy tools have been updated/reinstalled so they all look "green" in
> manage tools.
>
> In Manage Dependencies there are a number of "Dependency resolved but
> version not found" warnings, otherwise the deps look clean.
>
> When running tools in Galaxy errors occur, after the jobs have correctly
> deployed on the cluster.
>
>
> The galaxy application raised errors like the two below.
>
> "Failed to activate conda environment! Error was:
> Fatal Python error: Py_Initialize: Unable to get the locale encoding
>    File
> "/home/.galaxy/tooldeps/python/2.7/iuc/package_python_2_7/8b09fe018cac/lib/python2.7/encodings/__init__.py",
> line 123
>      raise C"
>
> "
>
> Failed to activate conda environment! Error was: CondaEnvironmentError:
> Environment error: Cannot activate environment bash. User does not have
> write access for conda symlinks.
>
>
> "
>
>
> Galaxy started and stopped using supervisord with config below:
>
>
> Thanks, hopefully someone has seen this before ...
>
>
> regards, Gerhard
>
>
>
> [program:web]
> command         = uwsgi --plugin python --virtualenv /home/.galaxy/.venv
> --ini-paste /home/.galaxy/config/galaxy.ini
> directory       = /home/.galaxy
> umask           = 022
> autostart       = true
> autorestart     = true
> startsecs       = 10
> user            = galaxy
> numprocs        = 1
> stopsignal      = INT
>
> [program:handler]
> command         = python ./scripts/galaxy-main -c
> /home/.galaxy/config/galaxy.ini --server-name=handler%(process_num)s
> directory       = /home/.galaxy
> process_name    = handler%(process_num)s
> numprocs        = 1
> umask           = 022
> autostart       = true
> autorestart     = true
> startsecs       = 10
> user            = galaxy
> environment     =
> VIRTUAL_ENV="/home/.galaxy/.venv",PATH="/home/.galaxy/.venv/bin:%(ENV_PATH)s",http_proxy="http://storage2.eth.hpc:3128/",https_proxy="http://storage2.eth.hpc:3128/"
> stdout_logfile  = /home/.galaxy/handler%(process_num)s.log
> redirect_stderr = true
>
> [group:galaxy]
> programs        = handler, web
>
> <http://www.sun.ac.za/english/Pages/Water-crisis.aspx>
>
> The integrity and confidentiality of this email is governed by these
> terms. Disclaimer <http://www.sun.ac.za/emaildisclaimer >
> Die integriteit en vertroulikheid van hierdie e-pos word deur die
> volgende bepalings gereël. Vrywaringsklousule
> <http://www.sun.ac.za/emaildisclaimer >
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/
>

The integrity and confidentiality of this email is governed by these terms. Disclaimer
Die integriteit en vertroulikheid van hierdie e-pos word deur die volgende bepalings gereël. Vrywaringsklousule