Missing requirements in xml wrappers in galaxy-dist?
Hi James & all, I have been getting some errors to do with the path environment variable. For instance, when uploading a sam file to our local galaxy instance, we got: "Traceback (most recent call last): File "/data/ugalaxy/galaxy-dist/tools/data_source/upload.py", line 394, in __main__() ..... line 63, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/local/lib/python2.7/subprocess.py", line 1228, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory" I can post the full error if you'd like, but basically the problem was that samtools wasn't in the PATH. This was because we have our tools installed in a non-standard place, so we are depending on the requirements being specified as James described below, and samtools isn't specified as a requirement in upload.xml, so when upload.py calls datatypes.py and tries to use samtools, it gives an error. I've found a couple of other examples like this - for instance samtools is also used by some picard scripts so should be specified as a requirement in the picard wrappers. These problems probably don't show up in most cases when people just have the tools installed as root and on their PATH by default? I'm going to be fixing these where I find them. Would it be helpful for me to contribute these tweaks back or would it be better just to raise an issue? Thanks, Clare On Wed, Nov 16, 2011 at 4:33 PM, Clare Sloggett <sloc@unimelb.edu.au> wrote:
Looks like it's working! The problem I had run into, in hindsight, was a) I hadn't set tool_dependency_dir as I didn't know about it b) if I had set it, I was installing tools to $SW/tool-name/version-number/ but the default tool wrappers in galaxy-dist don't have version numbers set, so they will just look in $SW/tool-name/ .
Can I suggest this be added to the wiki somewhere under the Admin pages? Apologies if it's there, I couldn't find it except under News Briefs at http://wiki.g2.bx.psu.edu/News%20Briefs/2010_11_24?highlight=%28tool_depende... .
As well as being on the wiki it would be useful to have it (commented out by default) in universe_wsgi.ini. I think the tool_dependency_dir variable isn't in there at all at the moment, at least in the galaxy-dist I have. It would also be useful to have a brief mention or link to it on http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup to save time for people like me who had tools installed in a non-standard place.
Thanks again! Clare
On Wed, Nov 16, 2011 at 3:01 PM, Clare Sloggett <sloc@unimelb.edu.au> wrote:
Great! Thanks James, this is exactly what I need.
On Wed, Nov 16, 2011 at 2:20 PM, James Taylor <james@jamestaylor.org> wrote:
On Nov 15, 2011, at 9:59 PM, Clare Sloggett wrote:
If this is the case, what is the best way to install and maintain two versions of the same tool? I can write code into the wrapper to find the correct version of the tool in a given case, but I was wondering if there is a more standard 'galaxy' way to configure this.
You should provide "tool_dependency_dir" in the config file and point it at some directory $SW where you will install tools under.
With this enabled, when a tool has a dependency, Galaxy will look for it under that directory and attempt to run a script to setup the environment. For example if you have tool with a dependency on foo version 1.3, Galaxy will look for:
$SW/foo/1.3/env.sh
and if found will source it as part of the job submission script. This usually contains something simple like
PATH=$PACKAGE_BASE/bin:$PATH
to add the binaries installed with the dependency to the path.
Ideally all dependencies used by Galaxy tools would be installed in this way.
-- E: sloc@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
-- E: sloc@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
-- E: sloc@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
Clare, it would be great if you can contribute these back. We have't gone through and added these everywhere yet. Pull requests in bitbucket would be easiest. Thanks! On Nov 16, 2011, at 9:29 PM, Clare Sloggett <sloc@unimelb.edu.au> wrote:
Hi James & all,
I have been getting some errors to do with the path environment variable. For instance, when uploading a sam file to our local galaxy instance, we got: "Traceback (most recent call last): File "/data/ugalaxy/galaxy-dist/tools/data_source/upload.py", line 394, in __main__() ..... line 63, in _get_samtools_version output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1] File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/local/lib/python2.7/subprocess.py", line 1228, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory"
I can post the full error if you'd like, but basically the problem was that samtools wasn't in the PATH. This was because we have our tools installed in a non-standard place, so we are depending on the requirements being specified as James described below, and samtools isn't specified as a requirement in upload.xml, so when upload.py calls datatypes.py and tries to use samtools, it gives an error. I've found a couple of other examples like this - for instance samtools is also used by some picard scripts so should be specified as a requirement in the picard wrappers.
These problems probably don't show up in most cases when people just have the tools installed as root and on their PATH by default?
I'm going to be fixing these where I find them. Would it be helpful for me to contribute these tweaks back or would it be better just to raise an issue?
Thanks, Clare
On Wed, Nov 16, 2011 at 4:33 PM, Clare Sloggett <sloc@unimelb.edu.au> wrote:
Looks like it's working! The problem I had run into, in hindsight, was a) I hadn't set tool_dependency_dir as I didn't know about it b) if I had set it, I was installing tools to $SW/tool-name/version-number/ but the default tool wrappers in galaxy-dist don't have version numbers set, so they will just look in $SW/tool-name/ .
Can I suggest this be added to the wiki somewhere under the Admin pages? Apologies if it's there, I couldn't find it except under News Briefs at http://wiki.g2.bx.psu.edu/News%20Briefs/2010_11_24?highlight=%28tool_depende... .
As well as being on the wiki it would be useful to have it (commented out by default) in universe_wsgi.ini. I think the tool_dependency_dir variable isn't in there at all at the moment, at least in the galaxy-dist I have. It would also be useful to have a brief mention or link to it on http://wiki.g2.bx.psu.edu/Admin/NGS%20Local%20Setup to save time for people like me who had tools installed in a non-standard place.
Thanks again! Clare
On Wed, Nov 16, 2011 at 3:01 PM, Clare Sloggett <sloc@unimelb.edu.au> wrote:
Great! Thanks James, this is exactly what I need.
On Wed, Nov 16, 2011 at 2:20 PM, James Taylor <james@jamestaylor.org> wrote:
On Nov 15, 2011, at 9:59 PM, Clare Sloggett wrote:
If this is the case, what is the best way to install and maintain two versions of the same tool? I can write code into the wrapper to find the correct version of the tool in a given case, but I was wondering if there is a more standard 'galaxy' way to configure this.
You should provide "tool_dependency_dir" in the config file and point it at some directory $SW where you will install tools under.
With this enabled, when a tool has a dependency, Galaxy will look for it under that directory and attempt to run a script to setup the environment. For example if you have tool with a dependency on foo version 1.3, Galaxy will look for:
$SW/foo/1.3/env.sh
and if found will source it as part of the job submission script. This usually contains something simple like
PATH=$PACKAGE_BASE/bin:$PATH
to add the binaries installed with the dependency to the path.
Ideally all dependencies used by Galaxy tools would be installed in this way.
-- E: sloc@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
-- E: sloc@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
-- E: sloc@unimelb.edu.au P: 03 903 53357 M: 0414 854 759
participants (2)
-
Clare Sloggett -
James Taylor