Dear all, with older Galaxies (prior to latest_15.03 I think), you could satisfy Galaxy's samtools dependence for indexing bam files by having a samtools executable in tool-dependencies/samtools/default/bin (with the tool-dependencies directory declared as tool_dependency_dir in galaxy.ini of course). Now (checked with latest_15.03 and .05), this is not working any more! The executable will still be used during bam uploads, but not when a bam file gets created by a tool. The reason is that before the job runner (tested this with the local job runner only) used to build the dependency shell command for dependency 'samtools' before finishing a job, but now the job wrapper finish method fails because it naively expects to find samtools on $PATH. Best, Wolfgang
hmm, no replies yet, so is anybody able to reproduce this behavior and would you not consider it a bug? Best, Wolfgang On 06/16/2015 03:11 PM, Wolfgang Maier wrote:
Dear all,
with older Galaxies (prior to latest_15.03 I think), you could satisfy Galaxy's samtools dependence for indexing bam files by having a samtools executable in tool-dependencies/samtools/default/bin (with the tool-dependencies directory declared as tool_dependency_dir in galaxy.ini of course).
Now (checked with latest_15.03 and .05), this is not working any more! The executable will still be used during bam uploads, but not when a bam file gets created by a tool.
The reason is that before the job runner (tested this with the local job runner only) used to build the dependency shell command for dependency 'samtools' before finishing a job, but now the job wrapper finish method fails because it naively expects to find samtools on $PATH.
Best, Wolfgang
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Wolfgang, I only can tell you that we also have problems with handling BAM files properly in Galaxy. Our issue is more due to unsorted BAM files, but as far as I understood this is because the metadata creation changed from using samtools to using pysam. Maybe this helps you in finding a workaround. Ciao, Bjoern
hmm, no replies yet, so is anybody able to reproduce this behavior and would you not consider it a bug?
Best, Wolfgang
On 06/16/2015 03:11 PM, Wolfgang Maier wrote:
Dear all,
with older Galaxies (prior to latest_15.03 I think), you could satisfy Galaxy's samtools dependence for indexing bam files by having a samtools executable in tool-dependencies/samtools/default/bin (with the tool-dependencies directory declared as tool_dependency_dir in galaxy.ini of course).
Now (checked with latest_15.03 and .05), this is not working any more! The executable will still be used during bam uploads, but not when a bam file gets created by a tool.
The reason is that before the job runner (tested this with the local job runner only) used to build the dependency shell command for dependency 'samtools' before finishing a job, but now the job wrapper finish method fails because it naively expects to find samtools on $PATH.
Best, Wolfgang
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
So it wasn't in 15.01 - it was before that but the local job runner was changed to set metadata as part of the job (I think probably sometime in 2014). When it was setting metadata externally it might have used slightly different dependency resolution strategies that would result in samtools being sourced for every metadata collection and this would have resulted in the behavior you described. I think a bigger problem was that the local job runner behaved so differently than other job runners and that has been resolved. So now we have to fix this for all of the job runners in a uniform way. I think the path forward is something like this... 1. Right now there is this option in galaxy.ini called enable_beta_tool_command_isolation - this causes the command generated by the tool be executed in an environment (script) separate from the metadata code generation environment. This needs to be enabled by default so that a tool's dependencies don't interfere with the metadata generation code. 2. The metadata generation code uniformly needs to source samtools dependencies - like the old local job runner code did. Cannot do this before the separation because it would prevent us from supporting multiple versions of samtools for instance. That shouldn't be a particularly hard process - there will probably be bumps in the road and so I haven't enabled it by default yet. So in the meantime I would just ensure Galaxy sources the required packages on all your nodes and/or destinations. You can modify environment variables directly in your job_conf.xml file and this will work for any runner so this is easier than ever. https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_... -John On Tue, Jun 16, 2015 at 2:11 PM, Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> wrote:
Dear all,
with older Galaxies (prior to latest_15.03 I think), you could satisfy Galaxy's samtools dependence for indexing bam files by having a samtools executable in tool-dependencies/samtools/default/bin (with the tool-dependencies directory declared as tool_dependency_dir in galaxy.ini of course).
Now (checked with latest_15.03 and .05), this is not working any more! The executable will still be used during bam uploads, but not when a bam file gets created by a tool.
The reason is that before the job runner (tested this with the local job runner only) used to build the dependency shell command for dependency 'samtools' before finishing a job, but now the job wrapper finish method fails because it naively expects to find samtools on $PATH.
Best, Wolfgang
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Bjoern Gruening -
John Chilton -
Wolfgang Maier