Re: [galaxy-dev] Dynamic job runner not being dispatched for a tool

7 Oct 2013

      Hi John,

Thank you and Ilya for pointing this out. I was hitting my head on this for 2 days, convinced that I had misconfigured something. The tags didn't seem to have any effect, so I tried adding the .lower to the tool.get call. That did the trick.

Thanks again, so much, for the help! You saved me a re-install and quite a bit of sanity.

Sincerely,

Carrie Ganote
________________________________________
From: jmchilton@gmail.com [jmchilton@gmail.com] on behalf of John Chilton [chilton@msi.umn.edu]
Sent: Monday, October 07, 2013 2:02 PM
To: Ganote, Carrie L
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Dynamic job runner not being dispatched for a tool

I can look into this some more, but I have a couple quick questions.
Can you try without the tags attributes, I am not sure what is going
to happen since they do not match each other.

Also, have you tried this fix, it looks like it could potentially be
the same problem.
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451cf22a1a7c8fa0...

-John

On Mon, Oct 7, 2013 at 12:46 PM, Ganote, Carrie L <cganote@iu.edu> wrote:
...
Hi List,
I'm a bit baffled by a new tool I've tried to add to Galaxy. The job runner
does not dispatch it correctly.
I added the tool via the toolshed, set up the executable on the cluster
side, and attempted to hook it up to the dynamic job runner. The destination
always ended up set as local.
Works fine if the default runner is pbs, but if it's local by default, it
will only dispatch locally.
Attempting to work backwards, I copied the wrapper to a local tool
directory, and pared everything down to a minimum to run. There must be
something missing. My trinity_all dynamic definition works fine, Newbler
doesn't. Is there anything obvious here?
~~~~~tool_conf.xml~~~~~~~~~~~~~~~~~~~~
<?xml version="1.0"?>
<toolbox>
  <section name="Assembly" id="assembly">
    <tool file="sr_assembly/runAssembly.xml" />
    <tool file="ngs_rna/trinity_all.xml" />
  </section>
<!-- I have to have two sections in order for any sections to show up at all
-->
  <section name="Send Data" id="send">
    <tool file="data_destination/epigraph.xml" />
    <tool file="data_destination/epigraph_test.xml" />
    <tool file="genomespace/genomespace_exporter.xml" />
  </section>
</toolbox>
~~~~~tools/sr_assembly/runAssembly.xml~~~~~~~~~~~~~~~~~~~~
<tool id="runAssembly" name="runAssembly" version="1.0.1">
<description>De novo assembly of Roche/454 reads using Newbler</description>
<command>
runAssembly $sanger_input -o $newbler_metrics
</command>
<inputs>
    <!-- READSEQ INFILES -->
    <param name="sanger_input" type="data" format="fasta,fastqsanger"
label="SE Fasta/Fastq file"/>
    <!-- for walltime control -->
    <param name="walltime" type="select" label="How long will your job
need">
          <option selected="true" value="1">8 hrs</option>
          <option value="48">48 hrs</option>
    </param>
  </inputs>
  <outputs>
   <data name="newbler_metrics" format="txt" />
  </outputs>
</tool>
~~~~~job_conf.xml~~~~~~~~~~~~~~~~~~~~
<?xml version="1.0"?>
<job_conf>
  <plugins workers="4">
    <plugin id="local" type="runner"
load="galaxy.jobs.runners.local:LocalJobRunner" />
    <plugin id="pbs" type="runner"
load="galaxy.jobs.runners.pbs:PBSJobRunner" />
  </plugins>
  <handlers default="handlers">
    <handler id="handler0" tags="handlers"/>
  </handlers>
  <destinations default="local">
    <destination id="local" runner="local"/>
    <destination id="pbs_normal" runner="pbs" tags="mycluster">
      <param
id="Resource_List">nodes=1:ppn=4,vmem=32gb,walltime=24:00:00</param>
    </destination>
    <destination id="newbler_d" runner="dynamic">
      <param id="type">python</param>
      <param id="function">newbler</param>
    </destination>
    <destination id="trinity_d" runner="dynamic">
      <param id="type">python</param>
      <param id="function">trinity_all</param>
    </destination>
  </destinations>
  <tools>
    <tool id="runAssembly" destination="newbler_d"/>
    <tool id="trinity_all" destination="trinity_d"/>
  </tools>
  <limits>
  </limits>
</job_conf>
~~~~~lib/galaxy/jobs/rules/dynamicjobs.py~~~~~~~~~~~~~~~~~~~~
import os
from galaxy.jobs import JobDestination
def newbler(job):
    incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
    walltime_para = incoming["walltime"]
    walltime = int(walltime_para.replace("\"",""))
    params = {}
    string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime
    params["Resource_List"] = string
    return JobDestination(runner="pbs", params=params)
def trinity_all(job):
    incoming = dict( [ ( p.name, p.value ) for p in job.parameters ] )
    walltime_para = incoming["walltime"]
    walltime = int(walltime_para.replace("\"",""))
    params = {}
    string = "vmem=500gb,nodes=1:ppn=32,walltime=%d:00:00"%walltime
    params["Resource_List"] = string
    return JobDestination(runner="pbs", params=params)
I re-installed the Roche 454 toolsuite from the toolshed once more to see if
it just needed a kick. I copied the tool id into my job_conf file to make
sure I made no mistakes on the id. I tried to send it to dynamic job runner
and to pbs runner. I pulled and merged this galaxy about a month ago, so it
should be mostly up to date - but I've had so many issues that I may have to
just re-install from scratch.
Thanks a ton for any advice,
Carrie Ganote
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/