Difficulty dispatching toolshed tool jobs via job_conf.xml
I am having trouble getting a toolshed tool to be dispatched to the destination I list in the job_conf.xml file. My job_conf.xml file looks like this: <?xml version="1.0"?> <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="20"/> <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" /> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="condor"> <destination id="local" runner="local"/> <destination id="condor" runner="condor"> <!-- With no params, jobs are submitted to the 'vanilla' universe with: notification = NEVER getenv = true Additional/override query ClassAd params can be specified with <param> tags, e.g <param id="request_cpus">8</param> --> </destination> </destinations> <tools> <tool id="upload1" destination="local"/> <!-- Upload File --> <tool id="ucsc_table_direct1" destination="local"/> <!-- UCSC Main --> ... stuff omitted ... <tool id="toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS: infoseq46/5.0.0" destination="local"/> </tools> </job_conf> So, you can see I have a default destination of condor, but I'm trying to run my toolshed EMBOSS infoseq tool on local. However, it is stubbornly running on condor. In lib/galaxy/tools/__init__.py:1132, I see this comment which got me wondering: # In the toolshed context, there is no job config. Is it possible to define tool destinations for toolshed tools? Are there some gotchas that I should know about? Any other ideas why my job is ignoring the config in job_conf.xml? (By the way, I can change say the upload1 tool to run on Condor by setting its destination in that file, so it is doing something.) The other thing I saw in the source code is stuff about old_id and toolshed guids. Do I need to understand this stuff? The paster.log contains the following when I submit the infoseq job: 147.158.130.216 - - [11/Sep/2013:13:53:34 +1300] "GET /tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0 HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" 147.158.130.216 - - [11/Sep/2013:13:53:40 +1300] "POST /tool_runner/index HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/tool_runner?tool_id=toolshed-dev.agresear..." "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" galaxy.jobs DEBUG 2013-09-11 13:53:40,886 (92) Working directory for job is: /home/galaxy-dev/galaxy/database/job_working_directory/000/92 galaxy.tools DEBUG 2013-09-11 13:53:40,886 Tool::get_job_destination: {'runner': 'condor', 'legacy': False, 'params': {}, 'tags': None, 'url': None, 'converted': False, 'id': 'condor'}. galaxy.jobs.handler DEBUG 2013-09-11 13:53:40,894 (92) Dispatching to condor runner (I added the debug output for Tool::get_job_destination to see what was going on.) Any ideas? cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Hi Simon, This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of: shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the <tool> tag in job_conf.xml: toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0 --nate On Sep 10, 2013, at 10:35 PM, Guest, Simon wrote:
I am having trouble getting a toolshed tool to be dispatched to the destination I list in the job_conf.xml file.
My job_conf.xml file looks like this:
<?xml version="1.0"?> <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="20"/> <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" /> </plugins> <handlers> <handler id="main"/> </handlers> <destinations default="condor"> <destination id="local" runner="local"/> <destination id="condor" runner="condor"> <!-- With no params, jobs are submitted to the 'vanilla' universe with: notification = NEVER getenv = true Additional/override query ClassAd params can be specified with <param> tags, e.g <param id="request_cpus">8</param> --> </destination> </destinations> <tools> <tool id="upload1" destination="local"/> <!-- Upload File --> <tool id="ucsc_table_direct1" destination="local"/> <!-- UCSC Main -->
... stuff omitted ...
<tool id="toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS:infoseq46/5.0.0" destination="local"/> </tools> </job_conf>
So, you can see I have a default destination of condor, but I'm trying to run my toolshed EMBOSS infoseq tool on local. However, it is stubbornly running on condor.
In lib/galaxy/tools/__init__.py:1132, I see this comment which got me wondering: # In the toolshed context, there is no job config.
Is it possible to define tool destinations for toolshed tools? Are there some gotchas that I should know about? Any other ideas why my job is ignoring the config in job_conf.xml? (By the way, I can change say the upload1 tool to run on Condor by setting its destination in that file, so it is doing something.) The other thing I saw in the source code is stuff about old_id and toolshed guids. Do I need to understand this stuff?
The paster.log contains the following when I submit the infoseq job:
147.158.130.216 - - [11/Sep/2013:13:53:34 +1300] "GET /tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0 HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" 147.158.130.216 - - [11/Sep/2013:13:53:40 +1300] "POST /tool_runner/index HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/tool_runner?tool_id=toolshed-dev.agresear..." "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0" galaxy.jobs DEBUG 2013-09-11 13:53:40,886 (92) Working directory for job is: /home/galaxy-dev/galaxy/database/job_working_directory/000/92 galaxy.tools DEBUG 2013-09-11 13:53:40,886 Tool::get_job_destination: {'runner': 'condor', 'legacy': False, 'params': {}, 'tags': None, 'url': None, 'converted': False, 'id': 'condor'}. galaxy.jobs.handler DEBUG 2013-09-11 13:53:40,894 (92) Dispatching to condor runner
(I added the debug output for Tool::get_job_destination to see what was going on.)
Any ideas?
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. ======================================================================= ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the <tool> tag in job_conf.xml:
Hi Nate, Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this. cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Hi Nate & Simon, was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or? Thanks, Bjoern
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the <tool> tag in job_conf.xml:
Hi Nate,
Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this.
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Björn, It should be fixed, the problem was with tools with uppercase characters in the guid: https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/ --nate On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning <bjoern.gruening@gmail.com>wrote:
Hi Nate & Simon,
was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or?
Thanks, Bjoern
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the <tool> tag in job_conf.xml:
Hi Nate,
Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this.
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Bjoern, Nate, That's good to hear. Sorry, I have been otherwise engaged for the last few months, and was unable to test this. Hoping to get back onto Galaxy stuff again in a month or so! cheers, Simon
-----Original Message----- From: Nate Coraor [mailto:nate@bx.psu.edu] Sent: Saturday, 15 February 2014 4:37 a.m. To: Björn Grüning Cc: Guest, Simon; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Difficulty dispatching toolshed tool jobs via job_conf.xml
Hi Björn,
It should be fixed, the problem was with tools with uppercase characters in the guid:
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/
--nate
On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi Nate & Simon,
was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or?
Thanks, Bjoern
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the <tool> tag in job_conf.xml:
Hi Nate,
Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this.
cheers, Simon
Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or
material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify
======================================================================= privileged the
sender immediately.
=======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Nate, sorry to bother you again, but that issue is still not fixed for me. The following is supposed to work, or? <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastx_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_tblastn_wrapper/" destination="24_cores_24G" /> <tool id="ncbi_blastp_wrapper" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_rpsblast_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_tblastx_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/peterjc/blast2go/blast2go/0.0.6" destination="20G_memory"/> Thanks, Bjoern Am 14.02.2014 16:37, schrieb Nate Coraor:
Hi Björn,
It should be fixed, the problem was with tools with uppercase characters in the guid:
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/
--nate
On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning <bjoern.gruening@gmail.com>wrote:
Hi Nate & Simon,
was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or?
Thanks, Bjoern
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the <tool> tag in job_conf.xml:
Hi Nate,
Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this.
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Björn, Please try it without the trailing slash on the tool ID. --nate On Wed, Feb 26, 2014 at 11:28 AM, Björn Grüning <bjoern.gruening@gmail.com>wrote:
Hi Nate,
sorry to bother you again, but that issue is still not fixed for me.
The following is supposed to work, or? <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_blastn_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_blastx_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_tblastn_wrapper/" destination="24_cores_24G" /> <tool id="ncbi_blastp_wrapper" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_rpsblast_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_tblastx_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/peterjc/blast2go/ blast2go/0.0.6" destination="20G_memory"/>
Thanks, Bjoern
Am 14.02.2014 16:37, schrieb Nate Coraor:
Hi Björn,
It should be fixed, the problem was with tools with uppercase characters in the guid:
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/
--nate
On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning <bjoern.gruening@gmail.com
wrote:
Hi Nate & Simon,
was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or?
Thanks, Bjoern
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool
shed
IDs when determining job destinations. You are actually supposed to be
able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try
in
the short term is using the percent encoded tool id in the <tool> tag
in
job_conf.xml:
Hi Nate,
Using the percent encoded form was the first thing I tried (before
emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this.
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Nate, without a trailing slash it works. I created PR #339, that removes trailing slashes from tool IDs. Thanks! Bjoern
Hi Björn,
Please try it without the trailing slash on the tool ID.
--nate
On Wed, Feb 26, 2014 at 11:28 AM, Björn Grüning <bjoern.gruening@gmail.com>wrote:
Hi Nate,
sorry to bother you again, but that issue is still not fixed for me.
The following is supposed to work, or? <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_blastn_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_blastx_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_tblastn_wrapper/" destination="24_cores_24G" /> <tool id="ncbi_blastp_wrapper" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_rpsblast_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_tblastx_wrapper/" destination="24_cores_24G" /> <tool id="toolshed.g2.bx.psu.edu/repos/peterjc/blast2go/ blast2go/0.0.6" destination="20G_memory"/>
Thanks, Bjoern
Am 14.02.2014 16:37, schrieb Nate Coraor:
Hi Björn,
It should be fixed, the problem was with tools with uppercase characters in the guid:
https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/
--nate
On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning <bjoern.gruening@gmail.com
wrote:
Hi Nate & Simon,
was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or?
Thanks, Bjoern
From: Nate Coraor [mailto:nate@bx.psu.edu]
This should work, it looks like there may be a bug with handling tool
shed
IDs when determining job destinations. You are actually supposed to be
able to use any of:
shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id
I'll take a look at this as soon as possible. One thing you might try
in
the short term is using the percent encoded tool id in the <tool> tag
in
job_conf.xml:
Hi Nate,
Using the percent encoded form was the first thing I tried (before
emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this.
cheers, Simon
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Björn Grüning -
Guest, Simon -
Nate Coraor