Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
Any ideas?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
Hey Michael,
I know there is a lot of interest in this, but these sorts of things are not possible with Galaxy workflows at this time. If you want the traceibility and UI for viewing results provided by Galaxy - you could consider driving tools externally via the API - at that point though the utility of running these jobs through Galaxy has dropped considerably.
There is a Trello card I would encourage you to vote for - https://trello.com/c/rwSeAwos
I have included a link to this e-mail as a reference for specific use cases to consider if or when this idea gets worked on.
Sorry, -John
On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell mepcotterell@gmail.com wrote:
Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
Any ideas?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Michael,
I, too, would love to have conditionals and loops, however may want to consider the following approach:
Instead of splitting the web service submission, check for results, and download of results into three steps, make it a single script.
I believe this would solve your problem completely. It largely simplify your use case, as a single script could
- - submit and remember a job's ID - - check periodically using that language's web bindings - - collect the results when done
This would mean it's a long-running task, which is perfectly suited to use in galaxy.
Cheers, Eric
On 03/06/2014 07:51 AM, John Chilton wrote:
Hey Michael,
I know there is a lot of interest in this, but these sorts of things are not possible with Galaxy workflows at this time. If you want the traceibility and UI for viewing results provided by Galaxy - you could consider driving tools externally via the API - at that point though the utility of running these jobs through Galaxy has dropped considerably.
There is a Trello card I would encourage you to vote for - https://trello.com/c/rwSeAwos
I have included a link to this e-mail as a reference for specific use cases to consider if or when this idea gets worked on.
Sorry, -John
On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell mepcotterell@gmail.com wrote:
Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
Any ideas?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
- -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu rasche.eric@yandex.ru
While I agree that would work, the tool I’m working with generates tools for web operations in a generic fashion. That is, you provide it a WDSL and a list of operations you want from that WSDL, and then tool XML files are generated for each of those operations.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Thursday, March 6, 2014 at 9:08 AM, Eric Rasche wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Michael,
I, too, would love to have conditionals and loops, however may want to consider the following approach:
Instead of splitting the web service submission, check for results, and download of results into three steps, make it a single script.
I believe this would solve your problem completely. It largely simplify your use case, as a single script could
- submit and remember a job's ID
- check periodically using that language's web bindings
- collect the results when done
This would mean it's a long-running task, which is perfectly suited to use in galaxy.
Cheers, Eric
On 03/06/2014 07:51 AM, John Chilton wrote:
Hey Michael,
I know there is a lot of interest in this, but these sorts of things are not possible with Galaxy workflows at this time. If you want the traceibility and UI for viewing results provided by Galaxy - you could consider driving tools externally via the API - at that point though the utility of running these jobs through Galaxy has dropped considerably.
There is a Trello card I would encourage you to vote for - https://trello.com/c/rwSeAwos
I have included a link to this e-mail as a reference for specific use cases to consider if or when this idea gets worked on.
Sorry, -John
On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote:
Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
Any ideas?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu (mailto:esr@tamu.edu) rasche.eric@yandex.ru (mailto:rasche.eric@yandex.ru) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux)
iQIcBAEBAgAGBQJTGIFFAAoJEMqDXdrsMcpV6hsQAKGv/W0dtZXAiuf855D2PU8k cbyMxliigbvlg0ieZNb0ZFAsx5LHRuek8sU0HMf0v14wxgs6vlwDtp143x5R0MIY zjhNQkj+C9IXyq/qjrqT8IObbMUGgPFJD/hzrP9YCT7gUKE9Dy1fxN2aznUJgJJU fmMD48E0/+7/y4dwdRaZ/LQv9jAKHKg1S6df6rYYWa1/hZtWrAGYrycygcrnt0aW Ai0iE6IXDTbPYZSjNcWLMPJ7FPU/MGdIF1Lia1PB9GLAqwY5Iln6WJ6OUdahNQzw Yx+0Te80r6wZVKHb5Bx8bszOEfPzxHcR0XABGn6r6zREoFD1kTQqugWyn0FL/v0C zibHMbFje+LDgUU47QoyNRGXcM6RoiVJGo7KTcDbP5HEWqQ7ozmZSN2ECeX1s1gf mzE9sJadLH3P8+vA6wO+xKtPpEzvC8VJHFROZNPJNVpVFAKU1RBRSo8qD1ZDZtTx LnwXmx0lK0spaJXXGg8g8tO23ilwy0JXRqxt56oPp7MlUtDJ7eDyQtmdQlPlNY4v +eqeJlGoRBlT3AS7CMfY4PWhrg3WZ0dLS+w/bG7Pg4uOj+iZ1qVEHDS4ugwqszNK eksGuXb/Uc9GGwB1xUssF9XMwf6u/xK9OqCuW8rgzFhm85BhqEJFqiJO408Ibx/S HPnswg0jhak5c8DKlajS =TirM -----END PGP SIGNATURE----- ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell mepcotterell@gmail.com wrote:
While I agree that would work, the tool I'm working with generates tools for web operations in a generic fashion. That is, you provide it a WDSL and a list of operations you want from that WSDL, and then tool XML files are generated for each of those operations.
So could your "WDSL Galaxy Tool Factory" also produce a wrapper script to go with the Galaxy Tool XML, where the wrapper script handles polling the service with the unique identifier assigned by the service?
Peter
The tool is generic. This means that it is supposed to work (and does work) with any WSDL. In the described use case, one of the operations/“generated tools" already serves as a polling tool: it takes a unique id and requests a result from the web service, then the web service either returns the results or returns a message indicating that the results are not ready. Conditionals/Loops would be a perfect way to solve this.
If we added a polling tool like you described, then that tool would need to Web service specific since not all web service adhere to the same naming conventions and level of asynchrony.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote:
While I agree that would work, the tool I'm working with generates tools for web operations in a generic fashion. That is, you provide it a WDSL and a list of operations you want from that WSDL, and then tool XML files are generated for each of those operations.
So could your "WDSL Galaxy Tool Factory" also produce a wrapper script to go with the Galaxy Tool XML, where the wrapper script handles polling the service with the unique identifier assigned by the service?
Peter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
We're not suggesting adding a "polling tool"; instead the suggestion was that your "WSDL tool generator" generate the XML tool definition and a wrapper script.
That wrapper script would handle whatever WSDL operation it was created for, polling, and collection of results (ALL IN ONE). Since polling is a WSDL specific operation as you say, this would solve that, as every generated tool would have code to submit+poll+collect.
Conditionals and loops would generate /many/ "empty" jobs saying "your results are not done yet", which would not lend themselves to inclusion in workflows. galaxy workflow loops are probably not the best way to solve this particular problem.
Additionally, If you write your WSDL tool generator in such a way that the generated tools handles the entire process: from submission to retrieval of results, then they could be used in workflows and used with the existing infrastructure, and jobs down the line could depend on the output of your generated tools.
Cheers, Eric
On 03/06/2014 08:25 AM, Michael E. Cotterell wrote:
The tool is generic. This means that it is supposed to work (and does work) with any WSDL. In the described use case, one of the operations/“generated tools" already serves as a polling tool: it takes a unique id and requests a result from the web service, then the web service either returns the results or returns a message indicating that the results are not ready. Conditionals/Loops would be a perfect way to solve this.
If we added a polling tool like you described, then that tool would need to Web service specific since not all web service adhere to the same naming conventions and level of asynchrony.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote:
While I agree that would work, the tool I'm working with generates tools for web operations in a generic fashion. That is, you provide it a WDSL and a list of operations you want from that WSDL, and then tool XML files are generated for each of those operations.
So could your "WDSL Galaxy Tool Factory" also produce a wrapper script to go with the Galaxy Tool XML, where the wrapper script handles polling the service with the unique identifier assigned by the service?
Peter
- -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu rasche.eric@yandex.ru
I’m sorry for the confusion. These really are three different operations that handled by three different web service operations with the web service. Other WSDLs might have similar operations, but the process might be different. There is no generic way to do what you just described.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Thursday, March 6, 2014 at 10:01 AM, Eric Rasche wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
We're not suggesting adding a "polling tool"; instead the suggestion was that your "WSDL tool generator" generate the XML tool definition and a wrapper script.
That wrapper script would handle whatever WSDL operation it was created for, polling, and collection of results (ALL IN ONE). Since polling is a WSDL specific operation as you say, this would solve that, as every generated tool would have code to submit+poll+collect.
Conditionals and loops would generate /many/ "empty" jobs saying "your results are not done yet", which would not lend themselves to inclusion in workflows. galaxy workflow loops are probably not the best way to solve this particular problem.
Additionally, If you write your WSDL tool generator in such a way that the generated tools handles the entire process: from submission to retrieval of results, then they could be used in workflows and used with the existing infrastructure, and jobs down the line could depend on the output of your generated tools.
Cheers, Eric
On 03/06/2014 08:25 AM, Michael E. Cotterell wrote:
The tool is generic. This means that it is supposed to work (and does work) with any WSDL. In the described use case, one of the operations/“generated tools" already serves as a polling tool: it takes a unique id and requests a result from the web service, then the web service either returns the results or returns a message indicating that the results are not ready. Conditionals/Loops would be a perfect way to solve this.
If we added a polling tool like you described, then that tool would need to Web service specific since not all web service adhere to the same naming conventions and level of asynchrony.
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell <mepcotterell@gmail.com (mailto:mepcotterell@gmail.com)> wrote:
While I agree that would work, the tool I'm working with generates tools for web operations in a generic fashion. That is, you provide it a WDSL and a list of operations you want from that WSDL, and then tool XML files are generated for each of those operations.
So could your "WDSL Galaxy Tool Factory" also produce a wrapper script to go with the Galaxy Tool XML, where the wrapper script handles polling the service with the unique identifier assigned by the service?
Peter
Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu (mailto:esr@tamu.edu) rasche.eric@yandex.ru (mailto:rasche.eric@yandex.ru) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux)
iQIcBAEBAgAGBQJTGI2vAAoJEMqDXdrsMcpVMsAQALuigNEIfG0qqR2pxULOEK39 GYkJ9q8RFRgmQAr5STM97XoeZD+mYyqzjrsdB8XL6LldkN0+XN+Aak8I4KbQlidu cWnZtjeY46DAxQdOcEzHCfL0apjgeOb+uCcxAtewVUBC4QMU11GkoL+54ZKoNWVV Vitt8N0hM4WJE+MukwZ9tCjjvpW7sVM1TUfx0bBb47qyx+sxH14JiaICBYpp6jDP Y3Kx/ncMwr62O2pMDuoP6YHn+ozKRwKa85uHEKx9mv/aQ1/0/F2Y3sd3hoj2fv4k wxw+7GY05CBUat6d4OH3bq8K6C2Z7Ye/nQc1FClM4ssjfXUEulXgzbOwwxmoyPqw fQPGIvWxr2YrewYsTScIO2m7kg7Y70d2gW+jRHFZJd9Yiz90ze5et5veqJ9vw7sl v71jZkkANF8ONdHE4cIkEYBbsNuhTJvSaYOvt0YAhMe18vvBe/AwQ4AwWKcsRYhn CnilCjICOcwJFDrRsVtAIIU/PmEl9w5tztRozHe+RtMNt4PFyn7qVdlgsFQogEQm FWhIpTAjioyc/46kIg5vqcA0BR8KJNllCOiFDDI7ZGJJAzykPjEeZshbQcDdA0Do Axfrr1ata/iHZsIuXsMQrVfiY8lr5XPimdXAdMPxdt+jThUP803lwDIgu9gD7wWm LBL7haO0S65UDiZ4Be8m =SvuS -----END PGP SIGNATURE----- ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I remember this being mentioned in a project called Tavaxy http://www.biomedcentral.com/1471-2105/13/77, but it's a separate piece of software (using Galaxy) and I don't know if it's been updated in a while so it may not be as useful for you. Not sure if anybody else had any experience with it?
It would be pretty cool to have conditionals and loops within workflows though.
Aaron
On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell <mepcotterell@gmail.com
wrote:
Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
Any ideas?
Sincerely, Michael E. Cotterell
Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell@gmail.com (mailto:mepcotterell@gmail.com) mepcott@uga.edu (mailto:mepcott@uga.edu) mec@cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Aaron Petkau -
Eric Rasche -
John Chilton -
Michael E. Cotterell -
Peter Cock