Dear galaxy-dev's,
about a year ago you discussed on variable number of inputs into a workflow (see http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-November/012012.html and http://thread.gmane.org/gmane.science.biology.galaxy.devel/4502/focus=4502, for example). We're interested in having something simple like a number of fastq files that have to undergo automated quality control and trimming. A workflow would describe steps that need to be done per file (like fastqc, quality trimmer and so on), and in the end, a tabular data summarizes statistics on all files. Of course, we could prebuild workflows for 2,3,4,5,..n input files and join the output, but it would be much cooler to have it variable. Since it's possible to start many instances of a workflow (i.e. one instance per file), this would be a good starting point. But how would one combine outputs of those instances? Is anyone out there having experience with such setups? Which files are involved in starting many instances of a workflow? Any other ideas or suggestions on how to go from here? Thanks a lot for any input!
Best, Jens
Max-Planck-Institute for Heart and Lung Research Bioinformatics Service - FGI Ludwigstraße 43 61231 Bad Nauheim
Phone. +49 6032 705 1765 Mail. jens.preussner@mpi-bn.mpg.demailto:jens.preussner@mpi-bn.mpg.de
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Galaxy developers,
I would be particularly interested in seeing this implemented as flow control blocks. Having such blocks in workflows would vastly expand the capabilities. Ideally there would be something like
- - for (i=0 to max) - - foreach (file in files) - - while - - if (could pass off "testing" to some external bit of code/another programme which would return 1/0) - - switch case
This would allow you to run a copy of the workflow foreach of Jens' files, and then run another foreach loop on the outputs of the foreach to concatenate them down to a single file (or something like that to combine them)
Just my two cents.
Cheers, Eric
On 12/10/2013 04:05 AM, Preussner, Jens wrote:
Dear galaxy-dev’s,
about a year ago you discussed on variable number of inputs into a workflow (see http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-November/012012.html and http://thread.gmane.org/gmane.science.biology.galaxy.devel/4502/focus=4502, for example). We’re interested in having something simple like a number of fastq files that have to undergo automated quality control and trimming. A workflow would describe steps that need to be done per file (like fastqc, quality trimmer and so on), and in the end, a tabular data summarizes statistics on all files. Of course, we could prebuild workflows for 2,3,4,5,..n input files and join the output, but it would be much cooler to have it variable. Since it’s possible to start many instances of a workflow (i.e. one instance per file), this would be a good starting point. But how would one combine outputs of those instances? Is anyone out there having experience with such setups? Which files are involved in starting many instances of a workflow? Any other ideas or suggestions on how to go from here? Thanks a lot for any input!
Best,
Jens
Max-Planck-Institute for Heart and Lung Research
Bioinformatics Service - FGI
Ludwigstraße 43
61231 Bad Nauheim
Phone. +49 6032 705 1765
Mail. jens.preussner@mpi-bn.mpg.de mailto:jens.preussner@mpi-bn.mpg.de
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
- -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr@tamu.edu rasche.eric@yandex.ru
In a former position I wrote extensions to Galaxy that allow workflows such this:
https://bitbucket.org/galaxy/galaxy-central/pull-request/116/multiple-file-d... https://bitbucket.org/msiappdev/galaxy-extras/commits/all http://bit.ly/beyond-proteomics
so believe me I understand this is important. I was recently hired by the Galaxy team and am now working on a more palatable variant on these ideas. More palatable also means more complex and more work to implement unfortunately... but I am working on it.
In the meantime, have you considered driving these kind of workflows via the API? I think the refinery platform (https://github.com/parklab/refinery-platform) for instance targets the Galaxy API and rewrites workflows at runtime to handle variable numbers of inputs mitigating Galaxy's limitations. This would require some custom development and some mechanism outside of the traditional Galaxy channels to launch the workflows and probably is not worth the effort unless you are talking about a small pool of fixed workflows.
-John
On Tue, Dec 10, 2013 at 4:05 AM, Preussner, Jens Jens.Preussner@mpi-bn.mpg.de wrote:
Dear galaxy-dev’s,
about a year ago you discussed on variable number of inputs into a workflow (see http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-November/012012.html and http://thread.gmane.org/gmane.science.biology.galaxy.devel/4502/focus=4502, for example). We’re interested in having something simple like a number of fastq files that have to undergo automated quality control and trimming. A workflow would describe steps that need to be done per file (like fastqc, quality trimmer and so on), and in the end, a tabular data summarizes statistics on all files. Of course, we could prebuild workflows for 2,3,4,5,..n input files and join the output, but it would be much cooler to have it variable. Since it’s possible to start many instances of a workflow (i.e. one instance per file), this would be a good starting point. But how would one combine outputs of those instances? Is anyone out there having experience with such setups? Which files are involved in starting many instances of a workflow? Any other ideas or suggestions on how to go from here? Thanks a lot for any input!
Best,
Jens
Max-Planck-Institute for Heart and Lung Research
Bioinformatics Service - FGI
Ludwigstraße 43
61231 Bad Nauheim
Phone. +49 6032 705 1765
Mail. jens.preussner@mpi-bn.mpg.de
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Eric Rasche -
John Chilton -
Preussner, Jens