multiple input files
Dear all We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files. Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs? I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is. Thanks for your help Ulf ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
On Tue, Oct 1, 2013 at 1:27 PM, Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files.
Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is.
Thanks for your help Ulf
Use <param type="data" multiple="true" ... > instead of using <repeat ...><param type="data" ...></repeat>. I asked John Chilton about this recently via Twitter, and it is now on the wiki, http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax Simple example here: https://bitbucket.org/galaxyp/galaxyp-toolshed-iquant/src/tip/iquant.xml?at=... Peter
Hi Ulf, What I do: 1. make a history, doing the steps you want to do on one input file 2. create a workflow of that history 3. assemble all input files in one history 4. run the workflow and select the multiple input files to run the workflow on. 5. Optionally: send the results to a new history, for every input file you will get a new history, properly named. Hope this helps, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 10/01/2013 02:27 PM, Ulf Schaefer wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files.
Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is.
Thanks for your help Ulf
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Thanks Peter, Yeah, looking over unified_genotyper.xml, you would probably just want to replace: #for $i, $input_bam in enumerate( $reference_source.input_bams ): -d "-I" "${input_bam.input_bam}" "${input_bam.input_bam.ext}" "gatk_input_${i}" #if str( $input_bam.input_bam.metadata.bam_index ) != "None": -d "" "${input_bam.input_bam.metadata.bam_index}" "bam_index" "gatk_input_${i}" ##hardcode galaxy ext type as bam_index #end if #end for with #for $i, $input_bam in enumerate( $reference_source.input_bams ): -d "-I" "${input_bam}" "${input_bam.ext}" "gatk_input_${i}" #if str( $input_bam.metadata.bam_index ) != "None": -d "" "${input_bam.metadata.bam_index}" "bam_index" "gatk_input_${i}" ##hardcode galaxy ext type as bam_index #end if #end for And: <repeat name="input_bams" title="BAM file" min="1" help="-I,--input_file <input_file>"> <param name="input_bam" type="data" format="bam" label="BAM file"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_data_table" table_name="gatk_picard_indexes" metadata_name="dbkey" metadata_column="dbkey" message="Sequences are not currently available for the specified build." /> <!-- fixme!!! this needs to be a select --> </param> </repeat> with <param name="input_bams" type="data" multiple="true" format="bam" label="BAM file"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_data_table" table_name="gatk_picard_indexes" metadata_name="dbkey" metadata_column="dbkey" message="Sequences are not currently available for the specified build." /> <!-- fixme!!! this needs to be a select --> </param> And: <repeat name="input_bams" title="BAM file" min="1" help="-I,--input_file <input_file>"> <param name="input_bam" type="data" format="bam" label="BAM file" > </param> </repeat> with <param name="input_bams" multiple="true" type="data" format="bam" label="BAM file" > </param> I have not tested these specific changes, so your millage may vary. I have no clue if those validators in that second block are going to work with multiple="true". If you test it out and there is some problem, please let me know I can try to fix it. I don't know if the Galaxy team wants to start replacing these blocks in its tools - this change would break existing workflows built on the unified genotyper and I am not sure the Galaxy team has any interest in using these kind of data blocks going forward. They are working out great for us on Galaxy-P though. -John On Tue, Oct 1, 2013 at 7:57 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Tue, Oct 1, 2013 at 1:27 PM, Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files.
Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is.
Thanks for your help Ulf
Use <param type="data" multiple="true" ... > instead of using <repeat ...><param type="data" ...></repeat>.
I asked John Chilton about this recently via Twitter, and it is now on the wiki, http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax
Simple example here: https://bitbucket.org/galaxyp/galaxyp-toolshed-iquant/src/tip/iquant.xml?at=...
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Sorry, my answer doesn't fit your question. :-) J Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 10/01/2013 03:08 PM, Joachim Jacob | VIB | wrote:
Hi Ulf,
What I do: 1. make a history, doing the steps you want to do on one input file 2. create a workflow of that history 3. assemble all input files in one history 4. run the workflow and select the multiple input files to run the workflow on. 5. Optionally: send the results to a new history, for every input file you will get a new history, properly named.
Hope this helps,
Joachim
Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team
On 10/01/2013 02:27 PM, Ulf Schaefer wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files.
Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is.
Thanks for your help Ulf
**************************************************************************
The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Dear Peter, dear John, dear all Thanks very much. This is exactly what I need and the code changes proposed work perfectly as they are. At a short glance even the validators work as intended. Now for the optional special bonus: Is there a way to define the size of the boxes? The ones I see have 4 lines. Can I make them resizable somehow or define a larger size? (size="10" or resize="true" does not seem to do the trick.) Not a very pressing issue though. @Joachim: I am using the same approach elsewhere, but the difference is that it runs a tool multiple times, instead of running it once on multiple inputs. But thanks anyway. Cheers Ulf On 01/10/13 14:08, John Chilton wrote:
Thanks Peter,
Yeah, looking over unified_genotyper.xml, you would probably just want to replace:
#for $i, $input_bam in enumerate( $reference_source.input_bams ): -d "-I" "${input_bam.input_bam}" "${input_bam.input_bam.ext}" "gatk_input_${i}" #if str( $input_bam.input_bam.metadata.bam_index ) != "None": -d "" "${input_bam.input_bam.metadata.bam_index}" "bam_index" "gatk_input_${i}" ##hardcode galaxy ext type as bam_index #end if #end for
with
#for $i, $input_bam in enumerate( $reference_source.input_bams ): -d "-I" "${input_bam}" "${input_bam.ext}" "gatk_input_${i}" #if str( $input_bam.metadata.bam_index ) != "None": -d "" "${input_bam.metadata.bam_index}" "bam_index" "gatk_input_${i}" ##hardcode galaxy ext type as bam_index #end if #end for
And:
<repeat name="input_bams" title="BAM file" min="1" help="-I,--input_file <input_file>"> <param name="input_bam" type="data" format="bam" label="BAM file"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_data_table" table_name="gatk_picard_indexes" metadata_name="dbkey" metadata_column="dbkey" message="Sequences are not currently available for the specified build." /> <!-- fixme!!! this needs to be a select --> </param> </repeat>
with
<param name="input_bams" type="data" multiple="true" format="bam" label="BAM file"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_data_table" table_name="gatk_picard_indexes" metadata_name="dbkey" metadata_column="dbkey" message="Sequences are not currently available for the specified build." /> <!-- fixme!!! this needs to be a select --> </param>
And:
<repeat name="input_bams" title="BAM file" min="1" help="-I,--input_file <input_file>"> <param name="input_bam" type="data" format="bam" label="BAM file" > </param> </repeat>
with
<param name="input_bams" multiple="true" type="data" format="bam" label="BAM file" > </param>
I have not tested these specific changes, so your millage may vary. I have no clue if those validators in that second block are going to work with multiple="true". If you test it out and there is some problem, please let me know I can try to fix it.
I don't know if the Galaxy team wants to start replacing these blocks in its tools - this change would break existing workflows built on the unified genotyper and I am not sure the Galaxy team has any interest in using these kind of data blocks going forward. They are working out great for us on Galaxy-P though.
-John
On Tue, Oct 1, 2013 at 7:57 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Tue, Oct 1, 2013 at 1:27 PM, Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files.
Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is.
Thanks for your help Ulf
Use <param type="data" multiple="true" ... > instead of using <repeat ...><param type="data" ...></repeat>.
I asked John Chilton about this recently via Twitter, and it is now on the wiki, http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax
Simple example here: https://bitbucket.org/galaxyp/galaxyp-toolshed-iquant/src/tip/iquant.xml?at=...
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
Dear all Related to my question yesterday I have another one: If I run a workflow in batch I can select pairs of files (the old fwd and rev reads fastq files) and the workflow is run however many times, depending on the number of pairs I select. Unfortunately all other parameters appear to be static, which is not ideal in my case. Specifically I am using bwa the map a large-ish number of (paired) samples to the same reference genome, while simultaneously setting read group information in the resulting sam files. So it's suboptimal to run this with a fixed SM value. Is there any way of dynamically setting the parameters? The set-at-runtime option also allows for a single parameter value, not a list. Ideally the set-at-runtime option would somehow allow me to input one parameter per file pair. Thanks very much in advance for your help. Cheers Ulf On 01/10/13 14:09, Joachim Jacob | VIB | wrote:
Sorry, my answer doesn't fit your question. :-) J
Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team
On 10/01/2013 03:08 PM, Joachim Jacob | VIB | wrote:
Hi Ulf,
What I do: 1. make a history, doing the steps you want to do on one input file 2. create a workflow of that history 3. assemble all input files in one history 4. run the workflow and select the multiple input files to run the workflow on. 5. Optionally: send the results to a new history, for every input file you will get a new history, properly named.
Hope this helps,
Joachim
Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team
On 10/01/2013 02:27 PM, Ulf Schaefer wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files.
Using the <repeat> in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is.
Thanks for your help Ulf
**************************************************************************
The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE **************************************************************************
On Tue, Oct 1, 2013 at 2:52 PM, Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear Peter, dear John, dear all
Thanks very much. This is exactly what I need and the code changes proposed work perfectly as they are. At a short glance even the validators work as intended.
Now for the optional special bonus:
Is there a way to define the size of the boxes? The ones I see have 4 lines. Can I make them resizable somehow or define a larger size? (size="10" or resize="true" does not seem to do the trick.) Not a very pressing issue though.
+1 According to the wiki, the "size" attribute is only used on "text" input, but generalising it or adding something like "lines" would make sense here - and also for "select" parameters. It would be really nice if the user could resize the box if they wanted to - when trying to do a multiple selection from a long file list being able to seem just four entries at a time is painful. I used to have a Firefox extension to do this to arbitrary controls - very handy. Peter
Agreed Peter, this is essential functionality. Trello card created. https://trello.com/c/J7aqer08 Thanks! -John On Fri, Oct 4, 2013 at 5:48 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Tue, Oct 1, 2013 at 2:52 PM, Ulf Schaefer <Ulf.Schaefer@phe.gov.uk> wrote:
Dear Peter, dear John, dear all
Thanks very much. This is exactly what I need and the code changes proposed work perfectly as they are. At a short glance even the validators work as intended.
Now for the optional special bonus:
Is there a way to define the size of the boxes? The ones I see have 4 lines. Can I make them resizable somehow or define a larger size? (size="10" or resize="true" does not seem to do the trick.) Not a very pressing issue though.
+1
According to the wiki, the "size" attribute is only used on "text" input, but generalising it or adding something like "lines" would make sense here - and also for "select" parameters.
It would be really nice if the user could resize the box if they wanted to - when trying to do a multiple selection from a long file list being able to seem just four entries at a time is painful. I used to have a Firefox extension to do this to arbitrary controls - very handy.
Peter
participants (4)
-
Joachim Jacob | VIB | -
John Chilton -
Peter Cock -
Ulf Schaefer