Appropriate method of coalescing tool output (not workflow)
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing. Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis. I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions? -- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
Have you checked out the barcode splitter? The output is quite similar, an HTML report with links to the split files. -Dannon On Oct 31, 2011, at 2:50 PM, Jesse Erdmann wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi, Jesse, IMHO, the Html datatype is perfect for this. By definition, the composite datatype can have any number of files (of any type!) in the files_path, but of course, your tool or wrapper will need to generate the legal html contents of that html page but that can be done when the files_path already has the outputs - the paths Please take a look at http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-September/003311.html and feel free to ask here if anything needs clarification. On Mon, Oct 31, 2011 at 2:50 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
Thanks for the suggestions guys. I'm still struggling with a couple of things. How do I give the user a link to expose the file from extra_files_path as a history item and how would I set metadata like file format and db on the extra files? Things like PDF and text work fine, but I'd really like to be able to have BED and WIG files in the history with the UCSC genome browser link. Thanks again! On Mon, Oct 31, 2011 at 2:05 PM, Ross <ross.lazarus@gmail.com> wrote:
Hi, Jesse, IMHO, the Html datatype is perfect for this. By definition, the composite datatype can have any number of files (of any type!) in the files_path, but of course, your tool or wrapper will need to generate the legal html contents of that html page but that can be done when the files_path already has the outputs - the paths
Please take a look at http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-September/003311.html and feel free to ask here if anything needs clarification.
On Mon, Oct 31, 2011 at 2:50 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
Sorry, I think I went a little too deep into the weeds and lost my way there last night. I think the answer is to just leave the BED/WIG files visible in the first place and hide and link to the other items generated that aren't as likely to be reused is subsequent tasks. The metadata, like which genome is associated with the output should be picked up from the parent file if I'm not mistaken. I think this should work well enough for my purposes, even though it's not quite as clean in the history as I was hoping. Thanks again! Jesse On Mon, Oct 31, 2011 at 9:03 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
Thanks for the suggestions guys. I'm still struggling with a couple of things. How do I give the user a link to expose the file from extra_files_path as a history item and how would I set metadata like file format and db on the extra files? Things like PDF and text work fine, but I'd really like to be able to have BED and WIG files in the history with the UCSC genome browser link.
Thanks again!
On Mon, Oct 31, 2011 at 2:05 PM, Ross <ross.lazarus@gmail.com> wrote:
Hi, Jesse, IMHO, the Html datatype is perfect for this. By definition, the composite datatype can have any number of files (of any type!) in the files_path, but of course, your tool or wrapper will need to generate the legal html contents of that html page but that can be done when the files_path already has the outputs - the paths
Please take a look at http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-September/003311.html and feel free to ask here if anything needs clarification.
On Mon, Oct 31, 2011 at 2:50 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
On Tue, Nov 1, 2011 at 9:51 AM, Jesse Erdmann <jerdmann@umn.edu> wrote:
files visible in the first place and hide and link to the other items generated that aren't as likely to be reused is subsequent tasks. The
That sounds like a good way to use Html outputs - use the html history item to hide all the 'informational' artifacts not likely to be needed for downstream tools but expose anything likely to be needed for further tool processing as a separate history item. Note that at a pinch, users can 'upload' individual html content files by pasting their links into the upload tool URL box if they really need them as individual history items but it's a slightly ugly process so best to explicitly expose the ones likely to be useful downstream.
metadata, like which genome is associated with the output should be picked up from the parent file if I'm not mistaken. I think this
Not automatic. AFAIK You need to explicitly ask - eg if you have an input file called bar and an output file defined something like: <data format="fasta" name="foo" metadata_source="bar" /> then output file foo will have metadata copied over from input file bar
should work well enough for my purposes, even though it's not quite as clean in the history as I was hoping. Thanks again!
Jesse
On Mon, Oct 31, 2011 at 9:03 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
Thanks for the suggestions guys. I'm still struggling with a couple of things. How do I give the user a link to expose the file from extra_files_path as a history item and how would I set metadata like file format and db on the extra files? Things like PDF and text work fine, but I'd really like to be able to have BED and WIG files in the history with the UCSC genome browser link.
Thanks again!
On Mon, Oct 31, 2011 at 2:05 PM, Ross <ross.lazarus@gmail.com> wrote:
Hi, Jesse, IMHO, the Html datatype is perfect for this. By definition, the composite datatype can have any number of files (of any type!) in the files_path, but of course, your tool or wrapper will need to generate the legal html contents of that html page but that can be done when the files_path already has the outputs - the paths
Please take a look at http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-September/003311.html and feel free to ask here if anything needs clarification.
On Mon, Oct 31, 2011 at 2:50 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
On Tue, Nov 1, 2011 at 9:07 AM, Ross <ross.lazarus@gmail.com> wrote:
On Tue, Nov 1, 2011 at 9:51 AM, Jesse Erdmann <jerdmann@umn.edu> wrote:
files visible in the first place and hide and link to the other items generated that aren't as likely to be reused is subsequent tasks. The
That sounds like a good way to use Html outputs - use the html history item to hide all the 'informational' artifacts not likely to be needed for downstream tools but expose anything likely to be needed for further tool processing as a separate history item.
Note that at a pinch, users can 'upload' individual html content files by pasting their links into the upload tool URL box if they really need them as individual history items but it's a slightly ugly process so best to explicitly expose the ones likely to be useful downstream.
Ah, that gives me an idea (I was going to describe it as good, but I'm not sure that would be correct :), I had been trying to use the dataset/unhide URL to unhide the item in the history, but that doesn't work with the filename in extra_files. If I were to just embed the properly constructed upload url as a link that might be easier on the user than describing how to unhide it just in case there was something they wanted to use for some reason.
metadata, like which genome is associated with the output should be picked up from the parent file if I'm not mistaken. I think this
Not automatic. AFAIK You need to explicitly ask - eg if you have an input file called bar and an output file defined something like:
<data format="fasta" name="foo" metadata_source="bar" />
then output file foo will have metadata copied over from input file bar
Yep, that's what I meant. Meaning that if I set that on my HTML file, the extra_files would hopefully pick up the metadata from the metadata_source.
should work well enough for my purposes, even though it's not quite as clean in the history as I was hoping. Thanks again!
Jesse
On Mon, Oct 31, 2011 at 9:03 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
Thanks for the suggestions guys. I'm still struggling with a couple of things. How do I give the user a link to expose the file from extra_files_path as a history item and how would I set metadata like file format and db on the extra files? Things like PDF and text work fine, but I'd really like to be able to have BED and WIG files in the history with the UCSC genome browser link.
Thanks again!
On Mon, Oct 31, 2011 at 2:05 PM, Ross <ross.lazarus@gmail.com> wrote:
Hi, Jesse, IMHO, the Html datatype is perfect for this. By definition, the composite datatype can have any number of files (of any type!) in the files_path, but of course, your tool or wrapper will need to generate the legal html contents of that html page but that can be done when the files_path already has the outputs - the paths
Please take a look at http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-September/003311.html and feel free to ask here if anything needs clarification.
On Mon, Oct 31, 2011 at 2:50 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
Hi, Jesse.
Yep, that's what I meant. Meaning that if I set that on my HTML file, the extra_files would hopefully pick up the metadata from the metadata_source.
AFAIK, not possible. No metadata is stored anywhere in the Galaxy database tables for any of the files *inside* an html object files_path and Galaxy makes no assumptions based on the html object metadata - I'm not sure it would be a good idea to change that current loose coupling. The contents of the files_path are just a collection of arbitrary files on disk as far as Galaxy is concerned and are displayed by guessing their mimetypes from their extensions - eg <a href="foo.html">foo</a> will generally display foo as html as expected when the link is clicked...
work with the filename in extra_files. If I were to just embed the properly constructed upload url as a link that might be easier on the user than describing how to unhide it just in case there was something they wanted to use for some reason.
Yes. All you need is html that correctly points to the items in files_path because that gives the user the URL they need in the html object display. eg <a href="foo.html">foo page of html</a> for a file called foo.html in the files_path containing valid html. If foo.html displays correctly when that link is selected, then the link is all the user needs - right click, 'copy link address' and then paste it into the upload URL box to move that arbitrary file into a history as a separate object - and like any other upload, metadata may need tweaking after the upload - which is why artifacts likely needed downstream are best written as separate history items with metadata correctly set. On Tue, Nov 1, 2011 at 11:03 AM, Jesse Erdmann <jerdmann@umn.edu> wrote:
On Tue, Nov 1, 2011 at 9:07 AM, Ross <ross.lazarus@gmail.com> wrote:
On Tue, Nov 1, 2011 at 9:51 AM, Jesse Erdmann <jerdmann@umn.edu> wrote:
files visible in the first place and hide and link to the other items generated that aren't as likely to be reused is subsequent tasks. The
That sounds like a good way to use Html outputs - use the html history item to hide all the 'informational' artifacts not likely to be needed for downstream tools but expose anything likely to be needed for further tool processing as a separate history item.
Note that at a pinch, users can 'upload' individual html content files by pasting their links into the upload tool URL box if they really need them as individual history items but it's a slightly ugly process so best to explicitly expose the ones likely to be useful downstream.
Ah, that gives me an idea (I was going to describe it as good, but I'm not sure that would be correct :), I had been trying to use the dataset/unhide URL to unhide the item in the history, but that doesn't work with the filename in extra_files. If I were to just embed the properly constructed upload url as a link that might be easier on the user than describing how to unhide it just in case there was something they wanted to use for some reason.
metadata, like which genome is associated with the output should be picked up from the parent file if I'm not mistaken. I think this
Not automatic. AFAIK You need to explicitly ask - eg if you have an input file called bar and an output file defined something like:
<data format="fasta" name="foo" metadata_source="bar" />
then output file foo will have metadata copied over from input file bar
Yep, that's what I meant. Meaning that if I set that on my HTML file, the extra_files would hopefully pick up the metadata from the metadata_source.
should work well enough for my purposes, even though it's not quite as clean in the history as I was hoping. Thanks again!
Jesse
On Mon, Oct 31, 2011 at 9:03 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
Thanks for the suggestions guys. I'm still struggling with a couple of things. How do I give the user a link to expose the file from extra_files_path as a history item and how would I set metadata like file format and db on the extra files? Things like PDF and text work fine, but I'd really like to be able to have BED and WIG files in the history with the UCSC genome browser link.
Thanks again!
On Mon, Oct 31, 2011 at 2:05 PM, Ross <ross.lazarus@gmail.com> wrote:
Hi, Jesse, IMHO, the Html datatype is perfect for this. By definition, the composite datatype can have any number of files (of any type!) in the files_path, but of course, your tool or wrapper will need to generate the legal html contents of that html page but that can be done when the files_path already has the outputs - the paths
Please take a look at http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-September/003311.html and feel free to ask here if anything needs clarification.
On Mon, Oct 31, 2011 at 2:50 PM, Jesse Erdmann <jerdmann@umn.edu> wrote:
I have an in-house Perl tool that generates a double digit number of outputs that we'd like the user to have access to, but don't necessarily need to be in the history. A complicating factor is that the exact number is determined at run time as user input can add any number of outputs that can't be determined in the scope of the tools' XML processing.
Ideally, the end result would be that executing the tool would produce a single output in the history with the other files either being hidden or in the extra_files path. The visible file should be an HTML file that can be used to describe and link to the other outputs as well as providing the option to either unhide or add the hidden outputs to the history for use in subsequent analysis.
I've looked at composite datatypes, but as far as I can tell the documentation and examples don't cover how to produce a composite datatype with a variable number of outputs. I briefly tried to use the deprecated <code> tag to iterate over the results with the other outputs added as hidden extra files to the index after execution, but wasn't really excited about using a deprecated feature and couldn't get it working quickly, regardless. Any other suggestions?
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123 ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
-- Jesse Erdmann Bioinformatics Analyst Masonic Cancer Center University of Minnesota jerdmann@umn.edu 612-626-3123
-- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
participants (3)
-
Dannon Baker -
Jesse Erdmann -
Ross