Hi all,
TLDR - Last 2 lines of this email
Some time ago we created a wrapper script for breseq, a tool that generates a friendly HTML report of mutations.
This was a perfect match for Galaxy given that we could simply set the "summary.html" file to be displayed using Galaxy's HTML datatype and place all extra media files in $htmlfile.files_path.
However the friendliness quickly became a pain with large datasets. The size of the "files_path" folder quickly grew to over 500MB if multiple references are used. Still this was not the biggest problem since the size is spread over lots of tiny HTML files which work fine in Galaxy's preview.
The real nasty hit us when we found that with the HTML format Galaxy does on-the-fly compression of the whole folder if a user clicks the "download" button. Needless to say most of the time users end up getting a server time-out and we get lots of broken pipes on the logs.
An additional complication of the HTML format is that only the size of the primary html file is taken into account. So most users didn't understand why a 10KB file would cause so much pain to download and why was their quota being taken for no apparent reason.
So after this we got stuck with 2 alternatives:
1) Customize the HTML datatype to do /a priori/ zipping instead of on-the-fly. Provide this pre-zipped file if users click on the download button. Fix the HTML datatype to display the correct size of the full dataset.
2) Provide 2 outputs, the usual HTML file output and an additional "complete report" which is a pre-packaged zip file generated by the wrapper.
After spending quite some time browsing the existing documentation and trying to make sense of the code we gave up on 1). Looking at datatype class definitions, in particular HTML and the class hierarchy, we couldn't even figure out how the "files_path" mechanism worked. Documentation or lack thereof didn't help either. Knowing what methods are mandatory and what, when and how are they called would be a major help.
So we jumped on 2). Getting the pre-packaged zip was quick and easy. We had to create a zip datatype but the existing Binary class worked fine:
<datatype extension="zip" type="galaxy.datatypes.binary:Binary" mimetype="application/zip" subclass="True" />
The second part of the problem was to disable the HTML "download" link. And now we are back into customizing the HTML datatype...
So long story short, if I want to customize what happens when a user clicks the download link of a given format, which methods should I be looking at?
Can you think of any alternative (better) way to solve the original problem?
Thanks, Renato
Hi Renato,
You probably need to look at overriding the display_data method in your subclass, see here for a simple example to render a binary composte file type as a simple text caption:
https://github.com/peterjc/galaxy_blast/blob/master/datatypes/blast_datatype...
You can also override _archive_main_file to change the name of he default main HTML file generated, or _archive_composite_dataset for full control - but these are not really intended as public interfaces.
(I did play with this a while back, but right now I forget which branch this was on...)
Peter
On Thu, Jul 17, 2014 at 6:53 PM, Renato Alves rjalves@igc.gulbenkian.pt wrote:
Hi all,
TLDR - Last 2 lines of this email
Some time ago we created a wrapper script for breseq, a tool that generates a friendly HTML report of mutations.
This was a perfect match for Galaxy given that we could simply set the "summary.html" file to be displayed using Galaxy's HTML datatype and place all extra media files in $htmlfile.files_path.
However the friendliness quickly became a pain with large datasets. The size of the "files_path" folder quickly grew to over 500MB if multiple references are used. Still this was not the biggest problem since the size is spread over lots of tiny HTML files which work fine in Galaxy's preview.
The real nasty hit us when we found that with the HTML format Galaxy does on-the-fly compression of the whole folder if a user clicks the "download" button. Needless to say most of the time users end up getting a server time-out and we get lots of broken pipes on the logs.
An additional complication of the HTML format is that only the size of the primary html file is taken into account. So most users didn't understand why a 10KB file would cause so much pain to download and why was their quota being taken for no apparent reason.
So after this we got stuck with 2 alternatives:
- Customize the HTML datatype to do /a priori/ zipping instead of
on-the-fly. Provide this pre-zipped file if users click on the download button. Fix the HTML datatype to display the correct size of the full dataset.
- Provide 2 outputs, the usual HTML file output and an additional
"complete report" which is a pre-packaged zip file generated by the wrapper.
After spending quite some time browsing the existing documentation and trying to make sense of the code we gave up on 1). Looking at datatype class definitions, in particular HTML and the class hierarchy, we couldn't even figure out how the "files_path" mechanism worked. Documentation or lack thereof didn't help either. Knowing what methods are mandatory and what, when and how are they called would be a major help.
So we jumped on 2). Getting the pre-packaged zip was quick and easy. We had to create a zip datatype but the existing Binary class worked fine:
<datatype extension="zip" type="galaxy.datatypes.binary:Binary" mimetype="application/zip" subclass="True" />
The second part of the problem was to disable the HTML "download" link. And now we are back into customizing the HTML datatype...
So long story short, if I want to customize what happens when a user clicks the download link of a given format, which methods should I be looking at?
Can you think of any alternative (better) way to solve the original problem?
Thanks, Renato
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Peter,
Thanks for the help.
I managed to do what I wanted by subclassing the Html datatype, defining it as composite and overriding _archive_composite_dataset to link to the pre-packaged zip file.
Defining it as composite was necessary as "html" seems to be treated specially in display_data() and added manually to the composite_types list.
Best, Renato
Quoting Peter Cock on 17-07-2014 20:19:
Hi Renato,
You probably need to look at overriding the display_data method in your subclass, see here for a simple example to render a binary composte file type as a simple text caption:
https://github.com/peterjc/galaxy_blast/blob/master/datatypes/blast_datatype...
You can also override _archive_main_file to change the name of he default main HTML file generated, or _archive_composite_dataset for full control - but these are not really intended as public interfaces.
(I did play with this a while back, but right now I forget which branch this was on...)
Peter
On Thu, Jul 17, 2014 at 6:53 PM, Renato Alves rjalves@igc.gulbenkian.pt wrote:
Hi all,
TLDR - Last 2 lines of this email
Some time ago we created a wrapper script for breseq, a tool that generates a friendly HTML report of mutations.
This was a perfect match for Galaxy given that we could simply set the "summary.html" file to be displayed using Galaxy's HTML datatype and place all extra media files in $htmlfile.files_path.
However the friendliness quickly became a pain with large datasets. The size of the "files_path" folder quickly grew to over 500MB if multiple references are used. Still this was not the biggest problem since the size is spread over lots of tiny HTML files which work fine in Galaxy's preview.
The real nasty hit us when we found that with the HTML format Galaxy does on-the-fly compression of the whole folder if a user clicks the "download" button. Needless to say most of the time users end up getting a server time-out and we get lots of broken pipes on the logs.
An additional complication of the HTML format is that only the size of the primary html file is taken into account. So most users didn't understand why a 10KB file would cause so much pain to download and why was their quota being taken for no apparent reason.
So after this we got stuck with 2 alternatives:
- Customize the HTML datatype to do /a priori/ zipping instead of
on-the-fly. Provide this pre-zipped file if users click on the download button. Fix the HTML datatype to display the correct size of the full dataset.
- Provide 2 outputs, the usual HTML file output and an additional
"complete report" which is a pre-packaged zip file generated by the wrapper.
After spending quite some time browsing the existing documentation and trying to make sense of the code we gave up on 1). Looking at datatype class definitions, in particular HTML and the class hierarchy, we couldn't even figure out how the "files_path" mechanism worked. Documentation or lack thereof didn't help either. Knowing what methods are mandatory and what, when and how are they called would be a major help.
So we jumped on 2). Getting the pre-packaged zip was quick and easy. We had to create a zip datatype but the existing Binary class worked fine:
<datatype extension="zip" type="galaxy.datatypes.binary:Binary" mimetype="application/zip" subclass="True" />
The second part of the problem was to disable the HTML "download" link. And now we are back into customizing the HTML datatype...
So long story short, if I want to customize what happens when a user clicks the download link of a given format, which methods should I be looking at?
Can you think of any alternative (better) way to solve the original problem?
Thanks, Renato
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Peter Cock -
Renato Alves