Hi all,
TLDR - Last 2 lines of this email
Some time ago we created a wrapper script for breseq, a tool that generates a friendly HTML report of mutations.
This was a perfect match for Galaxy given that we could simply set the "summary.html" file to be displayed using Galaxy's HTML datatype and place all extra media files in $htmlfile.files_path.
However the friendliness quickly became a pain with large datasets. The size of the "files_path" folder quickly grew to over 500MB if multiple references are used. Still this was not the biggest problem since the size is spread over lots of tiny HTML files which work fine in Galaxy's preview.
The real nasty hit us when we found that with the HTML format Galaxy does on-the-fly compression of the whole folder if a user clicks the "download" button. Needless to say most of the time users end up getting a server time-out and we get lots of broken pipes on the logs.
An additional complication of the HTML format is that only the size of the primary html file is taken into account. So most users didn't understand why a 10KB file would cause so much pain to download and why was their quota being taken for no apparent reason.
So after this we got stuck with 2 alternatives:
1) Customize the HTML datatype to do /a priori/ zipping instead of on-the-fly. Provide this pre-zipped file if users click on the download button. Fix the HTML datatype to display the correct size of the full dataset.
2) Provide 2 outputs, the usual HTML file output and an additional "complete report" which is a pre-packaged zip file generated by the wrapper.
After spending quite some time browsing the existing documentation and trying to make sense of the code we gave up on 1). Looking at datatype class definitions, in particular HTML and the class hierarchy, we couldn't even figure out how the "files_path" mechanism worked. Documentation or lack thereof didn't help either. Knowing what methods are mandatory and what, when and how are they called would be a major help.
So we jumped on 2). Getting the pre-packaged zip was quick and easy. We had to create a zip datatype but the existing Binary class worked fine:
<datatype extension="zip" type="galaxy.datatypes.binary:Binary" mimetype="application/zip" subclass="True" />
The second part of the problem was to disable the HTML "download" link. And now we are back into customizing the HTML datatype...
So long story short, if I want to customize what happens when a user clicks the download link of a given format, which methods should I be looking at?
Can you think of any alternative (better) way to solve the original problem?
Thanks, Renato