Hi Nick, If you're running your own local instance, then nothing is impossible - it's just a bit ugly... Nate Coraor wrote, On 02/21/2011 12:36 PM:
[...] many tools that operate on these files produces multiple, uncompressed large files which quickly eat up the disk allocation. [...] With no way to compress or archive files produced by intermediate steps in a workflow, [...]
Here's a tool that compresses an input galaxy dataset and then deletes the input file. Deleting the input dataset from underneath galaxy's feet obviously goes against everything galaxy stands for, and I'm sure the Galaxy team does not endorse such solutions. It will also slightly make your database out-of-sync with the real files on the disk. But hey - desperate times call for desperate means :) ======== <tool id="cshl_compress_input" name="Compress Input File"> <description>for advanced users only!</description> <command>gzip -c '$input' > '$output' && rm '$input'</command> <inputs> <param format="data" name="input" type="data" label="Dataset to Compress" /> <param format="data" name="waitforinput" type="data" label="Tool to wait for" /> </inputs> <outputs> <data format="gzip" name="output" /> </outputs> <help> **What it does** DO NOT USE THIS TOOL UNLESS YOU KNOW WHAT YOU'RE DOING. </help> </tool> ============ The second "input" parameter in this tool is there only to force this tool to run after another tool (which needs the uncompressed input file) - you should connect this tool carefully in your workflow. Making the output format "gzip" ensures the new compressed files can't be used with any regular tool. Then create a similar "uncompress" tool that does the opposite.
And this brings me to the second problem. Getting large files out of Galaxy. The only way to save large files from Galaxy (that I can see) is the save icon, which downloads the file via http. This take *ages* for a large file and also causes big headaches for my firefox browser.
Here are three solutions (in varying level of ugliness) to get files out of galaxy: 1. This simple tool will tell you the full path of your dataset: ========= <tool id="cshl_get_dataset_full_path" name="Get dataset full path"> <description>for advanced users only!</description> <command>readlink -f '$input' > '$output'</command> <inputs> <param format="data" name="input" type="data" label="Show full path of dataset" /> </inputs> <outputs> <data format="txt" name="output" /> </outputs> <help> **What it does** DO NOT USE THIS TOOL UNLESS YOU KNOW WHAT YOU'RE DOING. </help> </tool> ========= run it on any input dataset, the output will contain the full path on your local system. It goes without saying that this is a security hazard, and only use this tool if you know what you're doing, and you trust your users. Once you have the full path, just access the file directly out of galaxy. 2. The following tool allows the user to export a dataset into a hard-coded directory (/tmp/galaxy_export in this example). This is just a proof of concept, and for a production environment you'll need to add validators to the "description" variable to prevent users from adding unwanted characters. But it works - once the tool is run, the selected dataset will appear under /tmp/galaxy_export/$USER/ . ============= <tool id="cshl_export_to_local" name="Export to local file"> <description>for advanced users only!</description> <command> mkdir -p /tmp/galaxy_export/$userEmail && ln -s '$input' '/tmp/galaxy_export/$userEmail/${input.hid}_${description}.${input.extension}' </command> <inputs> <param format="data" name="input" type="data" label="Dataset to Export" /> <param name="description" type="text" size="30" label="File name" /> </inputs> <outputs> <data format="txt" name="output" /> </outputs> <help> **What it does** DO NOT USE THIS TOOL UNLESS YOU KNOW WHAT YOU'RE DOING. </help> </tool> ================ 3. Last but not least, if you have access to the database, getting the dataset path is easy if you now the dataset number or the dataset hash-id (and you have them as links on the galaxy web page). This solution is not for the faint of heart, but if you want I show examples of how to get from one to the other. -gordon