- galaxy-dev - lists.galaxyproject.org

History Import/Export
by Assaf Gordon 30 Jun '10

30 Jun '10

Kudos for this nice feature. It would be very useful, both for backups and for exchanging histories between Galaxies. But as usual ;) I have couple of comments: 1. Exporting big files doesn't work. You first create the tar file, then add the individual datasets, and only then stream the tar file to the user. very bad. Try exporting a history with 18M-paired-end-76nt-reads FASTQ files (and remember that they are duplicated because you guys still insist on grooming). The proxy (apache at least) will surely timeout. BTW, since you're using native python Tarfile, the python process will keep being on 100% CPU long after the client connection timed out. 20 users exporting large histories at the same time will constitute a DDOS attack ;) 2. Importing large files (after you'll fix exporting large files...) will cause the same problems as uploading huge FASTQ files (and you're seeing more and more complaints about those). Combining 1+2, I would suggest the following: 1. Make the export a new "job". This will allow it to run for a long time, in a separate process, not blocking python and not tying the user to the browser (also remember: downloading 3GB with your browser is no fun). Since your code already store "temporary" tarballs in a dedicated "export" directory, just keep them there for an arbitrary amount of time (let say: 4 days), and provide the user will a link to download the history at his/her own convenience. 2. Allow importing a history from a URL. This will also have the benefit of not transferring huge tarballs through the users' personal computer when they want to transfer histories between Galaxies. They will go to one Galaxy, run the "export" job, get a download link, wait until the export job is done, then go to the other Galaxy, and simply import the history with that link. And a bonus: You've checked the tarball for malicious files, but not the JSON attributes. If I import a history with a dataset that have: [ { "file_name": "../../../etc/passwd" } ] I can get any file on the system into Galaxy (this happens in <galaxy>/lib/galaxy/web/controllers/history.py:580). If I import "/dev/urandom" it'll be even more fun. I would recommend against checking for ".." or "/" (as in line 457), because a determined person could possibly circumvent that with tricks like ".\u002E/etc/passwd" etc.). Instead, construct the full path (as set in the dataset attributes), then check with os.path.abspath() that it resides in the temporary import directory. Comments are welcomed, -gordon

4 5

minor error while upgrading database to version 50
by Assaf Gordon 30 Jun '10

30 Jun '10

Hi, After pulling the latest changes and upgrading the DB, I get the following error: === 45 -> 46... Migration script to create tables for handling post-job actions. done 46 -> 47... Add a user_id column to the job table. Updating user_id column in job table for 76013 rows... Updated the user_id column for 75646 rows in the job table. 367 rows have no user_id since the value was NULL in the galaxy_session table. done 47 -> 48... Add a state column to the history_dataset_association and library_dataset_dataset_association table. done 48 -> 49... Migration script to add the api_keys table. done 49 -> 50... ======================================== This script drops tables that were associated with the old Galaxy Cloud functionality. ======================================== DEBUG:0050_drop_cloud_tables:Dropping cloud tables failed: (InternalError) cannot drop table cloud_provider because other objects depend on it HINT: Use DROP ... CASCADE to drop the dependent objects too. '\nDROP TABLE cloud_provider' {} done ==== However, Galaxy runs fine afterwards. -gordon

2 3

Bowtie with FASTA and Illumina FASTQ
by Assaf Gordon 30 Jun '10

30 Jun '10

Hi all, Here's a patch that allow the Bowtie wrapper in Galaxy to use FASTA files and Illumina FASTQ files, without grooming. The input type is automatically detected (based on the dataset's type). The patch is also available here: http://cancan.cshl.edu/labmembers/gordon/files/galaxy_bowtie_wrapper_fasta_… To import it, run: hg import http://cancan.cshl.edu/labmembers/gordon/files/galaxy_bowtie_wrapper_fasta_… Comments are welcomed, -gordon

1 0

N-way splitter (poor man's parallelization)
by Assaf Gordon 30 Jun '10

30 Jun '10

Hi all, I've written a tool to split an input file into several output files. This tool can be used in workflows, has variable number of output datasets (2 to 10), and supports splitting FASTA/FASTQ files (keeping groups of 2/4 lines together when splitting). It's a stop-gap hack until Galaxy allows automatic splitting/looping in workflows. Try it out here: http://cancan.cshl.edu/publicgalaxy/root?tool_id=nway_splitter_tags Download from here: http://cancan.cshl.edu/labmembers/gordon/files/galaxy_nway_splitter.tar.bz2 Comments are welcomed, -gordon

1 0

broken links in workflow
by Michael Wilson 30 Jun '10

30 Jun '10

Dear Galaxy team, I have been making use of the workflow feature in Galaxy and I think it is an excellent way to make use of the tools and share analyses with others. Recently I have noticed that some of steps within the workflow have been disconnected due to tool changes. "Problems were encountered loading this workflow (possibly a result of tool upgrades). Please review the following parameters and then save." A week ago I reconnected the workflow steps yesterday found that some step were disconnected. I corrected this yesterday and was able to save and test it. Today however I have returned to the workflow and the again several of the connections between functions have been lost. I was planning on using and publishing a link to this workflow (i.e. for a manuscript). Is this a good idea or is it still a beta function which will not be stable? I have a bit of a complex workflow and I am not sure if it is a good idea to continue to repair it? Any feedback on how one should work with the workflows would be great. Perhaps one nice feature would be a way to freeze/lock-down a workflow so it will run inspite of future Galaxy updgrades (in addition to having a more plastic one that evolves along with the upgrades). Also a way to export the workflow in a meaningful way would be useful as it can be difficult to remember all the step once the links are broken. Thanks again for your help. Best regards, Mike

2 1

Illumina srf format...
by Davide Cittaro 30 Jun '10

30 Jun '10

Hi all, which format is appropriate for illumina srf files? As I try to upload them I get an error about inappropriate format... I thought srf was similar to sff... I guess I'll add another binary data format then :-) d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

1 0

Command tag in Galaxy
by Sumedha Ganjoo 29 Jun '10

29 Jun '10

Hi, I have been trying to do nested if-else and use logical operators to decide the executable to be called and the parameters passed to it, in a tool xml file. Is it possible to do so in Galaxy tool files? How? I have been getting errors in parsing the nested if-else. Thanks in advance. Regards, Sumedha -- Sumedha Ganjoo Graduate Assistant, Department Of Computer Science, University Of Georgia, Athens, GA , USA

2 1

Adding users under external authentication
by Davide Cittaro 29 Jun '10

29 Jun '10

Hi all, I know it doesn't make sense, but would it be possible to allow user creation even when there's exteral authentication (managed by a proxy)? I mean, suppose there's a new user and I have to give him rights on a new library... I can't do it until the user had not logged at least once... I would like to instruct galaxy there will be an user USER@DOMAIN before he logs in. d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

2 4

error message printed unescaped into html stream - cant figure out what the error message is
by Michael Pheasant 28 Jun '10

28 Jun '10

Hi I'm using yesterday's latest galaxy-dist (rev 4cdf4cca0f31) and trying to 'Upload directory of files' to a library, as admin user (see the section in http://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFiles) When this job runs, I get the success message: "Added 47 datasets to the library 'New data library test' (each is selected). Click the Go button at the bottom of this page to edit the permissions on these datasets if necessary." But all the 47 files listed have an error message in the "Information" column: "Job error (click name for more info)" When I click on the name and select "Edit Information" i get to a page about the dataset which has a partial error message in the "Info" text box: "uploaded fastq fileTraceback (most recent call last): File " Then a bit more of the message outside the text box in the frame html, the rest is lost: " __main__() File "/mnt/array/home/galaxy/galaxy-dist/tools/data_source/upload.py", line 345, in __mai" size="40"/>" *) I view the source for the frame and see that the error message from python has been injected directly into the value field of the "info" text element without any html escaping (see below). *) I cant see the error in the log, I'm running the server in the foreground using run.sh *) Line 345 of upload.py is marked below: ... if dataset.type == 'composite': files_path = output_paths[int( dataset.dataset_id )][1] add_composite_file( dataset, json_file, output_path, files_path ) else: add_file( dataset, json_file, output_path ) /* LINE 345 */ *) I cant see the message in the log either *) Source of frame with injected error msg: <div class="toolForm"> <div class="toolFormTitle">Edit attributes of s_8_1_sequence.txt</div> <div class="toolFormBody"> <form name="edit_attributes" action="/library_common/ldda_edit_info?library_id=adb5f5c93f827949&show_deleted=False&cntrller=library_admin&folder_id=adb5f5c93f827949&use_panels=False" method="post"> <input type="hidden" name="id" value="b351d620879d6814"/> <div class="form-row"> <label>Name:</label> <input type="text" name="name" value="s_8_1_sequence.txt" size="40"/> <div style="clear: both"></div> </div> <div class="form-row"> <label>Info:</label> <input type="text" name="info" value="uploaded fastq fileTraceback (most recent call last): File "/mnt/array/home/galaxy/galaxy-dist/tools/data_source/upload.py", line 353, in <module> __main__() File "/mnt/array/home/galaxy/galaxy-dist/tools/data_source/upload.py", line 345, in __mai" size="40"/> <div style="clear: both"></div> </div> Cheers Mike -- Dr Michael Pheasant Software Engineer Queensland Facility for Advanced Bioinformatics Level 6, QBP University of Queensland, QLD 4072 T: +61 (0)7 3346 2070 F: +61 (0)7 3346 2101 www.qfab.org

2 1

Admin/developer toolbar to speed up tool development.
by Renaud Gaujoux 28 Jun '10

28 Jun '10

Hi Galaxy team, for making galaxy tools development easier I made a little change to my file 'tool_form.mako': it allows to reload a tool config from the tool's page itself, using a simple ajax links. The general idea is to provide the administrators/developers with a toolbar of useful quick links. Some are some screenshots are attached. It is a very quick(dirty?) solution that does not handle error cases etc... but maybe you will find the principle interesting. The code is the following at line 201 of file 'tool_form.mako': <div class="toolFormTitle"> <table><tbody><tr> <td> %if tool.has_multiple_pages: ${tool.name} (step ${tool_state.page+1} of ${tool.npages})</div> %else: ${tool.name} %endif </td> %if trans.user and app.config.is_admin_user( trans.user ): <td><a onclick="$('.adminOptions').toggle();return false;" href="#" style="font-size: 8pt; color: rgb(48, 48, 170);">+ Options</a></td> <td style="display: none;" class="adminOptions">::</td> <td style="display: none; font-size: 8pt;" class="adminOptions"> <a class="icon-button arrow-circle tooltip" onclick="$.post('/admin/tool_reload', {tool_id:'nmf'}, function(data){ $('#result').toggle(); $('#result').html(data); $('#result').html($('#result .donemessage')); $('#result').toggle().delay(3000); location.reload();}); return false;" href="#" title="Reload config">Relaod config</a> </td> %endif </tr></tbody></table> %if trans.user and app.config.is_admin_user( trans.user ): <div id="result"></div> %endif </div> Bests, Renaud Renaud Gaujoux Computational Biology - University of Cape Town South Africa ### UNIVERSITY OF CAPE TOWN This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity. ###

1 0