April 2013 - galaxy-dev - lists.galaxyproject.org

Compressed data files in Galaxy? (e.g. GZIP or BGZF)
by Peter Cock 26 Nov '13

26 Nov '13

Hello all, What is the current status in Galaxy for supporting compressed files? We've talked about this before, for example in addition to FASTQ, many of us have expressed a wish to work with gzipped FASTQ. I understand that some have customized their local Galaxy installations to use gzipped FASTQ as a specific data type - I'm more interested in a general file format neutral solution. Also, I'd like to be able to used BGZF (not just GZIP) because it is better for random access - see for example http://blastedbio.blogspot.com/2011/11/bgzf-blocked-bigger-better-gzip.html - and makes it much easier to break up large datafiles for sharing over a cluster (i.e. it could be exploited in the current Galaxy code for splitting large sequence files). The 11 May 2012 Galaxy Development News Brief http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009757.html mentions tabix indexing - that uses bgzip, so is there something general in place yet to allow tool wrappers to say they accept not just given file formats, but different compressed versions of file formats? Ideally I'd like to be able to write an XML tool description saying a tool produced BGZF compressed tabular data, or GZIP compressed Sanger FASTQ etc. Similarly, I'd like to specify my tool accepts FASTA or gzipped FASTA (including BGZF FASTA). While for older tools if they say they accept only uncompressed FASTA, Galaxy could automatically decompress any compressed FASTA entries in my history on demand. Peter

3 4

Tool unit tests using composite datatypes
by Peter Cock 18 Nov '13

18 Nov '13

Hello all, I'd like to be able to write some simple <test> entries for some of the BLAST+ tools using composite datatypes as input or output (i.e. small BLAST databases). This doesn't seem to be mentioned or hinted at on the wiki: http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax?action=show&redi… Is it possible to use a composite datatype as a test input? If so how? Normal datatypes are loaded into the test history using using the upload tool - does that mean I first need to extend the relevant datatypes to allow them to be uploaded? Example: Run blastp using a small query FASTA file and a small database, check the output (eg tabular). Is it possible to use a composite datatype as a test output? If so how? Example: Run makeblastdb using a small FASTA file, and check the output (a small BLAST database). Thanks, Peter

7 26

Contributing to genome indexes on rsync server
by Brad Chapman 08 Nov '13

08 Nov '13

Hi all; Is there a way for community members to contribute indexes to the rsync server? This resource is awesome and I'm working on migrating the CloudBioLinux retrieval scripts to use this instead of the custom S3 buckets we'd set up previously: https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/biodata/gala… It's great to have this as a public shared resource and I'd like to be able to contribute back. From an initial pass, here are the things I'd like to do: - Include bowtie2 indexes for more genomes. - Include novoalign indexes for a number of commonly used genomes. - Clean up hg19 to include a full canonically sorted hg19, with indexes. Broad has a nice version prepped so GATK will be happy with it, and you need to stick with this ordering if you're ever going to use a GATK tool on it. Right now there is a partial hg19canon (without the random/haplotype chromosomes) and the structure is a bit complex. What's the best way to contribute these? Right now I have a lot of the indexes on S3. For instance, the hg19 indexes are here: https://s3.amazonaws.com/biodata/genomes/hg19-bowtie.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bowtie2.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bwa.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-novoalign.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-seq.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-ucsc.tar.xz I'm happy to format these differently or upload somewhere that would make it easy to include. Thanks again for setting this up, I'm looking forward to working off a shared repository of data, Brad

6 8

Re: [galaxy-dev] [galaxy-user] Inquiring
by Nate Coraor 04 Nov '13

04 Nov '13

Hi Yan, I've moved this discussion to the galaxy-dev list since it pertains to a local installation of Galaxy. Responses to your questions follow, in-line. Yan Luo wrote: > Dear Sir, > > (1)We installed Galaxy, but recently the user can't registered and got the > following error, how can we fix it? > > Sever error > An error occurred. See the error logs for more information.(To turn debug on > to display ...). Since debug = False in universe_wsgi.ini, you should be able to find a more detailed error message in the log file. If starting Galaxy with: % sh run.sh --daemon The default log file is 'paster.log' in Galaxy's root directory. > (2) Could you please let me know if there is any command to stop galaxy? If starting with the --daemon flag (as above), you can use: % sh run.sh --stop-daemon If running in the foreground, you can use Ctrl-C to terminate the process. There is a recent bug whereby Ctrl-C is ineffective on some platforms under Python 2.6 - in this case you will have to kill/pkill the process manually. We are working on a fix for the latter. > (3) If I reset universe_wsgi.ini file and want to set an administrator > user(I can add a line in the above file), how can I get the password? Should > I stop galaxy(See question 2) first? then run "./setup.sh" and "./run.sh". setup.sh would have only been necessary prior to running Galaxy the first time, however, this step has recently been removed. If you are referencing documentation that still refers to setup.sh, please let us know so we can update it - I did notice this was still on the "Production Server" page, so I removed it from there. You no longer need to run setup.sh at all. > (4) If I run "setup.sh", will a new file "universe_wsgi.ini" be generated? > if I want to change this file,should I edit it before "run.sh" and after > "setup.sh". Is it right? setup.sh and its replacements in run.sh and the Galaxy application itself never overwrite files, they only create files from sample files if they do not exist. > (5) I read some of your docs, command "sh setup.sh"(sh run.sh) and > "./setup.sh"(./run.sh), which one is correct under Linux? Both syntaxes are effectively the same in most cases. --nate > > Looking forward to hearing from you. > > Best Wises, > > Yan Luo, Ph.D. > NIH > <http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=By%20the%20way,> > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

4 7

Deploying LOC files for tool built-in data during a tool installation
by Jean-Frédéric Berthelot 17 Oct '13

17 Oct '13

Hi list, The tool I am currently wrapping has built-in data, which may be used by the tool users (through a relevant < from_data_table> + .LOC file configuration). They are .fasta databases which are rather small and are thus bundled in the tool distribution package. Thanks to the tool_dependencies.xml file, said distribution package is downloaded at install time, code is compiled, and since they are here, the data files are copied to $INSTAL L_DIR too , ready to be used. After that, the user still has to edit tool-data/my_fancy_data_files.loc ; but the thing is, during the install I know where these data files are (since I copied those there), so I would like to save the user the trouble and set up this file automagically. I would have two questions: 1/ Is it okay to have tool built-in data files in $INSTAL L_DIR, or would it be considered bad practice? 2/ Is there a way to set up the tool-data/my_fancy_data_files.loc during the install? Here are the options I though of: *shipping a “real” my_fancy_data_files.loc.sample with the good paths already set-up, which is going to be copied as the .loc file (a rather ugly hack) *using more <action type="shell_command"> during install to create my_fancy_data_files.loc (but deploying this file it is not part of the tool dependency install per se) *variant of the previous : shipping my_fancy_data_files.loc as part of the tool distribution package, and copy it through shell_command (same concern than above). Any thoughts? Cheers, -- Jean-Frédéric Bonsai Bioinformatics group

5 10

Reserved variables in param tags
by Kohler Manuel 12 Sep '13

12 Sep '13

Hi, I have a question regarding the param tag. I would like to pass on the user email to a external python script. I tried to use it like this: <param name="email" type="hidden" value=$__user_email__ /> <param name="experiment" type="select" label="Experiment" help="select Experiment" refresh_on_change="true" dynamic_options="getExperiments(email)"/> This does not work. Ideally I would like to have something like this: dynamic_options="getExperiments($__user_email__)" Has someone done this before? Cheers Manuel -- Manuel Kohler Center for Information Sciences and Databases (C-ISD) Department of Biosystems Science & Engineering (D-BSSE) ETH Zurich, Maulbeerstrasse (1078, 1.02), CH-4058 Basel, +41 61 387 3132

2 1

error checking active jobs : KeyError: 'destination'
by Geert Vandeweyer 19 Aug '13

19 Aug '13

Hi, I'm getting errors in the handler logs after upgrading galaxy stable to today's release. I specified the "dynamic" job runner as default, which returns a job for pbs. These errors appear as soon as the server is up and running. What could be the issue here? galaxy.jobs.runners ERROR 2013-04-02 17:24:23,411 Unhandled exception checking active jobs Traceback (most recent call last): File "/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 336, in monitor self.check_watched_items() File "/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 382, in check_watched_items ( failures, statuses ) = self.check_all_jobs() File "/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 463, in check_all_jobs pbs_server_name = self.__get_pbs_server(pbs_job_state.job_destination.params) File "/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 221, in __get_pbs_server return job_destination_params['destination'].split('@')[-1] KeyError: 'destination' -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

3 2

Setting torque queues
by Scott Hazelhurst 14 Aug '13

14 Aug '13

[Apologies for my previous email : finger slippage meant I sent before completing the email] Dear dev I would be grateful for some assistance in setting up our galaxy server to submit jobs to our torque server.. We are using release_2013.04.01 I have tried going through the install information but clearly I¹ve misunderstood something, and unfortunately most examples come from earlier versions Here is my job_conf.xml file ‹ this is just a temporary file while I set things up. My intention is to run clustalw on our cluster using the torque queue WitsLong, assuming our torque server is galaxy.bioinf.wits.ac.za Below that is the output I get in the log. From the torque log it complains that there is no default queue specified. Any help would be appreciated. Thanks Scott <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> </plugins> <handlers default="handlers"> <handler id="main" tags="handlers,pbs"/> </handlers> <destinations default="WitsLong"> <destination id="local" runner="local"/> <destination id="WitsLong" runner="pbs"> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> <param id="-p">128</param> </destination> </destinations> <tools> <tool id="foo" handler="trackster_handler"> <param id="source">trackster</param> </tool> <tool id="clustalw" destination="WitsLong"/> </tools> </job_conf> galaxy.jobs DEBUG 2013-04-30 15:28:13,481 (33) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/33 galaxy.jobs.handler DEBUG 2013-04-30 15:28:13,527 (33) Dispatching to pbs runner 146.141.75.218, 146.141.1.81 - - [30/Apr/2013:15:28:12 +0200] "POST /galaxy/tool_runner/index HTTP/1.1" 200 - "http://galaxy.bioinf.wits.ac.za/galaxy/tool_runner/rerun?id=9752b387803d3e1 e" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-04-30 15:28:14,213 (33) Persisting job destination (destination id: WitsLong) Exception AssertionError: AssertionError('State <sqlalchemy.orm.state.InstanceState object at 0x7f79682e5150> is not present in this identity map',) in <bound method InstanceState._cleanup of <sqlalchemy.orm.state.InstanceState object at 0x7f79682e5150>> ignored galaxy.jobs.handler INFO 2013-04-30 15:28:14,522 (33) Job dispatched galaxy.tools DEBUG 2013-04-30 15:28:15,189 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2013-04-30 15:28:15,192 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:15,957 (33) submitting file /home/galaxy/galaxy-dist/database/pbs/33.sh galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:15,958 (33) command is: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/000/dataset_2.dat" -o "/home/galaxy/galaxy-dist/database/files/000/dataset_60.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/000/dataset_61.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:15,963 (33) pbs_submit failed (try 1/5), PBS error 15039: Route rejected by all destinations galaxy.jobs DEBUG 2013-04-30 15:28:16,400 (34) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/34 galaxy.jobs.handler DEBUG 2013-04-30 15:28:16,416 (34) Dispatching to pbs runner 146.141.75.218, 146.141.1.81 - - [30/Apr/2013:15:28:13 +0200] "POST /galaxy/tool_runner/index HTTP/1.1" 200 - "http://galaxy.bioinf.wits.ac.za/galaxy/tool_runner/rerun?id=9752b387803d3e1 e" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-04-30 15:28:16,668 (34) Persisting job destination (destination id: WitsLong) galaxy.jobs.handler INFO 2013-04-30 15:28:16,689 (34) Job dispatched galaxy.tools DEBUG 2013-04-30 15:28:16,937 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2013-04-30 15:28:16,938 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:17,160 (34) submitting file /home/galaxy/galaxy-dist/database/pbs/34.sh galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:17,161 (34) command is: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/000/dataset_2.dat" -o "/home/galaxy/galaxy-dist/database/files/000/dataset_62.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/000/dataset_63.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:17,162 (34) pbs_submit failed (try 1/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:17,971 (33) pbs_submit failed (try 2/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:19,166 (34) pbs_submit failed (try 2/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:19,975 (33) pbs_submit failed (try 3/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:21,170 (34) pbs_submit failed (try 3/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:21,978 (33) pbs_submit failed (try 4/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:23,173 (34) pbs_submit failed (try 4/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:23,982 (33) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:25,176 (34) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs ERROR 2013-04-30 15:28:25,984 (33) All attempts to submit job failed galaxy.jobs.runners.pbs ERROR 2013-04-30 15:28:27,179 (34) All attempts to submit job failed <table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;"> <tr> <td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td> </tr> </table

2 1

Adding GATK tools to Galaxy Cloudman
by Iry Witham 12 Aug '13

12 Aug '13

Hi Team, I am trying to get GATK tools added to my cloud instance and keep getting the following error: [Errno 2] No such file or directory: '/mnt/galaxyData/tmp/tmpszWRLP/GenomeAnalysisTKLite-2.3-9-gdcdccbb/GenomeAnalysisTK.jar'. Then when I try to install it again I get the following: All selected tool dependencies are already installed. I also get the following error: Initializing repository installation failed. Thanks, Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 3

samtools BAM to SAM tmp directory
by Matt Shirley 03 Jul '13

03 Jul '13

Is there a reason that the samtools BAM to SAM does not respect the new_file_path set in the config file? The tmp directory handling by different tool wrappers seems to be an issue right now on systems with small system tmp directories. -- Matt Shirley Ph.D Candidate - BCMB Pevsner Lab <http://pevsnerlab.kennedykrieger.org/> Johns Hopkins Medicine

3 4