July 2013 - galaxy-dev - lists.galaxyproject.org

Several jobs stalled in the queue: Job runner assigned but no external ID recorded
by Derrick Lin 04 Feb '14

04 Feb '14

Hi guys, I have several jobs have been stuck in the queue for long time, I checked the handler log and found the following message: Job runner assigned but no external ID recorded, adding to the job handler queue Is there a way that I can clean these orphan jobs out of the queue? Cheers, Derrick

1 1

A new tool_data_table_conf.xml created under tool-data directory
by Derrick Lin 17 Jan '14

17 Jan '14

Hi guys, After installing the BWA from the Tools Shed, I find there is a new tool_data_table_conf.xml got created under tool-data directory. The file contains the entries of BWA's loc files. I believe the installed BWA still relies on the tool_data_table_conf.xml under galaxy root dir. Anyone can clarify what the one under tool-data dir is for? Regards, Derrick

3 7

FTP Problem: Users cannot see their uploads files.
by Misharl mon 07 Dec '13

07 Dec '13

Hi everybody, I have managed to get proftpd to work, it can connect to the galaxy sql database, and users can log to upload files in their directory. But there is a problem, when a galaxy user logs in the galaxy web platform, the user can't see his upload files since Galaxy doesn't have the rights to open the directory. How can i change the permissions in proftpd so that galaxy can open the user directory? Thanks in advance to all. Mish

4 17

Missing test results on (Test) Tool Shed
by Peter Cock 29 Nov '13

29 Nov '13

Hi Greg & Dave, I really like the new features on the (Test) Tool Shed for searching my repositories: * Latest revision missing tool tests * Latest revision failing tool tests * Latest revision all tool tests pass However there are some teething problems. Some of my tools are listed under "Latest revision failing tool tests", but when I go to look at them, no test results are shown (passing or failing): http://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3 http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip In some cases there is indeed a failing test, for instance this is due to a bug in the test framework: http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://trello.com/c/KdGX3hkh And here there are missing dependencies (due to restrictive licensing problems): http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp https://trello.com/card/-/506338ce32ae458f6d15e4b3/770 Peter

4 77

Compressed data files in Galaxy? (e.g. GZIP or BGZF)
by Peter Cock 26 Nov '13

26 Nov '13

Hello all, What is the current status in Galaxy for supporting compressed files? We've talked about this before, for example in addition to FASTQ, many of us have expressed a wish to work with gzipped FASTQ. I understand that some have customized their local Galaxy installations to use gzipped FASTQ as a specific data type - I'm more interested in a general file format neutral solution. Also, I'd like to be able to used BGZF (not just GZIP) because it is better for random access - see for example http://blastedbio.blogspot.com/2011/11/bgzf-blocked-bigger-better-gzip.html - and makes it much easier to break up large datafiles for sharing over a cluster (i.e. it could be exploited in the current Galaxy code for splitting large sequence files). The 11 May 2012 Galaxy Development News Brief http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009757.html mentions tabix indexing - that uses bgzip, so is there something general in place yet to allow tool wrappers to say they accept not just given file formats, but different compressed versions of file formats? Ideally I'd like to be able to write an XML tool description saying a tool produced BGZF compressed tabular data, or GZIP compressed Sanger FASTQ etc. Similarly, I'd like to specify my tool accepts FASTA or gzipped FASTA (including BGZF FASTA). While for older tools if they say they accept only uncompressed FASTA, Galaxy could automatically decompress any compressed FASTA entries in my history on demand. Peter

3 4

Samtools and idxstats
by Michiel Van Bel 20 Nov '13

20 Nov '13

Hi, I would like to inquire whether anyone has attempted to implement the idxstats tool from samtools into Galaxy? The xml-file for idxstats is not present in the Galaxy source code, which led me to try and implement it myself. However, the main problem I face is that the idxstats tool silently relies on having an index file available (within the same directory) for the bam file you which to print the stats for. E.g. samtools idxstats PATH/test.bam searches for PATH/test.bam.bai and gives an error when this file is not present. And somehow I cannot model this behavior in Galaxy. A different solution would of course be to ask the author(s) of samtools to have an option available where the user can directly indicate the path to the index file. regards, Michiel PS: I've searched the mailing list archives for this problem but did not find any matches. Apologies if I somehow missed the answer. -- ================================================================== Michiel Van Bel, PhD Expert Bioinformatician Tel:+32 (0)9 331 36 95 fax:+32 (0)9 3313809 VIB Department of Plant Systems Biology, Ghent University Technologiepark 927, 9052 Gent, BELGIUM mibel(a)psb.vib-ugent.be http://www.psb.vib-ugent.be http://bioinformatics.psb.ugent.be ==================================================================

3 4

Tool unit tests using composite datatypes
by Peter Cock 18 Nov '13

18 Nov '13

Hello all, I'd like to be able to write some simple <test> entries for some of the BLAST+ tools using composite datatypes as input or output (i.e. small BLAST databases). This doesn't seem to be mentioned or hinted at on the wiki: http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax?action=show&redi… Is it possible to use a composite datatype as a test input? If so how? Normal datatypes are loaded into the test history using using the upload tool - does that mean I first need to extend the relevant datatypes to allow them to be uploaded? Example: Run blastp using a small query FASTA file and a small database, check the output (eg tabular). Is it possible to use a composite datatype as a test output? If so how? Example: Run makeblastdb using a small FASTA file, and check the output (a small BLAST database). Thanks, Peter

7 26

Contributing to genome indexes on rsync server
by Brad Chapman 08 Nov '13

08 Nov '13

Hi all; Is there a way for community members to contribute indexes to the rsync server? This resource is awesome and I'm working on migrating the CloudBioLinux retrieval scripts to use this instead of the custom S3 buckets we'd set up previously: https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/biodata/gala… It's great to have this as a public shared resource and I'd like to be able to contribute back. From an initial pass, here are the things I'd like to do: - Include bowtie2 indexes for more genomes. - Include novoalign indexes for a number of commonly used genomes. - Clean up hg19 to include a full canonically sorted hg19, with indexes. Broad has a nice version prepped so GATK will be happy with it, and you need to stick with this ordering if you're ever going to use a GATK tool on it. Right now there is a partial hg19canon (without the random/haplotype chromosomes) and the structure is a bit complex. What's the best way to contribute these? Right now I have a lot of the indexes on S3. For instance, the hg19 indexes are here: https://s3.amazonaws.com/biodata/genomes/hg19-bowtie.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bowtie2.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bwa.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-novoalign.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-seq.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-ucsc.tar.xz I'm happy to format these differently or upload somewhere that would make it easy to include. Thanks again for setting this up, I'm looking forward to working off a shared repository of data, Brad

6 8

Re: [galaxy-dev] [galaxy-user] Inquiring
by Nate Coraor 04 Nov '13

04 Nov '13

Hi Yan, I've moved this discussion to the galaxy-dev list since it pertains to a local installation of Galaxy. Responses to your questions follow, in-line. Yan Luo wrote: > Dear Sir, > > (1)We installed Galaxy, but recently the user can't registered and got the > following error, how can we fix it? > > Sever error > An error occurred. See the error logs for more information.(To turn debug on > to display ...). Since debug = False in universe_wsgi.ini, you should be able to find a more detailed error message in the log file. If starting Galaxy with: % sh run.sh --daemon The default log file is 'paster.log' in Galaxy's root directory. > (2) Could you please let me know if there is any command to stop galaxy? If starting with the --daemon flag (as above), you can use: % sh run.sh --stop-daemon If running in the foreground, you can use Ctrl-C to terminate the process. There is a recent bug whereby Ctrl-C is ineffective on some platforms under Python 2.6 - in this case you will have to kill/pkill the process manually. We are working on a fix for the latter. > (3) If I reset universe_wsgi.ini file and want to set an administrator > user(I can add a line in the above file), how can I get the password? Should > I stop galaxy(See question 2) first? then run "./setup.sh" and "./run.sh". setup.sh would have only been necessary prior to running Galaxy the first time, however, this step has recently been removed. If you are referencing documentation that still refers to setup.sh, please let us know so we can update it - I did notice this was still on the "Production Server" page, so I removed it from there. You no longer need to run setup.sh at all. > (4) If I run "setup.sh", will a new file "universe_wsgi.ini" be generated? > if I want to change this file,should I edit it before "run.sh" and after > "setup.sh". Is it right? setup.sh and its replacements in run.sh and the Galaxy application itself never overwrite files, they only create files from sample files if they do not exist. > (5) I read some of your docs, command "sh setup.sh"(sh run.sh) and > "./setup.sh"(./run.sh), which one is correct under Linux? Both syntaxes are effectively the same in most cases. --nate > > Looking forward to hearing from you. > > Best Wises, > > Yan Luo, Ph.D. > NIH > <http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=By%20the%20way,> > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

4 7

Defining $GALAXY_CPUS for use in tool wrappers
by Peter Cock 24 Oct '13

24 Oct '13

Hello all, Re: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-October/011557.html Something I raised during the GCC2013, and we talked about via Twitter as well was a Galaxy environment variable for use within Tool Wrappers setting the number of threads/CPUs to use. The idea is that you can configure a default value, and then override this per runner or per tool etc. James Taylor had suggested calling this environment variable $GALAXY_CPUS which seem fine to me (personally I'd say threads not CPUs but I don't really mind). e.g. <command>my_tool --threads "\$GALAXY_CPUS" --input "$input" --output "$output"</command> Everyone I spoke to about this seemed positive about the idea. This would/should be integrated into the various cluster back ends, for example for SGE/OGE the number of threads is already configurable via the DRMAA settings and available as the environment variable $NSLOTS for non-MPI jobs, so my guess is all Galaxy needs to do is something like this: $ hg diff diff -r ce0d758bb995 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py Tue Jul 30 12:30:30 2013 +0100 +++ b/lib/galaxy/jobs/runners/drmaa.py Tue Jul 30 16:10:40 2013 +0100 @@ -43,6 +43,7 @@ # - execute the command # - take the command's exit code ($?) and write it to a file. drm_template = """#!/bin/sh +export GALAXY_CPUS="$NSLOTS" GALAXY_LIB="%s" if [ "$GALAXY_LIB" != "None" ]; then if [ -n "$PYTHONPATH" ]; then Is there an open Trello card for this? Thanks, Peter

5 11