September 2013 - galaxy-dev - lists.galaxyproject.org

FTP Problem: Users cannot see their uploads files.
by Misharl mon 07 Dec '13

07 Dec '13

Hi everybody, I have managed to get proftpd to work, it can connect to the galaxy sql database, and users can log to upload files in their directory. But there is a problem, when a galaxy user logs in the galaxy web platform, the user can't see his upload files since Galaxy doesn't have the rights to open the directory. How can i change the permissions in proftpd so that galaxy can open the user directory? Thanks in advance to all. Mish

4 17

Missing test results on (Test) Tool Shed
by Peter Cock 29 Nov '13

29 Nov '13

Hi Greg & Dave, I really like the new features on the (Test) Tool Shed for searching my repositories: * Latest revision missing tool tests * Latest revision failing tool tests * Latest revision all tool tests pass However there are some teething problems. Some of my tools are listed under "Latest revision failing tool tests", but when I go to look at them, no test results are shown (passing or failing): http://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3 http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip In some cases there is indeed a failing test, for instance this is due to a bug in the test framework: http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go https://trello.com/c/KdGX3hkh And here there are missing dependencies (due to restrictive licensing problems): http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp https://trello.com/card/-/506338ce32ae458f6d15e4b3/770 Peter

4 77

Compressed data files in Galaxy? (e.g. GZIP or BGZF)
by Peter Cock 26 Nov '13

26 Nov '13

Hello all, What is the current status in Galaxy for supporting compressed files? We've talked about this before, for example in addition to FASTQ, many of us have expressed a wish to work with gzipped FASTQ. I understand that some have customized their local Galaxy installations to use gzipped FASTQ as a specific data type - I'm more interested in a general file format neutral solution. Also, I'd like to be able to used BGZF (not just GZIP) because it is better for random access - see for example http://blastedbio.blogspot.com/2011/11/bgzf-blocked-bigger-better-gzip.html - and makes it much easier to break up large datafiles for sharing over a cluster (i.e. it could be exploited in the current Galaxy code for splitting large sequence files). The 11 May 2012 Galaxy Development News Brief http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009757.html mentions tabix indexing - that uses bgzip, so is there something general in place yet to allow tool wrappers to say they accept not just given file formats, but different compressed versions of file formats? Ideally I'd like to be able to write an XML tool description saying a tool produced BGZF compressed tabular data, or GZIP compressed Sanger FASTQ etc. Similarly, I'd like to specify my tool accepts FASTA or gzipped FASTA (including BGZF FASTA). While for older tools if they say they accept only uncompressed FASTA, Galaxy could automatically decompress any compressed FASTA entries in my history on demand. Peter

3 4

Samtools and idxstats
by Michiel Van Bel 20 Nov '13

20 Nov '13

Hi, I would like to inquire whether anyone has attempted to implement the idxstats tool from samtools into Galaxy? The xml-file for idxstats is not present in the Galaxy source code, which led me to try and implement it myself. However, the main problem I face is that the idxstats tool silently relies on having an index file available (within the same directory) for the bam file you which to print the stats for. E.g. samtools idxstats PATH/test.bam searches for PATH/test.bam.bai and gives an error when this file is not present. And somehow I cannot model this behavior in Galaxy. A different solution would of course be to ask the author(s) of samtools to have an option available where the user can directly indicate the path to the index file. regards, Michiel PS: I've searched the mailing list archives for this problem but did not find any matches. Apologies if I somehow missed the answer. -- ================================================================== Michiel Van Bel, PhD Expert Bioinformatician Tel:+32 (0)9 331 36 95 fax:+32 (0)9 3313809 VIB Department of Plant Systems Biology, Ghent University Technologiepark 927, 9052 Gent, BELGIUM mibel(a)psb.vib-ugent.be http://www.psb.vib-ugent.be http://bioinformatics.psb.ugent.be ==================================================================

3 4

Tool unit tests using composite datatypes
by Peter Cock 18 Nov '13

18 Nov '13

Hello all, I'd like to be able to write some simple <test> entries for some of the BLAST+ tools using composite datatypes as input or output (i.e. small BLAST databases). This doesn't seem to be mentioned or hinted at on the wiki: http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax?action=show&redi… Is it possible to use a composite datatype as a test input? If so how? Normal datatypes are loaded into the test history using using the upload tool - does that mean I first need to extend the relevant datatypes to allow them to be uploaded? Example: Run blastp using a small query FASTA file and a small database, check the output (eg tabular). Is it possible to use a composite datatype as a test output? If so how? Example: Run makeblastdb using a small FASTA file, and check the output (a small BLAST database). Thanks, Peter

7 26

Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
by Dave Bouvier 14 Nov '13

14 Nov '13

Peter, I apologize for the delay, here is the information you requested: ubuntu@ip-10-0-0-243:~$ uname -a Linux ip-10-0-0-243 3.8.0-25-generic #37-Ubuntu SMP Thu Jun 6 20:47:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ubuntu@ip-10-0-0-243:~$ arch x86_64 ubuntu@ip-10-0-0-243:~$ echo $0 -bash --Dave B. On 08/26/2013 09:44 AM, Peter Cock wrote: > (Off list) > > Hi Dave, Greg, > > Just a reminder - without a bit more information about the Tool Shed > cluster I am stuck about how to debug this install issue. > > Thanks, > > Peter > > > On Wed, Aug 21, 2013 at 2:05 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> On Wed, Aug 21, 2013 at 1:45 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >>> Peter, >>> >>> Here is the relevant log output for the installation of ncbi_blast_plus. A >>> quick google informs me that exit code 64 means "command line usage error", >>> for what it's worth. >>> >>> ... >>> tool_shed.util.shed_util_common DEBUG 2013-08-20 23:34:47,886 Adding new row >>> (or updating an existing row) for repository 'package_blast_plus_2_2_26' in >>> the tool_shed_repository table, status set to 'New'. >>> >>> Warning: local() encountered an error (return code 64) while executing ' >>> if [[ "$(uname)" == "Linux" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi && >>> if [[ "$(arch)" == "x86_64" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then export >>> FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi && >>> echo Fetching $FILENAME && >>> if [[ "$(uname)" == "Linux" ]]; then wget >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> if [[ "$(uname)" == "Darwin" ]]; then curl -O >>> ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi && >>> tar -zxvf $FILENAME && >>> echo Downloaded and decompressed >>> ' >>> >> >> Thanks Dave, >> >> I've made a note on this Trello card about exposing the log information >> of unsuccessful builds on the Tool Shed, >> https://trello.com/c/HVGrShnC/1042-tool-shed-should-test-installation-of-pa… >> >> Can you tell me what version of Linux this is, and which shell? >> >> $ uname >> ... >> >> $ arch >> ... >> >> $ echo $0 >> ... >> >> That might give me a clue about the nature of the failure. My >> hunch is that I am using something bash specific. >> >> The simplest solution would be if you could run each of these >> command by hand to see if it triggers the same error: >> >> if [[ "$(uname)" == "Linux" ]]; then export FILENAME="ncbi-blast-2.2.26+-ia32-linux.tar.gz"; fi >> if [[ "$(arch)" == "x86_64" ]]; then export FILENAME="ncbi-blast-2.2.26+-x64-linux.tar.gz"; fi >> if [[ "$(uname)" == "Darwin" ]]; then export FILENAME="ncbi-blast-2.2.26+-universal-macosx.tar.gz"; fi >> echo Fetching $FILENAME >> if [[ "$(uname)" == "Linux" ]]; then wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> if [[ "$(uname)" == "Darwin" ]]; then curl -O ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.2.26/$FILENAME; fi >> tar -zxvf $FILENAME >> echo Downloaded and decompressed >> >> Thanks, >> >> Peter

9 23

Contributing to genome indexes on rsync server
by Brad Chapman 08 Nov '13

08 Nov '13

Hi all; Is there a way for community members to contribute indexes to the rsync server? This resource is awesome and I'm working on migrating the CloudBioLinux retrieval scripts to use this instead of the custom S3 buckets we'd set up previously: https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/biodata/gala… It's great to have this as a public shared resource and I'd like to be able to contribute back. From an initial pass, here are the things I'd like to do: - Include bowtie2 indexes for more genomes. - Include novoalign indexes for a number of commonly used genomes. - Clean up hg19 to include a full canonically sorted hg19, with indexes. Broad has a nice version prepped so GATK will be happy with it, and you need to stick with this ordering if you're ever going to use a GATK tool on it. Right now there is a partial hg19canon (without the random/haplotype chromosomes) and the structure is a bit complex. What's the best way to contribute these? Right now I have a lot of the indexes on S3. For instance, the hg19 indexes are here: https://s3.amazonaws.com/biodata/genomes/hg19-bowtie.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bowtie2.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-bwa.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-novoalign.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-seq.tar.xz https://s3.amazonaws.com/biodata/genomes/hg19-ucsc.tar.xz I'm happy to format these differently or upload somewhere that would make it easy to include. Thanks again for setting this up, I'm looking forward to working off a shared repository of data, Brad

6 8

Re: [galaxy-dev] [galaxy-user] Inquiring
by Nate Coraor 04 Nov '13

04 Nov '13

Hi Yan, I've moved this discussion to the galaxy-dev list since it pertains to a local installation of Galaxy. Responses to your questions follow, in-line. Yan Luo wrote: > Dear Sir, > > (1)We installed Galaxy, but recently the user can't registered and got the > following error, how can we fix it? > > Sever error > An error occurred. See the error logs for more information.(To turn debug on > to display ...). Since debug = False in universe_wsgi.ini, you should be able to find a more detailed error message in the log file. If starting Galaxy with: % sh run.sh --daemon The default log file is 'paster.log' in Galaxy's root directory. > (2) Could you please let me know if there is any command to stop galaxy? If starting with the --daemon flag (as above), you can use: % sh run.sh --stop-daemon If running in the foreground, you can use Ctrl-C to terminate the process. There is a recent bug whereby Ctrl-C is ineffective on some platforms under Python 2.6 - in this case you will have to kill/pkill the process manually. We are working on a fix for the latter. > (3) If I reset universe_wsgi.ini file and want to set an administrator > user(I can add a line in the above file), how can I get the password? Should > I stop galaxy(See question 2) first? then run "./setup.sh" and "./run.sh". setup.sh would have only been necessary prior to running Galaxy the first time, however, this step has recently been removed. If you are referencing documentation that still refers to setup.sh, please let us know so we can update it - I did notice this was still on the "Production Server" page, so I removed it from there. You no longer need to run setup.sh at all. > (4) If I run "setup.sh", will a new file "universe_wsgi.ini" be generated? > if I want to change this file,should I edit it before "run.sh" and after > "setup.sh". Is it right? setup.sh and its replacements in run.sh and the Galaxy application itself never overwrite files, they only create files from sample files if they do not exist. > (5) I read some of your docs, command "sh setup.sh"(sh run.sh) and > "./setup.sh"(./run.sh), which one is correct under Linux? Both syntaxes are effectively the same in most cases. --nate > > Looking forward to hearing from you. > > Best Wises, > > Yan Luo, Ph.D. > NIH > <http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=By%20the%20way,> > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

4 7

Running functional tests is too slow
by Peter Cock 31 Oct '13

31 Oct '13

Hi all, I'd like to echo Florent's concern that run_functional_tests.sh is too slow, and that this discourages Tool Authors from adding more tests to their tools: https://trello.com/c/wL21d2do/1017-functional-tests-take-too-long (And encourage people to vote up that issue ;) ) Florent has identified one clear bottleneck is the creation of a fresh SQLite database each time, upon which the growing number of schema migration calls must be performed. Could we not cache an empty but up to date copy of this database? i.e. Something like this: 1. run_functional_tests.sh starts 2. If it exists, copy empty_test_sqlite_database.db, otherwise create new test SQLite database as now. 3. Check the schema version. 4. If the temporary SQlite database is out of date, run the migration, and then save a copy of this now up to date database as empty_test_sqlite_database.db, 5. run the tests Perhaps the empty SQLite database could even be cached in BitBucket too, for a faster first test run with a clean checkout? Separately, running the tests themselves seems overly slow - can anything be tweaked here? For example, is there any point executing the external set_meta script in the test environment? Regards, Peter P.S. On a related note, my achievement last Friday was to get TravisCI doing continuous integration testing of a GitHub repository of Galaxy tools: https://travis-ci.org/peterjc/galaxy_blast https://github.com/peterjc/galaxy_blast For those not familiar with this system, the idea is that via a special .travis.yml configuration file, each time new commits are pushed to GitHub, the latest code is checked out and tested on a virtual machine. i.e. Continuous in the sense of running the tests after each change, rather than just once a night. Right now the tests for the BLAST+ suite and associated tool wrappers like Blast2GO takes from 15 to 20 minutes, which I feel could be much improved.

2 5

Defining $GALAXY_CPUS for use in tool wrappers
by Peter Cock 24 Oct '13

24 Oct '13

Hello all, Re: http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010153.html http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-October/011557.html Something I raised during the GCC2013, and we talked about via Twitter as well was a Galaxy environment variable for use within Tool Wrappers setting the number of threads/CPUs to use. The idea is that you can configure a default value, and then override this per runner or per tool etc. James Taylor had suggested calling this environment variable $GALAXY_CPUS which seem fine to me (personally I'd say threads not CPUs but I don't really mind). e.g. <command>my_tool --threads "\$GALAXY_CPUS" --input "$input" --output "$output"</command> Everyone I spoke to about this seemed positive about the idea. This would/should be integrated into the various cluster back ends, for example for SGE/OGE the number of threads is already configurable via the DRMAA settings and available as the environment variable $NSLOTS for non-MPI jobs, so my guess is all Galaxy needs to do is something like this: $ hg diff diff -r ce0d758bb995 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py Tue Jul 30 12:30:30 2013 +0100 +++ b/lib/galaxy/jobs/runners/drmaa.py Tue Jul 30 16:10:40 2013 +0100 @@ -43,6 +43,7 @@ # - execute the command # - take the command's exit code ($?) and write it to a file. drm_template = """#!/bin/sh +export GALAXY_CPUS="$NSLOTS" GALAXY_LIB="%s" if [ "$GALAXY_LIB" != "None" ]; then if [ -n "$PYTHONPATH" ]; then Is there an open Trello card for this? Thanks, Peter

5 11