April 2013 - galaxy-dev - lists.galaxyproject.org

Galaxy Keeps Freezing
by Kim Spradling 02 May '13

02 May '13

I have been trying to join the intervals of two datasets side-by-side, but the job never finishes. I've gotten a message stating that Galaxy is unavailable a couple of times, but I have tried running this job multiple times with no success. Any help will be greatly appreciated. Thanks, Kim

2 1

Installing FreeBayes via Toolshed: Tool dependency installation error
by Iry Witham 01 May '13

01 May '13

Hi All, I am trying to get Freebayes installed on a cloud man instance from the galaxy toolshed. I have successfully installed the Samtools 0.1.18 package, but the freebayes package errors. The Error is: Tool shed repository: freebayes Tool shed repository changeset revision: 28e422c589ac Tool dependency status: Error Tool dependency installation error: /bin/sh: cmake: not found make[1]: *** [../bamtools/lib/libbamtools.a] Error 127 make: *** [all] Error 2 Tool dependency installation directory: /mnt/galaxyTools/tools/freebayes/0.9.6_9608597d12e127c847ae03aa03440ab63992fedf/devteam/freebayes/28e422c589ac I get the following message after the error occurs: This tool dependency is not installed correctly (see the Tool dependency installation error below). Choose Uninstall this tool dependency from the Repository Actions menu, correct problems if necessary, and try installing the dependency again. I have followed this procedure and still have the same error. I have looked at the Installation.log file and this is what I found: ############################################# git clone --recursive git://github.com/ekg/freebayes.git STDOUT Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/.git/ Submodule 'bamtools' (git://github.com/pezmaster31/bamtools.git) registered for path 'bamtools' Submodule 'intervaltree' (git://github.com/ekg/intervaltree.git) registered for path 'intervaltree' Submodule 'vcflib' (git://github.com/ekg/vcflib.git) registered for path 'vcflib' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/bamtools/.git/ Submodule path 'bamtools': checked out '3fe66b92028b60629826d827ee9058405fa7e4be' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/intervaltree/.git/ Submodule path 'intervaltree': checked out 'd151b487804861dc9f932e9f1fe4f8c499673cec' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/.git/ Submodule path 'vcflib': checked out '480d96e16a552d36369dc9cb2b39ceedb260944a' Submodule 'fastahack' (git://github.com/ekg/fastahack.git) registered for path 'fastahack' Submodule 'fsom' (git://github.com/ekg/fsom.git) registered for path 'fsom' Submodule 'intervaltree' (git://github.com/ekg/intervaltree.git) registered for path 'intervaltree' Submodule 'multichoose' (git://github.com/ekg/multichoose.git) registered for path 'multichoose' Submodule 'smithwaterman' (git://github.com/ekg/smithwaterman.git) registered for path 'smithwaterman' Submodule 'tabixpp' (git://github.com/ekg/tabixpp.git) registered for path 'tabixpp' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/fastahack/.git/ Submodule path 'fastahack': checked out '6acaf9875f2dbcdf86c9c690a8d188000e90d388' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/fsom/.git/ Submodule path 'fsom': checked out '1b888f32c4bd571d1a0bea88d39cd7d7f23ea4b8' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/intervaltree/.git/ Submodule path 'intervaltree': checked out '1290744283cef8076bb8a2968d4899b7228435f4' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/multichoose/.git/ Submodule path 'multichoose': checked out '73d35daa18bf35729b9ba758041a9247a72484a5' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/smithwaterman/.git/ Submodule path 'smithwaterman': checked out 'd9724f156c07cf16d00d251bebc20c6383eb6bde' Initialized empty Git repository in /mnt/galaxyData/tmp/tmpCUrxI7/freebayes/vcflib/tabixpp/.git/ Submodule path 'tabixpp': checked out 'fbdde774adfc79ca75a65b4321b89b67cd81b921' ############################################# ############################################# git clone --recursive git://github.com/ekg/freebayes.git STDERR ############################################# ############################################# git checkout 9608597d12e127c847ae03aa03440ab63992fedf STDOUT M bamtools M vcflib ############################################# ############################################# git checkout 9608597d12e127c847ae03aa03440ab63992fedf STDERR Note: checking out '9608597d12e127c847ae03aa03440ab63992fedf'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name HEAD is now at 9608597... example pipeline script ############################################# ############################################# git submodule update --recursive STDOUT Submodule path 'bamtools': checked out '8a5d6508d8fc52b25c9129037e44cc448bafebc7' Submodule path 'vcflib': checked out '3f0d5e19596252b858a7f6eff1578e4295f549c8' Submodule path 'fastahack': checked out 'd1bcd7a2d52ee0e2823884c8ede22ce228d3c060' Submodule path 'smithwaterman': checked out '2882633a3b89f820d692a9b3316f4891dc3a1ff8' ############################################# ############################################# git submodule update --recursive STDERR warning: unable to rmdir fsom: Directory not empty ############################################# ############################################# make STDOUT cd src && make make[1]: Entering directory `/mnt/galaxyData/tmp/tmpCUrxI7/freebayes/src' cd ../bamtools && mkdir -p build && cd build && cmake .. && make make[1]: Leaving directory `/mnt/galaxyData/tmp/tmpCUrxI7/freebayes/src' ############################################# ############################################# make STDERR /bin/sh: cmake: not found make[1]: *** [../bamtools/lib/libbamtools.a] Error 127 make: *** [all] Error 2 ############################################# Any ideas on how to resolve this? Thanks, Iry Witham The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 5

Re: [galaxy-dev] BLAST Database error: No alias or index file found
by Mike Dyall-Smith 01 May '13

01 May '13

Thanks Peter. My answers are below: >What query sequences are you using? I have just been using one fasta protein sequence. >Meanwhile monitor the system with top Top says only a max of 27% cpu usage, but the linux screen eventually freezes, and I have to restart. I am not sure how to read out RAM and disk IO from top. >grep blastp paster.log Tried that, and it says there is no such file or directory >Could you try running BLAST from the host Mac OX Yes. And it works fine! I get a good match in a relatively short time. I then made a very small protein database, checked it by commandline blastp in both host OS X and in guest linux, and it worked fine. Added it to the blastdb_p.loc file, restarted and saw it listed in galaxy. Tried to use it for a blastp, and got the same error as before with the huge NCBI nr database. So, it is not a matter of size.... Thanks for your comments about RAM and blast searches. It gives me hope that I can get galaxy running usefully. I only chose biolinux because of the suite of programs and the apparent ease of use. The other reason was that I could not install galaxy on OS X (10.6). I get errors that others have noted on the discussion lists but no-one seems to have a solution for. regards, Mike DS -------------------------------------------------------------- On Tue, Apr 30, 2013 at 10:49 PM Peter Cock wrote: > Hi Mike, > > On Tue, Apr 30, 2013 at 12:54 PM, Mike Dyall-Smith > <mike.dyallsmith(a)gmail.com> wrote: > > Dear Peter, thanks for the advice. I think I can now run blastp from the > > commandline, both in the host and in the linux virtual machine. I say > > 'think' because it runs in both cases, but then either never completes (I > > cancelled after 30 min on OS X) or freezes (VBox/ubuntu). However, the > > blastp search in galaxy gives the same error as before. This, I don't > > understand. > > What query sequences are you using? I'd try just one or two protein > sequences in a small FASTA file, and rerun blastp against nr. > Meanwhile monitor the system with top or similar to see what > the CPU usage, RAM usage, and disk IO is like. > > That should help determine if this is a simple as not enough RAM > leading to lots of paging to disk, and therefore a very slow search. > > > I simplified the directory structure (so the path) to the database, and > > altered the appropriate configuration files (blastdb_p.loc, > > blast_environment.sh). With 'env', I see: > BLASTDB=/media/sf_mikeds_bioinf/db > > I also checked that "ls /media/sf_mikeds_bioinf/db/nr*" listed all the > > database files. > > > > The relevant lines are now: > > ----------------from blastdb_p.loc file------------------------- > > #Your blastdb_p.loc file should include an entry per line for each "base > > name" > > #you have stored. For example: > > # > > #nr_05Jun2010 NCBI NR (non redundant) 05 Jun 2010 > > /data/blastdb/05Jun2010/nr > > #nr_15Aug2010 NCBI NR (non redundant) 15 Aug 2010 > > /data/blastdb/15Aug2010/nr > > nr_08_Apr2013 NCBI_nrprot_08Apr2013 /media/sf_mikeds_bioinf/db/nr > > ... > > ---------------------------------------------------------------------- > > That looks OK. > > > The blast+ blastp error from galaxy is: > > ----------------------------------------------------------------------- > > An error occurred running this job: blastp: 2.2.26+ > > Package: blast 2.2.26, build Aug 15 2012 17:48:54 > > BLAST Database error: No alias or index file found for protein database > > [/media/sf_mikeds_bioinf/db/nr] in search path > > [/var/lib/galaxy-server/database/job_working_directory/000/18::] > > ----------------------------------------------------------------------- > > > > Very strange, clearly something is not right with the Galaxy config. > on the bright side, Galaxy is finding the binaries . Could you try this > on the Galaxy log file, > > $ grep blastp paster.log > > > However, while this might point to a real issue for the use of galaxy > within > > VirtualBox/ubuntu (and with the database on an external USB3 drive), I > > suspect my idea of running a local instance of galaxy on my macbook is > not > > possible. I have 8 Gb of RAM, and set 4Gb for use in the linux guest > system. > > The Genbank nr protein db directory has files totaling 24 Gb. > > The oldest cluster nodes we're still using have 8GB of RAM, and are > fine to run BLASTP against NR with. The previous nodes only have > 2GB and could not cope - so the threshold is somewhere in between. > I suspect your guest VM with only 4GB of RAM is struggling. > > Could you try running BLAST from the host Mac OX X instead, with > access to the full 8GB of RAM? > > > Running blastp > > from the commandline does not give a result in any reasonable length of > > time, even when a single query sequence is used (both on the host and > guest > > systems). Even if I were able to handle raw sequence datasets of moderate > > size (less than 1 Gb? as yet untested) in Galaxy, it would be of little > use > > to me if I can't blastp or blastx search the resulting contigs. Do I set > up > > galaxy to send blast search queries to NCBI (i.e. many thousands of > > queries??) or is there some more elegant solution? > > ... > Regards, > > Peter >

2 5

Adding data sources in the main galaxy instance
by Fingerman, Ian (NIH/NLM/NCBI) [E] 01 May '13

01 May '13

I have a question regarding data sources currently available on the main instance of Galaxy (https://main.g2.bx.psu.edu) How can one get added as a new data source? I work at the NCBI on the Epigenomics database (http://www.ncbi.nlm.nih.gov/epigenomics) . We currently hold a large volume of NGS data in the form of wig files, and some users have expressed interest in using Galaxy for data analysis. We provide an easy to use user interface for examining these data as seen here (http://www.ncbi.nlm.nih.gov/epigenomics/browse/) and currently host over 4200 data tracks. I think it would useful if we could integrate Galaxy functionality into our resource. I also thing providing our resource as a "Data Source" would also be convenient for users less familiar with our database but are current Galaxy users. I have been looking through the Galaxy wiki and I am struggling to find documentation that details step-by-step, what exactly needs to be done. One thing to note, I am not a developer, I'm the scientific lead for the project and my programming/developing skills are lacking. I was hoping someone could point me to thorough documentation mainly to pass on to developers on my team. I guess also understanding my options with regards to integrating or interfacing with Galaxy would be very valuable to me too. Thank you for any help/suggestions you may have. Ian Ian Fingerman, Ph.D. Staff Scientist NIH/NLM/NCBI Building 45, Room 4AN28D-29 45 Center Drive MSC-6510 Bethesda, MD 20894 Phone: (301) 496-6806 fingerma(a)ncbi.nlm.nih.gov

4 4

Creating multiple datasets in a libset
by Rob Leclerc 30 Apr '13

30 Apr '13

I'm looking at example_watch_folder.py and it's not clear from the example how you submit multiple datasets to a library. In the example, the first submit returns a libset [] with only a single entry and then proceeds to iterate through each dataset in the libset in the following section: data = {} data['folder_id'] = library_folder_id data['file_type'] = 'auto' data['dbkey'] = '' data['upload_option'] = 'upload_paths' *data['filesystem_paths'] = fullpath* data['create_type'] = 'file' libset = submit(api_key, api_url + "libraries/%s/contents" % library_id, data, return_formatted = False) time.sleep(5) for ds in libset: if 'id' in ds: wf_data = {} wf_data['workflow_id'] = workflow['id'] wf_data['history'] = "%s - %s" % (fname, workflow[ 'name']) wf_data['ds_map'] = {} for step_id, ds_in in workflow['inputs' ].iteritems(): wf_data['ds_map'][step_id] = {'src':'ld', 'id' :ds['id']} res = submit( api_key, api_url + 'workflows', wf_data, return_formatted=False) Rob Leclerc, PhD <http://www.linkedin.com/in/robleclerc> <https://twitter.com/#!/robleclerc> P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.leclerc(a)aya.yale.edu

2 6

print out workflow execution commands!
by Hakeem Almabrazi 30 Apr '13

30 Apr '13

Hi everyone, Is there an easy way to view/save/print out the executed commands and their parameters for a workflow? I want to see what is going on for certain steps in my workflow. Regards, Hak

2 5

Galaxy API prob with HTTP_REMOTE_USER
by Fabien Mareuil 30 Apr '13

30 Apr '13

Hi all, We encountoured an error with the galaxy api since the last galaxy-dist revision. here is an example of a command line that doesn't work (xxx stand for the admin key) : ./display.py xxxxxxxxxxxxxxxxxxxxxxxxxx http://localhost:18084/api/users we obtain the following error: HTTP Error 500: Internal Server Error As you can see, the request is done locally, without going through a proxy or apache. The galaxy log is the following: 127.0.0.1 - - [23/Apr/2013:19:55:26 +0200] "GET /api/users?key= xxxxxxxxxxxxxxxxxxxxx HTTP/1.1" 500 - "-" "Python-urllib/2.6" Error - <type 'exceptions.AssertionError'>: use_remote_user is set but no HTTP_REMOTE_USER variable Do you have an idea to solve this problem? Thank in advance for your answer. Best regards -- Fabien Mareuil | Centre d'Informatique pour la Biologie fabien.mareuil(a)pasteur.fr | Institut Pasteur 25,28 rue du Docteur Roux 75015 Paris, France

4 15

Tophat for Illumina is not running
by lilach noy 30 Apr '13

30 Apr '13

Hello, I'm new to Galaxy so I think i'm at the right place asking this, though please let me know if I got it all wrong. I sent a Tophat query on Thursday about 5 days ago and it is still waiting to ran (gray background). Is to O.k? A bug? Should i keep waiting or stop everything and restart? If this is not the address for reporting a suspicious bug, whom does this concern? Thank you very much, Lilach

2 1

Bug or design? Unit tests use *.loc rather than *.loc.sample
by Peter Cock 30 Apr '13

30 Apr '13

Hello all, Given the precedent that the test framework uses tool_conf.xml.sample rather than tool_conf.xml, I would expect it to use example.loc.sample rather than example.loc when running unit tests with parameters whose values come from a loc file. This is not the case. When writing unit tests of the Tool Shed, we can assume that as part of the automated installation example.loc will be generated from the supplied example.loc.sample file - and so for things like the nightly automated testing on the Tool Shed, the files will be the same. However, on real Galaxy installations, the administrator will likely edit example.loc and it will not match example.loc.sample any more. This means that for writing a unit test, I cannot make any assumptions about the contents of example.loc, but it would be reasonable to expect example.loc.sample setup will still match the unit tests. Is it sensible to make the test framework load *.loc.sample rather than *.loc for this reason? If not, how exactly are unit tests dependent on values in a loc file meant to be written? This is a problem for me writing tests for Blast2GO (property files in blast2go.loc), BLAST+ (databases in blastdb.loc etc). It hasn't yet been a problem for Effective T3 as here my effectiveT3.loc file still matches the sample file. Thanks, Peter

1 0

Teaching using Galaxy
by David Joly 30 Apr '13

30 Apr '13

Hi everybody! I am currently creating a "bioinformatics" course for undergraduate (biology students with no knowledge of programming). I would like to use Galaxy as their everyday platform where they would learn the basics and use the appropriate tools (BLAST and databases, multiple alignment, phylogenetics, dealing with "omics" data, and so on). Is there any available resources about using Galaxy for teaching (undergraduates)? Any suggestions of good textbooks? Not a Galaxy textbook of course, but a "bioinformatics textbook" that would be a good companion to help the students understand the basics behind the tools. Thanks, DJ

4 6