February 2013 - galaxy-dev - lists.galaxyproject.org

Tool Shed examples defining datatype conversion
by Peter Cock 10 Mar '13

10 Mar '13

Hello all, I'd like to define 'blastxml' to 'tabular' conversion within Galaxy so that this can be used automatically (e.g. Tools expecting just tabular data would have converted BLAST XML files listed in the drop down). The functionality to do this exists in my Python tool 'blastxml_to_tabular' currently in the 'ncbi_blast_plus' repository, which might have to move into 'blast_datatypes' for this to work. http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus http://toolshed.g2.bx.psu.edu/view/devteam/blast_datatypes I've skimmed over the instructions here, but would like to look at a couple of real examples to make sure I've understood: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-February/013434.html Are there any good examples of 3rd party Galaxy datatypes defining conversion like this? Thanks, Peter

3 3

upload of local file ends up with a broken pipe.
by Jason Evans 07 Mar '13

07 Mar '13

I am having difficulties with my local instance of Galaxy on Mac OS X. Upon using the local file upload tool I get an error resultant of a broken pipe in python. The command-line output is as follows. 127.0.0.1 - - [25/Feb/2013:12:36:32 -0400] "GET /history HTTP/1.1" 500 - "http://localhost:8080/tool_runner/upload_async_message" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 50438) Traceback (most recent call last): File "/Users/jjevans/Work/Tools/Galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 641, in __init__ self.finish() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 694, in finish self.wfile.flush() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) error: [Errno 32] Broken pipe ---------------------------------------- 127.0.0.1 - - [25/Feb/2013:12:36:34 -0400] "GET /api/histories/2a56795cad3c7db3 HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" galaxy.jobs.manager DEBUG 2013-02-25 12:36:35,289 (77) Job assigned to handler 'upload_handler' My instance has worked for this task in the past, but I'm not sure if it has been since my last Galaxy update or perhaps something with my configuration changed. thanks for your help, Jason

2 3

Please help with fastx broken pipe
by Jorge Sepulveda 07 Mar '13

07 Mar '13

Hi, Fresh new installation of galaxy on a MacPro 12 core Mountain Lion Trying to run fastx_quality_stats: galaxy.jobs.manager DEBUG 2013-01-16 16:03:49,646 (7) Job assigned to handler 'main' galaxy.jobs DEBUG 2013-01-16 16:03:54,759 (7) Working directory for job is: /Users/JS/galaxy-dist/database/job_working_directory/000/7 galaxy.jobs.handler DEBUG 2013-01-16 16:03:54,759 dispatching job 7 to local runner galaxy.jobs.handler INFO 2013-01-16 16:03:54,854 (7) Job dispatched galaxy.jobs.runners.local DEBUG 2013-01-16 16:03:54,919 Local runner: starting job 7 galaxy.jobs.runners.local DEBUG 2013-01-16 16:03:55,198 executing: cat '/Users/JS/galaxy-dist/database/files/000/dataset_2.dat' | fastx_quality_stats -Q 33 -o '/Users/JS/galaxy-dist/database/files/000/dataset_7.dat' galaxy.jobs.runners.local DEBUG 2013-01-16 16:03:55,396 execution finished: cat '/Users/JS/galaxy-dist/database/files/000/dataset_2.dat' | fastx_quality_stats -Q 33 -o '/Users/JS/galaxy-dist/database/files/000/dataset_7.dat' galaxy.jobs DEBUG 2013-01-16 16:03:55,440 Tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2013-01-16 16:03:55,496 setting dataset state to ERROR galaxy.tools DEBUG 2013-01-16 16:03:55,618 Error opening galaxy.json file: [Errno 2] No such file or directory: '/Users/JS/galaxy-dist/database/job_working_directory/000/7/galaxy.json' galaxy.jobs DEBUG 2013-01-16 16:03:55,679 job 7 ended 127.0.0.1 - - [16/Jan/2013:16:03:59 -0400] "GET /history HTTP/1.1" 200 - "http://127.0.0.1:8080/root" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" Galaxy reports the following error: /bin/sh: fastx_quality_stats: command not found cat: stdout: Broken pipe When run on the cmd line, it runs fine: $ cat '/Users/JS/galaxy-dist/database/files/000/dataset_2.dat' | fastx_quality_stats -Q 33 -o '/Users/JS/galaxy-dist/database/files/000/dataset_6.dat' $ more /Users/JS/galaxy-dist/database/files/000/dataset_6.dat column count min max sum mean Q1 med Q3 IQR lW rW A_Count C_Count G_Count T_Count N_Count Max_count 1 912999 7 46 31022469 33.98 32 35 37 5 25 44 153534 361457 322516 75492 0 912999 2 912999 3 46 29744797 32.58 31 34 36 5 24 43 316245 255564 197711 143479 0 912999 Jorge Sepulveda, MD, PhD Associate Director for Laboratory Medicine, Medical Director of Laboratory Informatics, Columbia University Medical Center Associate Professor, Department of Pathology & Cell Biology Columbia University College of Physicians and Surgeons PH1590B 622 W 168 St New York, NY 10032 Cell: (917) 862-6050 Office: 212-305-6360 E-mail:jls2282@columbia.edu

2 1

Problem running Tophat for Illumina
by greg 06 Mar '13

06 Mar '13

Hi guys, On our local galaxy install I'm getting the error below when I run tophat for illumina like this: http://snag.gy/HsWyS.jpg Any ideas? Thanks, Greg Error: Dataset generation errors Dataset 27: Tophat for Illumina on data 15: deletions Tool execution generated the following error message: Error in tophat: tophat: option --no-closure-search not recognized for detailed help see http://tophat.cbcb.umd.edu/manual.html cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/junctions.bed': No such file or directory cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/insertions.bed': No such file or directory cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/accepted_hits.bam': No such file or directory cp: cannot stat `/home/galaxy/job_working_directory/000/238/tophat_out/deletions.bed': No such file or directory The tool produced the following additional output: TopHat v2.0.6 tophat -p 4 -a 8 -m 0 -i 70 -I 500000 -g 20 --library-type fr-unstranded --max-insertion-length 3 --max-deletion-length 3 --coverage-search --min-coverage-intron 50 --max-coverage-intron 20000 --no-closure-search --initial-read-mismatches 2 --segment-mismatches 2 --segment-length 25 --min-segment-intron 50 --max-segment-intron 500000 /usr/local/galaxy/indexes/Mus_musculus/UCSC/mm10/Sequence/Bowtie2Index/genome /home/galaxy/000/dataset_306.dat

3 5

Composite datatype output for Cuffdiff
by Jim Johnson 06 Mar '13

06 Mar '13

Checking to see if there is any interest in including a parameter option to select outputs for cuffdiff, potentially including a composite output and a cummeRbund sqlite database. Issues: cuffdiff produces 21 output files, which is a little unwieldy in a galaxy history. cummeRbund generates its database when given a cuffdiff output directory, but manually hooking up 21 outputs to the cummerbund_wrapper is a pain. I've put demo code in the testtoolshed under the name repository name cummerbund http://jjohnson@testtoolshed.g2.bx.psu.edu/repos/jjohnson/cummerbund This includes new datatypes defined in datatypes_conf.xml and implemented in cuffdata.py:  <datatype extension="cuffdata" type="galaxy.datatypes.cuffdata:CuffDiffData"/>  <datatype extension="cuffdatadb" type="galaxy.datatypes.cuffdata:CuffDataDB"/> The cuffdiff wrapper has a multiple select parameter to choose which output files to put in the history. In addition to the 21 cuffdiff outputs, the wrapper can also generate: cuffdata - which is a composite HTML output with links to the 21 cuffdiff outputs cuffdatadb - which is the cummeRbund SQLite database I also added utility tools: cuffdata_datasets - which will take files from the composite cuffdata and copy them as datasets into the history cuffdata_cummerbund - which generates the cummeRbund cuffdatadb from the composite cuffdata I updated the cummerbund_wrapper: with tryCatch so that a R error on a plot won't exit the Rscript to include a small image of each plot on the html page added plots for : dispersion, scatter matrix, MDS, and PCA Thanks, JJ

7 10

GALAXY LOCAL INSTALLATION: Tools and Data
by Rafael Hernández 05 Mar '13

05 Mar '13

Hi, I've installed Galaxy in local and following the Tool dependencies<http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies> manual I installed all necessary tools. However for some of then such as BWA, Bowtie,... it's usually needed to create indexes. Where should I store those indexes? In the galaxy directory or in each tool directory? How can I tell galaxy where the indexes are? I find a manual's page talking about it / http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) suggesting to organize the various index files for each build, but I don't know if these folders should be stored into Galaxy directory (eg. in tool-data directory) or what. Could you please explain this a little? Thank you and regards, Rafa Hernández de Diego Genomics of Gene Expression Lab. Bioinformatics and Genomics Department Centro de Investigación Príncipe Felipe (CIPF) C/ Eduardo Primo Yúfera 3 46012 Valencia, Spain

3 4

Manage API keys in admin panel
by Rémy Dernat 05 Mar '13

05 Mar '13

Hi, I would like to create a tool or a template that could list and (re)generate key for the user you want if you are login as an admin. I found the api_keys function in the user controller but I don't know how to use it. It seems that the decorators before the function checked if you are login as the good user... I also found the mako template that used this api_keys function but it is still unclear for me. Here is my sql function (FULL OUTER JOIN compatibility for MySQL) that list every keys even if they are empty : SELECT `galaxy_user`.`id`, `api_keys`.`user_id`, `api_keys`.`key`, `galaxy_user`.`email`, `galaxy_user`.`username` FROM `galaxy_user` LEFT JOIN `api_keys` on `api_keys`.`user_id` = `galaxy_user`.`id` UNION SELECT `galaxy_user`.`id`, `api_keys`.`user_id`, `api_keys`.`key`, `galaxy_user`.`email`, `galaxy_user`.`username` FROM `galaxy_user` RIGHT JOIN `api_keys` on `api_keys`.`user_id` = `galaxy_user`.`id` I know how to do that in PHP but it won't use python framework sqlalchemy (= only for mysql or pgsql...) ! Morevoer I don't know how do you generate your api_keys. Is it a md5 sum based on username / password ? Or is it just a random key with fix length ? Any help would be gracefull, Regards, Remy

1 2

Search software on public Galaxy servers
by Bernardo Bello 04 Mar '13

04 Mar '13

Hi community, Due to the increasing Galaxy public servers PublicGalaxyServers<http://wiki.galaxyproject.org/PublicGalaxyServers> it would be very useful to have a search tool in order to look for software ready-to-use in the public servers. Since I know this is not available. Will it be feasible? Here is the Trello Card https://trello.com/c/FdF5h17c Regards, Bernardo -- *Bernardo Bello Ortí* PhD student CReSA-IRTA Campus de Bellaterra-Universitat Autònoma de Barcelona Edifici CReSA 08193 Bellaterra (Barcelona, Spain) Tel.: 647 42 52 63 *www.cresa.es * * *

4 4

Resizing EBS on Amazon
by Scooter Willis 04 Mar '13

04 Mar '13

Used cloud man to create a new cluster on Feb 22 and picked 500GB as the initial size of the data drive. Working with TCGA exome DNA seq data didn't take long to fill that up. Used the cloud man admin interface to resize from 500GB to 1TB and the resize operation took 15 hours. Not sure if that is expected so wanted to give some heads up in case that is an area for optimization. Since I now have a local storage problem as I need to work with more than 1TB of data I tried to go the route of setting a S3 bucket using Fuse. Ran into a problem where the first s3fs software I tried to install had a version issue with Ubuntu 10. I remember something in a support email that better support for Amazon S3 was in the works. Can you provide any guidance or thoughts on how to work with more than 1TB of data using cost effective S3 versus expensive EBS? The same applies for storing results at S3. With s3fs the file system can hide many of the complexities of moving files back and forth with caching where working with 30GB+ files isn't going to be fun. Thanks Scooter

2 1

Bug in handling user job limits
by Lance Parsons 04 Mar '13

04 Mar '13

I submitted a trello "ticket" [(https://trello.com/c/6vxkqdjT) regarding this, but wanted to make sure I brought it to someone's attention (it's causing me some queue issues with my instance of galaxy). When registered_user_job_limit and anonymous_user_job_limit are set in universe.wsgi jobs cannot be run, instead the following error occurs: galaxy.jobs.handler ERROR 2012-12-04 12:44:51,869 failure running job 22396 Traceback (most recent call last): File "/data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py", line 183, in __monitor_step job_state = self.__check_if_ready_to_run( job ) File "/data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py", line 253, in __check_if_ready_to_run state = self.__check_user_jobs( job ) File "/data/local/galaxy/galaxy-prod/lib/galaxy/jobs/handler.py", line 274, in __check_user_jobs if not self.user_job_count: AttributeError: 'JobHandlerQueue' object has no attribute 'user_job_count' Commenting out the lines for registered_user_job_limit and anonymous_user_job_limit in universe.wsgi allows job to be queue one again. It looks like this is due to the fact that `self.__clear_user_job_count()` on line 159 of `handler.py` is only called when jobs are tracked in the database. If jobs are not tracked in the database (as in my case), the error occurs. Perhaps the fix would be to simply move the call outside the `if` block. It appears this was broken in the 2012-11-13 revision (73e05bc). -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

3 5