July 2012 - galaxy-dev - lists.galaxyproject.org

Auto delete dataset after workflow run?
by Praveen Raj Somarajan 03 Dec '12

03 Dec '12

Hello All, Is there a way to set "auto delete" some datafiles at the end of workflow run? I'm asking this because it is sometimes useful to delete some outputs (say intermediate files, .logs, etc), for example "auto delete" .sam file after executing sam-to-bam tool, which would free-up lot of space. This is especially useful when work with large number of samples. Look forward to any comments/suggestions. Best, Raj. ________________________________ This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions that are unlawful. This e-mail may contain viruses. Ocimum Biosolutions has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system. OCIMUMBIO SOLUTIONS (P) LTD

3 4

makeblastdb wrapper
by Espin, Almudena 09 Nov '12

09 Nov '12

Hi guys, I am writing a wrapper for makeblastdb. The problem is that Galaxy doesn’t support the blastdb data format, so this has to be added if a makeblastdb wrapper is to work properly. I have problems modifying the registry.py, Could anyone specify which changes have to be done? Thank you

2 3

Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch
by Sascha Kastens 03 Nov '12

03 Nov '12

Hi Dev-Team, are you planning or maybe working on an update which enables the possiblilty to run workflows in batch mode with paired end data? Cheers, Sascha

3 3

Column alignment in tabular display
by Peter Cock 31 Oct '12

31 Oct '12

Hello all, I'm curious about the logic used for column alignment when displaying tabular files (which for me was the headline feature of the last Galaxy Release - I'm very pleased to have this added at last). https://bitbucket.org/galaxy/galaxy-central/issue/553 Quoting from templates/dataset/tabular_chunked.mako function renderCell(cell_contents, index, colspan){ if (colspan !== undefined){ return $('<td>').attr('colspan', colspan).addClass('stringalign').text(cell_contents); } else if (DATASET_TYPES[index] == 'str'){ /* Left align all str columns, right align the rest */ return $('<td>').addClass('stringalign').text(cell_contents);; } else{ return $('<td>').text(cell_contents); } } It seems to me that the logic should be right align all numeric columns, left align the rest. At very least, columns of type 'list' should also be left aligned. Otherwise you can easily have a situation where on clicking the "eye" you get an apparently empty dataset (unless you scroll right or down). This can easily happen with FASTA to tabular (default settings), where the descriptions have commas in them (common with annotation) thus the first column is considered to be a list (and the second column type str for the sequence). If some of the later sequences in the file have very long descriptions, right aligning the description column makes the dataset appear empty. I can provide a screen shot if it would help to explain, but here is my proposed patch. Regards, Peter -- $ hg diff templates/dataset/tabular_chunked.mako diff -r 8657ada96fca templates/dataset/tabular_chunked.mako --- a/templates/dataset/tabular_chunked.mako Fri Jun 15 16:05:48 2012 +0100 +++ b/templates/dataset/tabular_chunked.mako Mon Jun 18 16:23:44 2012 +0100 @@ -17,8 +17,8 @@ if (colspan !== undefined){ return $('<td>').attr('colspan', colspan).addClass('stringalign').text(cell_contents); } - else if (DATASET_TYPES[index] == 'str'){ - /* Left align all str columns, right align the rest */ + else if (DATASET_TYPES[index] == 'str' || DATASET_TYPES[index] == 'list'){ + /* Left align all str and list columns, right align the rest */ return $('<td>').addClass('stringalign').text(cell_contents);; } else{

2 3

Hiding a tool via tool_conf.xml
by Peter Cock 30 Oct '12

30 Oct '12

Hi all, I'm aware that in an individual tool's XML wrapper you can add hidden="true" to the <tool> element to hide the tool from the listing shown in Galaxy's left hand panel (but the tool is still loaded and can be called by old workflows etc). Can this be done via the tool_conf.xml file as well? This would seem useful from a system configuration point of view where it seems wrong to edit the tool wrappers themselves. In fact, it might also be nice to add hidden="true" support to <section> as well as <tool> in tool_conf.xml - although I'm not so sure how useful that would be. Peter

1 2

Re: [galaxy-dev] Using $NSLOTS in tools to control thread number
by Peter Cock 19 Oct '12

19 Oct '12

On Fri, Jun 15, 2012 at 4:38 PM, James Taylor <james(a)jamestaylor.org> wrote: > This is exactly what I think we should do (and have for a long time), but I > think the variable should be something like: > > GALAXY_CPUS > > (threads is not accurate, a multithread or multiprocess job might want to use > this info, something even more abstract than CPUS might make sense, but > SLOTS has never made sense to me). I agree that a Galaxy specific name makes a lot of sense, and that the SGE term "slots" is a bit odd. Using CPUS however is potentially ambiguous with CPUs vs cores - my desktop has two quad core CPUs, i.e. 2 CPUs but 8 cores. Where do you think this number should come from? A new entry in the runner URL is simple albeit potentially redundant with cluster-specific entries in the runner URL. As to the alternative (doing it automatically), for PBS and SGE determining the number of cores from the cluster configuration and/or parsing the cluster runner URL sounds doable - what about the other backends? Peter

2 2

Getting syntax error on Join + environment_setup_file not read
by Peter Schmitt 04 Sep '12

04 Sep '12

New galaxy install. Running the tutorial Galaxy 101 and trying to do the join. Get the following error in the stderr of the job: Traceback (most recent call last): File "/opt/galaxy/galaxy-dist/tools/new_operations/gops_join.py", line 20, in ? from galaxy.tools.util.galaxyops import * File "/opt/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py", line 15, in ? from galaxy import util, jobs, model File "/opt/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 15, in ? from galaxy import util, model File "/opt/galaxy/galaxy-dist/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/opt/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line 146 finally: ^ SyntaxError: invalid syntax Traceback (most recent call last): File "./scripts/set_metadata.py", line 24, in ? import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/opt/galaxy/galaxy-dist/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/opt/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line 146 finally: ^ SyntaxError: invalid syntax Also I believe the line, environment_setup_file = scripts/galaxy.sh, is not being used any longer. When I put the line: export DRMAA_LIBRARY_PATH=/opt/drmaa/1.0.12/lib/libdrmaa.so.1.0.12, in there the server didn't see it. When I added that line to the top of run.sh, it does. -- * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid Computational Genetics Lab Dartmouth College, HB 6223 L24 Berry/Baker Library Hanover, NH 03755 Phone: 603-646-8109 http://discovery.dartmouth.edu http://columbia.dartmouth.edu/grid http://www.epistasis.org *

2 1

Setup questions: data locations, MySQL, and cores
by Steven Peckins 30 Aug '12

30 Aug '12

I've been asked to set up a local Galaxy installation specifically for large datasets (tens of terabytes). Is there a list of default locations where Galaxy puts data? As an admin, that would be my first question, but it's not obvious from the documentation, and for large datasets, it's important to know. (During testing, with Galaxy in the default location, /home/galaxy, an attempt to upload and decompress a 50GB file wreaked havoc on the server due to heavy I/O. Changing the tmp directory so it wasn't reading and writing massive amounts of data to the same filesystem at the same time helped, but that would have been nice to know ahead of time.) Is the database used for datasets at all or just user account data? In other words, if users are crunching terabytes of data, do I need to worry about the amount of space on the filesystem that hosts the database? What exactly are the disadvantages of using MySQL over PostgreSQL? Several places in the docs state that it is preferred but not why. Is it a bigger problem than installing another database that I have no experience with over one that is already installed and with which I am already familiar? Is an MPI configuration necessary for getting full use out of a multicore system? The docs seem to indicate that Galaxy will use multiple cores ("Without a cluster, you'll be limited to the number of cores in your server...") but take pains to say that GIL won't allow more than a single thread ("This means that regardless of the number of cores in your server, Galaxy can only use one" and "having a multi-core system will not improve the Galaxy framework's performance out of the box since Galaxy can use (at most) one core at a time"). We have a 48-core machine, and there will only be two or three users. Thanks, --steve

2 1

Tool test question
by christopher fletez-brant 29 Aug '12

29 Aug '12

Dear Dev List, I am currently writing tests for a tool I am developing and am coming up short in understanding why it's failing. Using the test data, I can generate output identical to the output provided for testing every time I run the tool from my local Galaxy server. However, when I run the tool test through run_functional_tests.sh, it always fails, and apparently fails on every line. Because I consistently see the same output from the Galaxy server-based tool, I'm sure I'm missing something simple and want to look at the test-generated output. Is there a way I can set run_functional_tests to print the generated tool output file in its entirety? Thank you, Kipper Fletez-Brant

2 6

xsendfile permissions
by Julian Pradinuk 29 Aug '12

29 Aug '12

Hi all, I'm wondering if someone who has downloads working through apache (using xsendfile) has encountered this error when downloading HTML files: (13)Permission denied: xsendfile: cannot open file: /galaxy_data/production/tmp/tmpN0ydYt/library_download.zip The output from ls -lZ /galaxy_data/production/tmp is: ... drwx------. galaxy galaxy unconfined_u:object_r:httpd_sys_content_t:s0 tmpN0ydYt ... The output from ls-lZ /galaxy_data/production/tmp/tmpN0ydYt/ is: -rw-rw-r--. galaxy galaxy unconfined_u:object_r:httpd_sys_content_t:s0 library_download.zip It's not a SElinux issue (if I sudo setenforce 0, the problem persists).. It seems it's a regular permissions issue (user apache is running httpd and does not permission to enter the tmpN0ydYt). Since galaxy creates these files with these permissions I don't know how to fix this issue. I'm also wondering why the HTML datasets are in the tmp folder not the files folder? Here's my apache config: <VirtualHost *:80> ServerName ________________ RewriteEngine on #RewriteOptions Inherit #RewriteLog "/etc/httpd/logs/rewrite_log" #RewriteLogLevel 5 RewriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy/static/style/(.*) /usr/local/galaxy/production/galaxy-dist/static/june_2007_style/blue/$1 [L] RewriteRule ^/galaxy/static/scripts/(.*) /usr/local/galaxy/production/galaxy-dist/static/scripts/packed/$1 [L] RewriteRule ^/galaxy/static/(.*) /usr/local/galaxy/production/galaxy-dist/static/$1 [L] RewriteRule ^/galaxy/favicon.ico /usr/local/galaxy/production/galaxy-dist/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /usr/local/galaxy/production/galaxy-dist/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://localhost:8080$1 [P] <Location "/galaxy"> order allow,deny allow from all # Compress all uncompressed content SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary # Send files through Apache XSendFile on XSendFilePath /galaxy_data </Location> <Location "/galaxy/static"> # Allow browsers to cache everything from /static for 6 hours ExpiresActive On ExpiresDefault "access plus 6 hours" </Location> </VirtualHost> Is there something wrong with it? Or have I missed something in universe.wsgi? Cheers, Julian

2 1