- galaxy-dev - lists.galaxyproject.org

Puppet Modules
by Eric Rasche 15 Mar '14

15 Mar '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sys Admins, I'm working on some puppet modules for managing galaxy. I'm not sure how many of you use puppet, but I figured it would be useful to the community. I'll place them on puppet forge when I'm done. So far I'm doing the following: - - enabling initial deployment (updating?) with the vcsrepo module - - managing the following files: job_conf.xml, tool_conf.xml, tool_sheds_conf.xml, universe_wsgi.ini If there's anything that you find important to be managed, or features you'd like to see, please email me and let me know! Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 esr(a)tamu.edu rasche.eric(a)yandex.ru -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAEBAgAGBQJTIH0RAAoJEMqDXdrsMcpV2g8P/08nTnb/RLdz6WGfDbryHtBB 6Q5yr4HV2IM/idgjpLQX93jRCyahVuS1E8Sgc1x9ViGml93/ssHJZ9GIsf1PJM5/ erygrVS6njhWgefMQpoEipfTOHqxCvHhJk5xnvRc/EeUj0Lkj6YKoVpzae/6XilH Tav9/ocTNrH40HBjuFGsK7Q/IP1C+vNr/q7GPXq6Ek6dWd23qxRCYszuGgS2t3s/ i/+YmZCwHH0I25ssujGsYzsrsTfOBVBNvAj9dUmvyE+9WSLZQVw4919VKXDKNuSs 7oFBPJD2hpqSKJxf7IFChFL9WhQ4e1gykx+Z+7MhBpekUeYvwIzABxFNmQLOdl+Q 4R+mEADyAopD2enl23XkePX/FRag839zYSeEOWjvEfC8qgUWFXEJyjJ9JLqXE+Bc QOAMpmC/BMI5qIxIvKJ2ASZuqBXFxdRcGs1xaqJ2bWN+vm1fD+jzpD33KjHcegQr gf0UvhJj3aoR0Pua3vuWkWP6wPBt08F6Qab/6tQGIeQUyzZRCGyVggd4SgN+Xml5 0OB5olauA+Nr0wD+tDiCQwE+iIL/U61nU5A0yCmEMVfBem8YyKPtjuNZ7E/OZ9fy gv4uVmGDIfM+IySnWlmxqX2nYb8AODrZiQifT3kG93jh4HtNaoc4yTNeRYcFwTNi r8tJpajbO8Q0MVVj+bWn =aFP5 -----END PGP SIGNATURE-----

3 3

Internal server error when importing workflows!
by Hakeem Almabrazi 15 Mar '14

15 Mar '14

Hi, I have upgraded my local galaxy to the latest stable version and it works fine but I get an error when I try to import workflows that were created older version of Galaxy. Here is the error I keep getting. Internal Server Error Galaxy was unable to successfully complete your request URL: http://10.7.10.21/workflow/import_workflow Module galaxy.web.framework.middleware.error:149 in __call__ >> <http://10.7.10.21/workflow/import_workflow> app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ >> <http://10.7.10.21/workflow/import_workflow> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> <http://10.7.10.21/workflow/import_workflow> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> <http://10.7.10.21/workflow/import_workflow> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request >> <http://10.7.10.21/workflow/import_workflow> body = method( trans, **kwargs ) Module galaxy.webapps.galaxy.controllers.workflow:1072 in import_workflow >> <http://10.7.10.21/workflow/import_workflow> workflow, missing_tool_tups = self._workflow_from_dict( trans, data, source=src, add_to_menu=add_to_menu ) Module galaxy.web.base.controller:1639 in _workflow_from_dict >> <http://10.7.10.21/workflow/import_workflow> module.save_to_step( step ) Module galaxy.workflow.modules:274 in save_to_step >> <http://10.7.10.21/workflow/import_workflow> step.tool_inputs = self.tool.params_to_strings( self.state.inputs, self.trans.app ) Module galaxy.tools:2418 in params_to_strings >> <http://10.7.10.21/workflow/import_workflow> return params_to_strings( self.inputs, params, app ) Module galaxy.tools.parameters:85 in params_to_strings >> <http://10.7.10.21/workflow/import_workflow> value = params[ key ].value_to_basic( value, app ) Module galaxy.tools.parameters.grouping:468 in value_to_basic >> <http://10.7.10.21/workflow/import_workflow> rval[ input.name ] = input.value_to_basic( value[ input.name ], app ) KeyError: 'index' Can someone let me know how to fix this issue? Thank you Hakeem

2 2

How To properly use GALAXY_SLOTS
by Geert Vandeweyer 15 Mar '14

15 Mar '14

Hi, Is there documentation for the proper setup and utilisation of the \${GALAXY_SLOTS:-4} style options? I noticed that tools with this setting, including BWA and GATK run single threaded, with the following settings respectively: BWA \${GALAXY_SLOTS:-4}" === ps output: python /galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_wrappers/bwa_wrapper.py --threads=1 --fileSource=indexed --ref=/galaxy/galaxy_references/hg19/bwa-0.5.9/base/hg19.fasta --do_not_build_index --input1=/galaxy/galaxy-dist/database/files/093/dataset_93021.dat --input2=/galaxy/galaxy-dist/database/files/093/dataset_93023.dat --output=/galaxy/galaxy-dist/database/files/000/112/dataset_112301.dat --genAlignType=paired --params=pre_set --suppressHeader=false xml config: --threads="\${GALAXY_SLOTS:-4}" GATK Base Recalibrator \${GALAXY_SLOTS:-8} ============== python /galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/gatk2_wrapper.py --max_jvm_heap 6g --stdout /galaxy/galaxy-dist/database/files/000/112/dataset_112292.dat -d -I /galaxy/galaxy-dist/database/files/000/112/dataset_112289.dat bam gatk_input -d /galaxy/galaxy-dist/database/files/_metadata_files/012/metadata_12686.dat bam_index gatk_input -p java -jar "$GATK2_PATH/GenomeAnalysisTK.jar" -T "BaseRecalibrator" $GATK2_SITE_OPTIONS --num_cpu_threads_per_data_thread ${GALAXY_SLOTS:-8} --no_standard_covs -R "/galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta" --out "/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat" -cov "ContextCovariate" -cov "CycleCovariate" -d --knownSites:dbsnp,%(file_type)s /galaxy/galaxy-dist/database/files/000/101/dataset_101086.dat vcf input_dbsnp_0 -p --pedigreeValidationType "STRICT" -d --intervals /galaxy/galaxy-dist/database/files/000/100/dataset_100129.dat bed input_intervals_0 -p --interval_set_rule "UNION" -p --downsampling_type "NONE" -p --baq "OFF" --baqGapOpenPenalty "40.0" --defaultBaseQualities "-1" --validation_strictness "STRICT" --interval_merging "ALL" java -Xmx6g -jar /galaxy/galaxy-dist/tool-data/shared/jars/gatk2//GenomeAnalysisTK.jar -T BaseRecalibrator --num_cpu_threads_per_data_thread 1 --no_standard_covs -R /galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out /galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov ContextCovariate -cov CycleCovariate --pedigreeValidationType STRICT --interval_set_rule UNION --downsampling_type NONE --baq OFF --baqGapOpenPenalty 40.0 --defaultBaseQualities -1 --validation_strictness STRICT --interval_merging ALL -I /tmp/tmp-gatk-fIuncH/gatk_input.bam --knownSites:dbsnp,vcf /tmp/tmp-gatk-fIuncH/input_dbsnp_0.vcf --intervals /tmp/tmp-gatk-fIuncH/input_intervals_0.bed Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer

4 5

How to install bowtie2 tool in galaxy
by Ravi Alla 14 Mar '14

14 Mar '14

Hi guys, I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions. - Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up? - What are repository dependencies? Why would I need these if bowtie2 is already installed on my system? - I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct? - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices. I hope someone on here can help me out. Thanks a lot Ravi.

5 11

Dataset's extra files
by Charles Girardot 14 Mar '14

14 Mar '14

Hi all, We have a local tool which role is to transfer (ie copy) a dataset file to a directory on our NFS. This is extremely convenient as it can be included within workflows and therefore save the time of clicking download button (we also have configurable renaming/compression as part of it). It is heavily used by our users. The problem is with datasets that have associated files like FASTQC as these extra files are simple not ignored... We'd like to improve our 'NFS_transfer' tool so it can deal with this in a similar fashion as the download button. Foreseen solution : * Check if a directory named 'dataset_<id>_files' exists within the dataset store * if so, 'cp -r' it into a tmp dir, cp the dataset itself into same tmp dir (with renaming on the fly) * zip/tar.gz the tmp dir * copy it to final NFS location Question is : is this the right way to do it ? As a non python specialist, it is a little tricky to find the right way to it (I can t locate the piece of code that does this in galaxy ie behind the download button). In particular, can I get the list of extra files using the '$galaxyFile' object given in the tool by : <param type="data" name="galaxyFile" label="File to transfer"/> i.e. in the same way we get the dataset name or file extension ($galaxyFile.dataset.name and $galaxyFile.ext) ? Any advise on how best to implement this, in a portable way, very appreciated. Thanks for your time, Charles ===================================== Charles Girardot Head of Genome Biology Computational Support (GBCS) European Molecular Biology Laboratory Tel: +49 6221 387 -8585 Fax: +49-(0)6221-387-8166 Email: charles.girardot(a)embl.de Room V205 Meyerhofstraße 1, 69117 Heidelberg, Germany =====================================

2 1

Problem with <change_format> and conditional inputs?
by Peter Cock 14 Mar '14

14 Mar '14

Hi all, An eagle eyed user has just spotted a bug in our BLAST wrappers and/or the Galaxy framework itself. 1. Run BLASTN (wrapper version v0.1.00 which is currently on the Test Tool Shed) 2. Select BLASTXML output, execute 3. While running click the "i" icon, which wrongly reports: Format:tabular 4. Check the "repeat", which correctly has XML output 5. Once finished, notice the output is XML, but Galaxy labelled it "tabular" 6. Workaround via the "pencil" icon, correct the datatype to "blastxml". This happens both on our production server: $ hg summary parent: 11218:26f58e05aa10 release_2013.11.04 Galaxy stable release for 2013.11.04. branch: stable And also on my development server with galaxy-central: $ hg summary parent: 13675:bc1173309bd5 tip Variable name fix for the tool shed's install and test framework. branch: default This used to work perfectly. We've not changed this area of the code in the BLAST wrappers directly - it uses a default output type of "tabular" and the <change_format> tag - expanding the macro we have: <outputs> <data name="output1" format="tabular" label="${blast_type.value} $query.name vs @ON_DB_SUBJECT@">  <change_format> <when input="out_format" value="0" format="txt"/> <when input="out_format" value="0 -html" format="html"/> <when input="out_format" value="2" format="txt"/> <when input="out_format" value="2 -html" format="html"/> <when input="out_format" value="4" format="txt"/> <when input="out_format" value="4 -html" format="html"/> <when input="out_format" value="5" format="blastxml"/> </change_format> </data> </outputs> What has changed recently in the NCBI BLAST wrappers is that input parameter "out_format" is now nested within a conditional block. Other tools using <change_format> seem to still work fine: https://github.com/peterjc/pico_galaxy/tree/master/tools/seq_filter_by_id Can anyone testing the BLAST+ update on the Test Tool Shed reproduce this? Any thoughts on the cause and fix? I have to attend a meeting now, so I've not dug any deeper yet... Regards, Peter

3 3

Write permission for database/files
by Alper Kucukural 14 Mar '14

14 Mar '14

Hi All, Galaxy opens a directory under database/files when the #of data files increased. So, it goes 1,2,3 … etc. I want to give a write permission for these specific directories when they are created without changing umask in the user profile. Is there anybody who can tell me where galaxy creates these directories. Thanks in advance. Best, Alper

2 1

Quota History not getting purged
by Alexander Kurze 14 Mar '14

14 Mar '14

Hi there, We have installed a local copy of galaxy but there seems to be a strange behavior with deleting data sets in the history panel. A few users complain that they have deleted their data sets (I can confirm that) but somehow the quota limit still doesn't show the appropriate freed space. Looking at their saved histories and adding up their data the quota limit should have changed. Any clues why this happens? I have also tried to run the clean up script but this didn't do anything on their quota limits. Best, Alex

2 3

Purging datasets as part of workflow?
by Sveinung Gundersen 14 Mar '14

14 Mar '14

Hi, Is there an automatic way to permanently delete and purge datasets that are the result of intermittent tools in a workflow? That is, how can one automatically keep only the first and the last steps of an workflow without having to manually delete all the intermittent datasets and running the purge scripts? Thanks, Sveinung Gundersen -- Sveinung Gundersen, PhD. Head engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway Email: sveinung.gundersen(a)medisin.uio.no. Phone: +47 93 00 94 54

3 2

Pick-you-own columns in BLAST+ tabular output
by Peter Cock 14 Mar '14

14 Mar '14

Hello all, I've mentioned before that a forthcoming update to the BLAST+ Galaxy wrappers would be adding a most customisable output option with pick-your-own columns. This is available to test now from the Test Tool Shed, along with other enhancements like more masking tools and an update to BLAST+ 2.2.29: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus (This will be released as the ncbi_blast_plus wrappers v0.1.0, and includes recent contributions from Jim Johnson and Björn Grüning - thank you both): Previously (for tabular output) we've only offered the standard 12 column output (-outfmt 6), and an extended set which has grown to currently include 25 columns. The new custom column mode offers the 12 standard columns, and the extended set (making 25 columns), plus other (to me less interesting) identifiers, some other miscellaneous columns, and the taxonomy columns (44 columns in all): [image: Inline image 1] [image: Inline image 2] As previously discussed, the list of columns has been grouped - the first two groups reflect the standard 12 columns and the columns used in Galaxy's (default) 25 column extended tabular output. The remaining columns I have split into three groups (assorted identifiers, miscellaneous, and taxonomy) which I hope is self explanatory. Note that this does *not* allow you pick the column order - they will be produced in the order shown. The underlying BLAST+ tools would support this, but the Galaxy GUI would I feel be too complicated. Some community testing/comments would be appreciated. Thanks, Peter

2 3