March 2014 - galaxy-dev - lists.galaxyproject.org

Internal server error when importing workflows!
by Hakeem Almabrazi 15 Mar '14

15 Mar '14

Hi, I have upgraded my local galaxy to the latest stable version and it works fine but I get an error when I try to import workflows that were created older version of Galaxy. Here is the error I keep getting. Internal Server Error Galaxy was unable to successfully complete your request URL: http://10.7.10.21/workflow/import_workflow Module galaxy.web.framework.middleware.error:149 in __call__ >> <http://10.7.10.21/workflow/import_workflow> app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ >> <http://10.7.10.21/workflow/import_workflow> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> <http://10.7.10.21/workflow/import_workflow> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> <http://10.7.10.21/workflow/import_workflow> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request >> <http://10.7.10.21/workflow/import_workflow> body = method( trans, **kwargs ) Module galaxy.webapps.galaxy.controllers.workflow:1072 in import_workflow >> <http://10.7.10.21/workflow/import_workflow> workflow, missing_tool_tups = self._workflow_from_dict( trans, data, source=src, add_to_menu=add_to_menu ) Module galaxy.web.base.controller:1639 in _workflow_from_dict >> <http://10.7.10.21/workflow/import_workflow> module.save_to_step( step ) Module galaxy.workflow.modules:274 in save_to_step >> <http://10.7.10.21/workflow/import_workflow> step.tool_inputs = self.tool.params_to_strings( self.state.inputs, self.trans.app ) Module galaxy.tools:2418 in params_to_strings >> <http://10.7.10.21/workflow/import_workflow> return params_to_strings( self.inputs, params, app ) Module galaxy.tools.parameters:85 in params_to_strings >> <http://10.7.10.21/workflow/import_workflow> value = params[ key ].value_to_basic( value, app ) Module galaxy.tools.parameters.grouping:468 in value_to_basic >> <http://10.7.10.21/workflow/import_workflow> rval[ input.name ] = input.value_to_basic( value[ input.name ], app ) KeyError: 'index' Can someone let me know how to fix this issue? Thank you Hakeem

2 2

How To properly use GALAXY_SLOTS
by Geert Vandeweyer 15 Mar '14

15 Mar '14

Hi, Is there documentation for the proper setup and utilisation of the \${GALAXY_SLOTS:-4} style options? I noticed that tools with this setting, including BWA and GATK run single threaded, with the following settings respectively: BWA \${GALAXY_SLOTS:-4}" === ps output: python /galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_wrappers/bwa_wrapper.py --threads=1 --fileSource=indexed --ref=/galaxy/galaxy_references/hg19/bwa-0.5.9/base/hg19.fasta --do_not_build_index --input1=/galaxy/galaxy-dist/database/files/093/dataset_93021.dat --input2=/galaxy/galaxy-dist/database/files/093/dataset_93023.dat --output=/galaxy/galaxy-dist/database/files/000/112/dataset_112301.dat --genAlignType=paired --params=pre_set --suppressHeader=false xml config: --threads="\${GALAXY_SLOTS:-4}" GATK Base Recalibrator \${GALAXY_SLOTS:-8} ============== python /galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/gatk2_wrapper.py --max_jvm_heap 6g --stdout /galaxy/galaxy-dist/database/files/000/112/dataset_112292.dat -d -I /galaxy/galaxy-dist/database/files/000/112/dataset_112289.dat bam gatk_input -d /galaxy/galaxy-dist/database/files/_metadata_files/012/metadata_12686.dat bam_index gatk_input -p java -jar "$GATK2_PATH/GenomeAnalysisTK.jar" -T "BaseRecalibrator" $GATK2_SITE_OPTIONS --num_cpu_threads_per_data_thread ${GALAXY_SLOTS:-8} --no_standard_covs -R "/galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta" --out "/galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat" -cov "ContextCovariate" -cov "CycleCovariate" -d --knownSites:dbsnp,%(file_type)s /galaxy/galaxy-dist/database/files/000/101/dataset_101086.dat vcf input_dbsnp_0 -p --pedigreeValidationType "STRICT" -d --intervals /galaxy/galaxy-dist/database/files/000/100/dataset_100129.dat bed input_intervals_0 -p --interval_set_rule "UNION" -p --downsampling_type "NONE" -p --baq "OFF" --baqGapOpenPenalty "40.0" --defaultBaseQualities "-1" --validation_strictness "STRICT" --interval_merging "ALL" java -Xmx6g -jar /galaxy/galaxy-dist/tool-data/shared/jars/gatk2//GenomeAnalysisTK.jar -T BaseRecalibrator --num_cpu_threads_per_data_thread 1 --no_standard_covs -R /galaxy/galaxy_references/hg19/srma-0.1.15/hg19.fasta --out /galaxy/galaxy-dist/database/files/000/112/dataset_112291.dat -cov ContextCovariate -cov CycleCovariate --pedigreeValidationType STRICT --interval_set_rule UNION --downsampling_type NONE --baq OFF --baqGapOpenPenalty 40.0 --defaultBaseQualities -1 --validation_strictness STRICT --interval_merging ALL -I /tmp/tmp-gatk-fIuncH/gatk_input.bam --knownSites:dbsnp,vcf /tmp/tmp-gatk-fIuncH/input_dbsnp_0.vcf --intervals /tmp/tmp-gatk-fIuncH/input_intervals_0.bed Best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer

4 5

How to install bowtie2 tool in galaxy
by Ravi Alla 15 Mar '14

15 Mar '14

Hi guys, I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions. - Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up? - What are repository dependencies? Why would I need these if bowtie2 is already installed on my system? - I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct? - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices. I hope someone on here can help me out. Thanks a lot Ravi.

5 11

Dataset's extra files
by Charles Girardot 15 Mar '14

15 Mar '14

Hi all, We have a local tool which role is to transfer (ie copy) a dataset file to a directory on our NFS. This is extremely convenient as it can be included within workflows and therefore save the time of clicking download button (we also have configurable renaming/compression as part of it). It is heavily used by our users. The problem is with datasets that have associated files like FASTQC as these extra files are simple not ignored... We'd like to improve our 'NFS_transfer' tool so it can deal with this in a similar fashion as the download button. Foreseen solution : * Check if a directory named 'dataset_<id>_files' exists within the dataset store * if so, 'cp -r' it into a tmp dir, cp the dataset itself into same tmp dir (with renaming on the fly) * zip/tar.gz the tmp dir * copy it to final NFS location Question is : is this the right way to do it ? As a non python specialist, it is a little tricky to find the right way to it (I can t locate the piece of code that does this in galaxy ie behind the download button). In particular, can I get the list of extra files using the '$galaxyFile' object given in the tool by : <param type="data" name="galaxyFile" label="File to transfer"/> i.e. in the same way we get the dataset name or file extension ($galaxyFile.dataset.name and $galaxyFile.ext) ? Any advise on how best to implement this, in a portable way, very appreciated. Thanks for your time, Charles ===================================== Charles Girardot Head of Genome Biology Computational Support (GBCS) European Molecular Biology Laboratory Tel: +49 6221 387 -8585 Fax: +49-(0)6221-387-8166 Email: charles.girardot(a)embl.de Room V205 Meyerhofstraße 1, 69117 Heidelberg, Germany =====================================

2 1

Problem with <change_format> and conditional inputs?
by Peter Cock 15 Mar '14

15 Mar '14

Hi all, An eagle eyed user has just spotted a bug in our BLAST wrappers and/or the Galaxy framework itself. 1. Run BLASTN (wrapper version v0.1.00 which is currently on the Test Tool Shed) 2. Select BLASTXML output, execute 3. While running click the "i" icon, which wrongly reports: Format:tabular 4. Check the "repeat", which correctly has XML output 5. Once finished, notice the output is XML, but Galaxy labelled it "tabular" 6. Workaround via the "pencil" icon, correct the datatype to "blastxml". This happens both on our production server: $ hg summary parent: 11218:26f58e05aa10 release_2013.11.04 Galaxy stable release for 2013.11.04. branch: stable And also on my development server with galaxy-central: $ hg summary parent: 13675:bc1173309bd5 tip Variable name fix for the tool shed's install and test framework. branch: default This used to work perfectly. We've not changed this area of the code in the BLAST wrappers directly - it uses a default output type of "tabular" and the <change_format> tag - expanding the macro we have: <outputs> <data name="output1" format="tabular" label="${blast_type.value} $query.name vs @ON_DB_SUBJECT@">  <change_format> <when input="out_format" value="0" format="txt"/> <when input="out_format" value="0 -html" format="html"/> <when input="out_format" value="2" format="txt"/> <when input="out_format" value="2 -html" format="html"/> <when input="out_format" value="4" format="txt"/> <when input="out_format" value="4 -html" format="html"/> <when input="out_format" value="5" format="blastxml"/> </change_format> </data> </outputs> What has changed recently in the NCBI BLAST wrappers is that input parameter "out_format" is now nested within a conditional block. Other tools using <change_format> seem to still work fine: https://github.com/peterjc/pico_galaxy/tree/master/tools/seq_filter_by_id Can anyone testing the BLAST+ update on the Test Tool Shed reproduce this? Any thoughts on the cause and fix? I have to attend a meeting now, so I've not dug any deeper yet... Regards, Peter

3 3

Write permission for database/files
by Alper Kucukural 15 Mar '14

15 Mar '14

Hi All, Galaxy opens a directory under database/files when the #of data files increased. So, it goes 1,2,3 … etc. I want to give a write permission for these specific directories when they are created without changing umask in the user profile. Is there anybody who can tell me where galaxy creates these directories. Thanks in advance. Best, Alper

2 1

Quota History not getting purged
by Alexander Kurze 15 Mar '14

15 Mar '14

Hi there, We have installed a local copy of galaxy but there seems to be a strange behavior with deleting data sets in the history panel. A few users complain that they have deleted their data sets (I can confirm that) but somehow the quota limit still doesn't show the appropriate freed space. Looking at their saved histories and adding up their data the quota limit should have changed. Any clues why this happens? I have also tried to run the clean up script but this didn't do anything on their quota limits. Best, Alex

2 3

Purging datasets as part of workflow?
by Sveinung Gundersen 15 Mar '14

15 Mar '14

Hi, Is there an automatic way to permanently delete and purge datasets that are the result of intermittent tools in a workflow? That is, how can one automatically keep only the first and the last steps of an workflow without having to manually delete all the intermittent datasets and running the purge scripts? Thanks, Sveinung Gundersen -- Sveinung Gundersen, PhD. Head engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway Email: sveinung.gundersen(a)medisin.uio.no. Phone: +47 93 00 94 54

3 2

Pick-you-own columns in BLAST+ tabular output
by Peter Cock 14 Mar '14

14 Mar '14

Hello all, I've mentioned before that a forthcoming update to the BLAST+ Galaxy wrappers would be adding a most customisable output option with pick-your-own columns. This is available to test now from the Test Tool Shed, along with other enhancements like more masking tools and an update to BLAST+ 2.2.29: http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus (This will be released as the ncbi_blast_plus wrappers v0.1.0, and includes recent contributions from Jim Johnson and Björn Grüning - thank you both): Previously (for tabular output) we've only offered the standard 12 column output (-outfmt 6), and an extended set which has grown to currently include 25 columns. The new custom column mode offers the 12 standard columns, and the extended set (making 25 columns), plus other (to me less interesting) identifiers, some other miscellaneous columns, and the taxonomy columns (44 columns in all): [image: Inline image 1] [image: Inline image 2] As previously discussed, the list of columns has been grouped - the first two groups reflect the standard 12 columns and the columns used in Galaxy's (default) 25 column extended tabular output. The remaining columns I have split into three groups (assorted identifiers, miscellaneous, and taxonomy) which I hope is self explanatory. Note that this does *not* allow you pick the column order - they will be produced in the order shown. The underlying BLAST+ tools would support this, but the Galaxy GUI would I feel be too complicated. Some community testing/comments would be appreciated. Thanks, Peter

2 3

tool_sheds_errors
by Wang, Xiaofei 14 Mar '14

14 Mar '14

Hi Folks, I figured out the problem of tool dependencies successfully. The installation status is installed for all tool sheds. The tools are also shown on left tool panel. But, I got problems when I used the tool sheds to analyze data. For example, When I used GFF-TO-BED (fml_gff3togtf), the error is: python: can't open file '~/softwares/shed_tools/toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf/ed53dca1c6ff/fml_gff3togtf/fml_gff_converter_programs /galaxy/bed_to_gff3_converter.py': [Errno 2] No such file or directory When I checked the ~/.../galaxy folder, there are four *.xml files. Then, I copied the four files from ~/.../script folder. Now, this tool works. But, I don't think this is a right way to figure out this because I also got similar problem (I think they are similar) when I used other tool sheds as below. So, I am guessing it might be a problem with my configure file (universe_wsgi.ini). Could anybody tell me why and how to figure it out? BWA wrapper error: when I used the BWA wrapper for Illunina, An error occurred with this dataset: Could not determine BWA version Error indexing reference sequence. /bin/sh: bwa: command not found I referred to this link to try to figure out the problem http://osdir.com/ml/galaxy-development-source-control/2011-10/msg00268.html. But, I think I need more details about the mentioned bwa binaries and path. When I used the snpEff, the error is: Unable to access jarfile /Users/pablocingolani/snpEff/snpEff.jar In fact, the snpEff.jar is located at ~/softwares/shed_tools/toolshed.g2.bx.psu.edu/repos/pcingola/snpeff/c052639fa666/snpeff/snpEff_2_1a/snpEff_2_1a Could you give me some ideas? Thank you so much! ________________________________ From: Greg Von Kuster [greg(a)bx.psu.edu] Sent: Tuesday, March 11, 2014 8:47 AM To: Wang, Xiaofei Cc: galaxy-dev(a)lists.bx.psu.edu Subject: Re: [galaxy-dev] tool_dependency_dir (Installed, missing tool dependencies) Hello, It's difficult to determine the cause of the problem in your environment based on the details you've provided. If you have defined your tool_dependeny_dir configuration setting in your universe_wsgi.ini to be ../tool_dependencies, then the tool shed installation process will install tool dependencies there. If you see them installed elsewhere, then that setting must have been set differently at the time they were installed. You can try setting it to the location where your tool dependencies have been installed (i.e., the Galaxy installation directory), restat your server and uninstall them. Then you can reset it to the more appropriate location (e.g., ../tool_dependencies), restart yiour server and reinstall them. If you have tool dependencies installed in multiple locations, you'll have to be careful to use a process that will eventually result in all tool dependencies installed into a single location. Greg Von Kuster On Mar 6, 2014, at 5:41 PM, "Wang, Xiaofei" <xfwang(a)ku.edu<mailto:xfwang@ku.edu>> wrote: Dear there, I am trying to install some tools from toolshed on my local galaxy instance, like bwa_wrappers, package_picard_1_56_0, package_samtools_0_1_18, and sam_to_bam. For the first time, I got a warning before installing, something like this " set the value of your tool_dependency_dir setting in universe_wsgi_ini and restart before installing". Then, I followed the answer on this thread (Problem installing tool_ Galaxy local) as below. I edited the universe_wsgi_ini like this: #Path to the directory in which managed tool dependencies are placed. To use # the dependency system, see the documentation at: # http://wiki.g2.bx.psu.edu/Admin/Config/<http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies>Tool<http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies>%20Dependencies<http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies> #tool_dependency_dir = None tool_dependency_dir = ../tool_dependencies Then, I restarted and installed the tool again. Galaxy create a directory named tool_dependencies at the same level in local file system hierarchy as the galaxy installation directory (same level as galaxy-dist). Some softwares such as bwa and picard are installed automatically here. But, when I checked in the admin "manage installed tool shed repositories", the installation status is "Installed, missing tool dependencies" for the four tools I mentioned above. I do not know why this happened in my case? I tried to figure it out follow this thread "The migrated BWA installed but missing dependencies". But, I did not get the idea. Could you tell me why this happen and how to fix it up? Thank you so much! ------------------------------------------------------------------------------------------ Your setting for tool_dependency_dir should not be as you've stated: tool_dependency_dir = tool_dependencies It should be an actual path on your local file system, something like: tool_dependency_dir = ../tool_dependencies If you use the above setting, then Galaxy should create a directory named tool_dependency_dir at the same level in your local file system hierarchy as your Galaxy installation directory. I cannot guarantee that this change will solve your problem, but it will be a step in the right direction. After making this change, let us know if you still see problems. Greg Von Kuster ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/

2 1