- galaxy-dev - lists.galaxyproject.org

tool and planemo problems
by Matthias Bernt 12 Jul '17

12 Jul '17

Dear list, In order to test the job resubmission feature I wrote a little tool for galaxy that just wastes a specified (minimum) time and memory (using different programming languages) for testing the new DRMAA job runner that I wrote. When linting it with planemo seemingly all tests are successful, but in the end I still get a fail -- verbose mode is also of no help here. Applying linter tests... CHECK .. CHECK: 7 test(s) found. Applying linter output... CHECK .. INFO: 1 outputs found. Applying linter inputs... CHECK .. INFO: Found 3 input parameters. Applying linter help... CHECK .. CHECK: Tool contains help section. .. CHECK: Help contains valid reStructuredText. Applying linter general... CHECK .. CHECK: Tool defines a version [0.1.0]. .. CHECK: Tool defines a name [Waste time and memory]. .. CHECK: Tool defines an id [sleepwaste]. .. CHECK: Tool targets 16.01 Galaxy profile. Applying linter command... CHECK .. INFO: Tool contains a command. Applying linter citations... WARNING .. WARNING: No citations found, consider adding citations to your tool. Applying linter tool_xsd... CHECK .. INFO: File validates against XML schema. Failed linting The test of the tool are all successful. Any ideas what is wrong? Once the problems are solved, would such a tool also be useful in the toolshed? Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 6

Issues with "bamtools split", running jobs don't show up in history (and fail after several hours)
by Previti 12 Jul '17

12 Jul '17

Dear all, I've been trying to properly install the "BAMTools split" tool in our local DKFZ-Galaxy instance, but it doesn't work properly. The strange thing is that you can launch a bamtools splitter job on our cluster, but it never shows up in the history...until it fails with following "error": "tool error An error occurred with this dataset: Unable to finish job" All versions (Galaxy, bamtools 2.4.0,samtools 1.3.1) are up to date and this is what I installed: Repository bamtools_split Name: bamtools_split Owner: devteam Synopsis: Tool for splitting BAM files Detailed description: Utility for splitting BAM files. It is based on BAMtools suite of tools by Derek Barnett. Content homepage: https://github.com/pezmaster31/bamtools Development repository: https://github.com/galaxyproject/tools-devteam/tree/master/tool_collections… Revision: 2:e2c1b62e1db1 Thanks and best regards, Christopher -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4434 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

1 0

Java memory requirements for Picard index?
by Previti 11 Jul '17

11 Jul '17

Dear all, I seemly have a java issue (out of memory) when trying to construct a picard index for the rat (rn6) genome build using the picard_index_builder_data_manager. This is strange because other, bigger genomes have not presented any problems... Does anybody have any suggestions? Cheers, Christopher Previti Fatal error: Exit code 1 () [Thu Jul 06 08:26:48 UTC 2017] picard.sam.CreateSequenceDictionary REFERENCE=/opt/galaxy/galaxy/database/jobs_directory/000/978/dataset_1974_files/rn6.fa OUTPUT=rn6.dict TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json [Thu Jul 06 08:26:48 UTC 2017] Executing as galaxy(a)xgs-node071.xgs.dkfz-heidelberg.de on Linux 3.10.0-514.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_92-b15; Picard version: 2.7.1-SNAPSHOT [Thu Jul 06 08:26:49 UTC 2017] picard.sam.CreateSequenceDictionary done. Elapsed time: 0.03 minutes. Runtime.totalMemory()=1051721728 To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at htsjdk.samtools.reference.FastaSequenceFile.readSequence(FastaSequenceFile.java:151) at htsjdk.samtools.reference.FastaSequenceFile.nextSequence(FastaSequenceFile.java:81) at picard.sam.CreateSequenceDictionary.makeSequenceDictionary(CreateSequenceDictionary.java:156) at picard.sam.CreateSequenceDictionary.doWork(CreateSequenceDictionary.java:137) at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208) at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95) at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105) Error building index. -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4434 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

2 2

any introduce for how Mako and JavaScript work in galaxy?
by Steven Shen 11 Jul '17

11 Jul '17

Hi everyone, I want to make some change for my local galaxy web, actually I can edit galaxy_path/static/scripts/bundled/*.js directly. It seems mako templates are used for galaxy, as well as many JavaScript files, but I don't know how they work. So is there any introduce for how mako and JavaScript files working in galaxy? Thank you very much. Steven

3 3

Re: [galaxy-dev] Improved DRMAAJobRunner (Peter van Heusden)
by Matthias Bernt 10 Jul '17

10 Jul '17

Dear Peter, my job_conf.xml is attached. Best, Matthuas On 08.07.2017 18:00, galaxy-dev-request(a)lists.galaxyproject.org wrote: > Send galaxy-dev mailing list submissions to > galaxy-dev(a)lists.galaxyproject.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.galaxyproject.org/listinfo/galaxy-dev > or, via email, send a message with subject or body 'help' to > galaxy-dev-request(a)lists.galaxyproject.org > > You can reach the person managing the list at > galaxy-dev-owner(a)lists.galaxyproject.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of galaxy-dev digest..." > > > HEY! This is important! If you reply to a thread in a digest, please > 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread. > 2. Strip out everything else in the digest that is not part of the thread you are responding to. > > Why? > 1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not. > 2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative. > > Today's Topics: > > 1. Re: any introduce for how Mako and JavaScript work in galaxy? > (Daniel Blankenberg) > 2. Re: ustacks - bad interpreter (Björn Grüning) > 3. Re: Improved DRMAAJobRunner (Peter van Heusden) (Matthias Bernt) > 4. Re: Improved DRMAAJobRunner (Peter van Heusden) > (Peter van Heusden) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 6 Jul 2017 22:36:27 -0400 > From: Daniel Blankenberg <dan.blankenberg(a)gmail.com> > To: Steven Shen <ishenweiyan(a)gmail.com> > Cc: galaxy-dev(a)lists.galaxyproject.org > Subject: Re: [galaxy-dev] any introduce for how Mako and JavaScript > work in galaxy? > Message-ID: > <CAE9nRFg+Edj2=3+K=6+FZCxj5=hb4toGW+Lm++E+SD1+J2+vbw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Steven, > > For a quick first pass, have you taken a look at https://github.com/ > galaxyproject/galaxy/blob/dev/client/README.md? > > > Thanks for using Galaxy, > > Dan > > On Thu, Jul 6, 2017 at 10:11 PM, Steven Shen <ishenweiyan(a)gmail.com> wrote: > >> Hi everyone, >> >> I want to make some change for my local galaxy web, actually I can edit >> galaxy_path/static/scripts/bundled/*.js directly. >> >> It seems mako templates are used for galaxy, as well as many JavaScript >> files, but I don't know how they work. So is there any introduce for how >> mako and JavaScript files working in galaxy? >> >> Thank you very much. >> >> Steven >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/ >> >

2 2

Fwd: Mismatched Files
by Changsu Dong 10 Jul '17

10 Jul '17

Hello, I am a galaxy user at Hunter College in New York. Recently we have encountered mismatching files throughout various tools under the same history. We are running our RNAseq analysis on the Dockerized containers. I need the BAM files (accepted hits) which produced by Tophats for the MulticovBed to get the read count. Those BAM files should be the binary 'accepted hit' files but instead they are showing random output from other tools(see the image below). This not only happened to BAM files but for most of the tools and most of the datasets. For example, the output for Groomer is not actually from Groomer but rather from the Tophat 'deletion' or an output from Cufflinks. We are not sure if you are familiar with this issue whether it's a container or a Galaxy known issue. It would be great if you could point me to a right direction on how to resolve this issue. [image: Inline image 1] Sincerely, Changsu

2 1

How to use the input freebayes "input_bams" to rename output
by Charles Girardot 10 Jul '17

10 Jul '17

Hi all, I am a big fan of dataset renaming in my workflows, it helps a lot when you process tons of samples in parallel. I have integrated FreeBayes in a workflow but renaming the output to “#{ input_bams }.vcf” fails. I remember that when the input is a list, one needs to use some weird formulas, for example in the tool “Filter BAM datasets on a variety of attributes (Galaxy Tool Version 0.0.1)” that also defines a “input_bams", we could solve the issue with : #{input_bams_0.input_bam} So I mimicked this and tried #{input_bams_0.input_bam} , #{input_bams_0.input} and #{input_bams_0.bam} without success. Does anyone has an idea what the correct spelling is for Freebayes ? I am sorry Jelle (our galaxy admin) is away this week so I can’t tell you our server version but it is ~2 years old. Thanks Charles ===================================== Charles Girardot Head of Genome Biology Computational Support (GBCS) and Senior Bioinformatician in the Furlong Lab European Molecular Biology Laboratory Tel: +49 6221 387 -8585 Fax: +49-(0)6221-387-8166 Email: charles.girardot(a)embl.de Web : http://gbcs.embl.de Room V205 Meyerhofstraße 1, 69117 Heidelberg, Germany =====================================

1 0

Re: [galaxy-dev] Improved DRMAAJobRunner (Peter van Heusden)
by Matthias Bernt 08 Jul '17

08 Jul '17

Hi Peter, > Code looks interesting. This is for Univa Grid Engine? In its current state its tailored for the Univa Grid Engine. But the drmaa library code that is based on job_info() and wait() should run on all grid engines. For the command line based code changes might be necessary for the different grid engines. (Currently there is only a small bug, because wait() is currently called twice for finished jobs: in the repeated polling and then in the final check. The second call won't work which causes a fallback to qacct. I will fix this soon.) I guess that the output of qstat and qacct will be different. But I think this can be configured one just needs a way to get the info which grid engine is running. > In terms of the upload jobs, are those not designed to be run as 'local' jobs and not with the 'real user' setting? Sounds reasonable, but this is not what is happening on my installation of galaxy. Any idea where I could start to find the problem. Best, Matthias

2 1

ustacks - bad interpreter
by David Meltzer 08 Jul '17

08 Jul '17

Good morning, I have a user attempting to run ustacks. They are receiving the following error Fatal error: Exit code 126 (Error in Stacks execution) /home/big_galaxy/galaxy/database/jobs_directory/079/79363/tool_script.sh: /home/big_galaxy/galaxy/tool_dependencies/_conda/envs/mulled-v1-8e6d35eac718a13db6458b9361ca87c93be9feed656f9938ebcc435642f3a0a3/bin/stacks_summary.py: /home/big_galaxy/galaxy/tool_dependencies/_conda/envs/mulled-v1-8e6d35eac718a1: bad interpreter: No such file or directory The tool produces an additional output on the bug report – ustacks parameters selected: Sample ID: 1 Min depth of coverage to create a stack: 3 Max distance allowed between stacks: 2 Max distance allowed to align secondary reads: 4 Max number of stacks allowed per de novo locus: 3 Deleveraging algorithm: disabled Removal algorithm: enabled Model type: SNP Alpha significance level for model: 0.05 Gapped alignments: disabled Parsing stacks_inputs/ZU-6.fq Loading RAD-Tags...done Loaded 933120 RAD-Tags. Inserted 256463 elements into the RAD-Tags hash map. 0 reads contained uncalled nucleotides that were modified. 65509 initial stacks were populated; 190954 stacks were set aside as secondary reads. Initial coverage mean: 10.9276; Std Dev: 25.3864; Max: 4650 Deleveraging trigger: 36; Removal trigger: 62 Calculating distance for removing repetitive stacks. Distance allowed between stacks: 1; searching with a k-mer length of 35 (36 k-mers per read); 1 k-mer hits required. Removing repetitive stacks. Removed 993 stacks. 64677 stacks remain for merging. Post-Repeat Removal, coverage depth Mean: 10.2655; Std Dev: 9.03472; Max: 62 Calculating distance between stacks... Distance allowed between stacks: 2; searching with a k-mer length of 23 (48 k-mers per read); 2 k-mer hits required. Merging stacks, maximum allowed distance: 2 nucleotide(s) 64677 stacks merged into 57327 loci; deleveraged 0 loci; blacklisted 189 loci. After merging, coverage depth Mean: 11.0747; Std Dev: 9.82823; Max: 91 Merging remainder radtags 217265 remainder sequences left to merge. Distance allowed between stacks: 4; searching with a k-mer length of 13 (58 k-mers per read); 6 k-mer hits required. Matched 26211 remainder reads; unable to match 191054 remainder reads. After remainders merged, coverage depth Mean: 11.5347; Std Dev: 9.95581; Max: 93 Calling final consensus sequences, invoking SNP-calling model... Number of utilized reads: 742066 Writing loci, SNPs, and alleles to 'stacks_outputs/'... Refetching sequencing IDs from stacks_inputs/ZU-6.fq... read 933120 sequence IDs. done. ustacks is done. I am not sure what is causing the bad interpreter: No such file or directory. The tools are correctly installed along with its requisite conda packages. Any help would be greatly appreciated. Best Regards, David

4 8

Improved DRMAAJobRunner
by Matthias Bernt 07 Jul '17

07 Jul '17

Dear list, I have implemented a new DRMAA job runner allows to detect runtime and memory violations. This is done by using the drmaa library functions job_info + wait or as fallback the commandline tools qstat + qacct. This has the additional advantage that the runner works also in setups using the external runner scripts to start jobs as real user (the original DRMAAJobRunner can not query these jobs at all). You may have a look here: https://github.com/galaxyproject/galaxy/pull/4275 I have successfully tested the but settings: galaxy user/real user submits the jobs. Also the resubmission in case of memory and time violations seems to work. Would be great to get some comments. One problem that I encountered is that upload jobs do not work in the real user setting (for the original and the new runner): The permissions of the uploaded file in /gpfs1/data/galaxy_server/galaxy-dev/database/tmp/ are not changed. Any idea what needs to be changed to get this running? Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 1