April 2018 - galaxy-dev - lists.galaxyproject.org

Monitoring Dashboard for Galaxy
by evan clark 03 Sep '18

03 Sep '18

I remember there previously being an additional tool that ran on a different port from galaxy that allowed for monitoring of performance and jobs running. Is this tool still packaged with galaxy and if so how can it be activated.

3 3

Job Script Integrity
by John Letaw 04 Jun '18

04 Jun '18

Hi all, I’m installing via GalaxyKickStart… I’m getting the following error: galaxy.jobs.runners ERROR 2017-11-06 19:14:05,263 (19) Failure preparing job Traceback (most recent call last): File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py", line 175, in prepare_job modify_command_for_container=modify_command_for_container File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py", line 209, in build_command_line container=container File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py", line 84, in build_command externalized_commands = __externalize_commands(job_wrapper, external_command_shell, commands_builder, remote_command_params) File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py", line 143, in __externalize_commands write_script(local_container_script, script_contents, config) File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/util/job_script/__init__.py", line 112, in write_script _handle_script_integrity(path, config) File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/util/job_script/__init__.py", line 147, in _handle_script_integrity raise Exception("Failed to write job script, could not verify job script integrity.") Exception: Failed to write job script, could not verify job script integrity. galaxy.model.metadata DEBUG 2017-11-06 19:14:05,541 Cleaning up external metadata files galaxy.model.metadata DEBUG 2017-11-06 19:14:05,576 Failed to cleanup MetadataTempFile temp files from /home/exacloud/lustre1/galaxydev/galaxyuser/database/jobs/000/19/metadata_out_HistoryDatasetAssociation_16_I8bhLX: No JSON object could be decoded I would like to further understand what it means to not verify integrity of a job script. Does this just mean there is a permissions error? Ownership doesn’t match up? Thanks, John

2 1

Text file busy and conda
by John Letaw 16 May '18

16 May '18

Hi all, I’m receiving the following error trying to access tools autoinstalled via conda: Fatal error: Exit code 126 () ~/galaxy/database/jobs_directory/000/38/tool_script.sh: line 41: ~/galaxy/database/dependencies/_conda/envs/mulled-v1-bb83f93efd111e042823e53ddfa74c32d81ba74cceca9445dfddfc9e940ff738/bin/samtools: Text file busy So, something is happening too fast I guess, meaning a process is attempting access while this is being created. Any ideas on how I can diagnose this? Not seeing any other errors floating around the logs. Maybe we are dealing with a lustre file locking issue? Thanks, John

2 2

Best practices for utilising Unique Molecular Indexing (UMI's)
by Brooks, Tony 04 May '18

04 May '18

Hi We are currently seeing a number of methods that are utilising the power of unique molecular indexing. Unfortunately, there is no consensus on how libraries should be configured, and therefore no consensus for how to deal with them within Galaxy. Often libraries that have the UMI placed directly downstream of the first (i7) index, such as ones using the IDT xGen adapter set (https://www.idtdna.com/pages/products/next-generation-sequencing/adapters/x…) Sometimes UMI's exist in place of the second (i5) index (https://www.neb.com/nebnext-direct/nebnext-direct-for-target-enrichment) In both cases, the recommended workflows are convoluted and all the necessary tools do not currently exist in the toolshed (so that the datasets need to be taken out of galaxy, processed and reloaded). It is possible to use bcl2fastq to output the UMI as an additional fastq file, but this would then require me to create a dataset triplicate (not pair) which afaik we can't do (yet). A quick Google/toolshed search had me find UMI-Tools & Je-Suite which both exist in Galaxy. Both these tools assume the UMI is "in-line" (i.e. at the beginning of the read 1 or read2 - not its own read), extract/remove the UMI and place it in the read header, where it is then used further down the line to dedup the bam file. Does anyone know of any tools that would take the UMI from a separate fastq and use it to tag the headers of actual read data. Or alternatively, a tool that will paste the UMI tag onto the 5' end of the read fastq? And whether these steps can be done within Galaxy, or maybe prior to fastq upload? Anyone have a method/workflow for UMI's? Thanks in advance Tony

2 6

extract extra info (logs) from history
by Jochen Bick 02 May '18

02 May '18

Hi, I would like to use the info (log) HISAT is showing, for another tool. e.g.: 26781724 reads; of these: 26781724 (100.00%) were unpaired; of these: 2445411 (9.13%) aligned 0 times 20399099 (76.17%) aligned exactly 1 time 3937214 (14.70%) aligned >1 times 90.87% overall alignment rate [bam_sort_core] merging from 21 f Is it possible to access these logs? Cheers Jochen

4 7

Gateways 2018 Call for Participation: Extended to May 16
by Katherine Lawrence 30 Apr '18

30 Apr '18

The deadline for the Gateways 2018 Call for Participation has been extended to Wednesday, May 16, 2018. Gateways 2018 (September 25-27, at the University of Texas at Austin) is accepting submissions of papers, demos, tutorials, and panels (2-4 pages) on the topic of science or engineering gateways. Topics may include their design, use, impact, development processes, sustainability, best practices, or any other aspect that you think fellow gateway creators or users will find interesting to learn. We also welcome educational topics directed toward the next generation of gateway creators. Submissions of papers, demos, tutorials, and panels are due May 16, 2018. There will be no further extension of this deadline. The poster session deadline (open to all) is August 1, 2018. Full submission details can be found here: https://sciencegateways.org/gateways2018/call

1 0

tool_conf.xml
by Nancy Ide 30 Apr '18

30 Apr '18

Hello I’m trying to include some of the tools available from usegalaxy.org in the tool menu of our instance of Galaxy. When I look at the list of tools in the Galaxy tool_conf.xml.main file, it does not seem to line up with what you see in the menu at usegalaxy.org. I’ve searched through various versions etc. on GitHub, but no luck—they all seem to be identical and do not seem to correspond with the menu at usegalaxy.org (I am looking specifically at the Text Manipulation part of the menu). Any clues on where I should look? Thanks, Nancy Ide ----------------------------------------------------- Nancy Ide Professor of Computer Science Department of Computer Science Vassar College Poughkeepsie, New York 12604-0520 USA tel: (+1 845) 437 5988 fax: (+1 845) 437 7498 email: ide(a)cs.vassar.edu http://www.cs.vassar.edu/~ide -----------------------------------------------------

2 2

Galaxy Cleanup
by Previti 27 Apr '18

27 Apr '18

Hi everybody, I have a hopefully basic questions regarding temporary output files. A user was generating >4TB output files via the program bedtools intersectbed. The jobs were cancelled, but I still see the (temporary) output files that were generated. Does Galaxy clean these up? Does it do this regularly? Would a quota prevent something like this from happening again or would the jobs have to finish before being "counted"? Thanks and best regards, Christopher -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

3 3

fonts on headless compute nodes
by Matthias Bernt 25 Apr '18

25 Apr '18

Dear list, I found that some tools (eg FastQC and MetaProSIP) have problems with missing fonts (the former is java and the latter caused by R) on our compute nodes which do not have xorg and no fonts installed. On the cluster head nodes several font related packages are installed: dejavu-fonts-common-2.33-1.el6.noarch xorg-x11-fonts-Type1-7.2-11.el6.noarch liberation-fonts-common-1.05.1.20090721-5.el6.noarch liberation-sans-fonts-1.05.1.20090721-5.el6.noarch dejavu-sans-fonts-2.33-1.el6.noarch urw-fonts-2.4-11.el6.noarch ghostscript-fonts-5.50-23.2.el6.noarch xorg-x11-fonts-misc-7.2-11.el6.noarch It seems that conda no fonts are available. Is there somebody with similar problems. Any hint which of the packages is/are actually needed. Maybe our admins could install them... I also found for MetaProSIP that several xorg-... packages are needed. Should such dependencies be handled by the conda packages or by the Galaxy tool. Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

3 4

Samtools running on submit-node and as a slurm job
by Previti 24 Apr '18

24 Apr '18

Dear all, We found something strange with our Galaxy instance... Our Galaxy server is configured as a submit node using slurm to submit to our cluster. This works fine for the most part, but today we saw a samtools index job running "locally" on the submit node instead of a cluster node. Is there any way of stopping this behavior? The submit nodes are not configured for data-intensive computation and everything runs significantly slower. I was running bamtools (splitting a bam-file into mapped into unmapped) at the time and the output bam-files were being indexed. Thanks and best regards, Christopher -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

2 1