Monitoring Dashboard for Galaxy
by evan clark
I remember there previously being an additional tool that ran on a
different port from galaxy that allowed for monitoring of performance
and jobs running. Is this tool still packaged with galaxy and if so how
can it be activated.
2 years
Job Script Integrity
by John Letaw
Hi all,
I’m installing via GalaxyKickStart…
I’m getting the following error:
galaxy.jobs.runners ERROR 2017-11-06 19:14:05,263 (19) Failure preparing job
Traceback (most recent call last):
File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py", line 175, in prepare_job
modify_command_for_container=modify_command_for_container
File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py", line 209, in build_command_line
container=container
File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py", line 84, in build_command
externalized_commands = __externalize_commands(job_wrapper, external_command_shell, commands_builder, remote_command_params)
File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py", line 143, in __externalize_commands
write_script(local_container_script, script_contents, config)
File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/util/job_script/__init__.py", line 112, in write_script
_handle_script_integrity(path, config)
File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/util/job_script/__init__.py", line 147, in _handle_script_integrity
raise Exception("Failed to write job script, could not verify job script integrity.")
Exception: Failed to write job script, could not verify job script integrity.
galaxy.model.metadata DEBUG 2017-11-06 19:14:05,541 Cleaning up external metadata files
galaxy.model.metadata DEBUG 2017-11-06 19:14:05,576 Failed to cleanup MetadataTempFile temp files from /home/exacloud/lustre1/galaxydev/galaxyuser/database/jobs/000/19/metadata_out_HistoryDatasetAssociation_16_I8bhLX: No JSON object could be decoded
I would like to further understand what it means to not verify integrity of a job script. Does this just mean there is a permissions error? Ownership doesn’t match up?
Thanks,
John
2 years, 3 months
Text file busy and conda
by John Letaw
Hi all,
I’m receiving the following error trying to access tools autoinstalled via conda:
Fatal error: Exit code 126 ()
~/galaxy/database/jobs_directory/000/38/tool_script.sh: line 41: ~/galaxy/database/dependencies/_conda/envs/mulled-v1-bb83f93efd111e042823e53ddfa74c32d81ba74cceca9445dfddfc9e940ff738/bin/samtools: Text file busy
So, something is happening too fast I guess, meaning a process is attempting access while this is being created. Any ideas on how I can diagnose this? Not seeing any other errors floating around the logs. Maybe we are dealing with a lustre file locking issue?
Thanks,
John
2 years, 4 months
Best practices for utilising Unique Molecular Indexing (UMI's)
by Brooks, Tony
Hi
We are currently seeing a number of methods that are utilising the power of unique molecular indexing. Unfortunately, there is no consensus on how libraries should be configured, and therefore no consensus for how to deal with them within Galaxy.
Often libraries that have the UMI placed directly downstream of the first (i7) index, such as ones using the IDT xGen adapter set
(https://www.idtdna.com/pages/products/next-generation-sequencing/adapters...). Sometimes UMI's exist in place of the second (i5) index (https://www.neb.com/nebnext-direct/nebnext-direct-for-target-enrichment).
In both cases, the recommended workflows are convoluted and all the necessary tools do not currently exist in the toolshed (so that the datasets need to be taken out of galaxy, processed and reloaded).
It is possible to use bcl2fastq to output the UMI as an additional fastq file, but this would then require me to create a dataset triplicate (not pair) which afaik we can't do (yet).
A quick Google/toolshed search had me find UMI-Tools & Je-Suite which both exist in Galaxy.
Both these tools assume the UMI is "in-line" (i.e. at the beginning of the read 1 or read2 - not its own read), extract/remove the UMI and place it in the read header, where it is then used further down the line to dedup the bam file.
Does anyone know of any tools that would take the UMI from a separate fastq and use it to tag the headers of actual read data. Or alternatively, a tool that will paste the UMI tag onto the 5' end of the read fastq? And whether these steps can be done within Galaxy, or maybe prior to fastq upload?
Anyone have a method/workflow for UMI's?
Thanks in advance
Tony
2 years, 4 months
extract extra info (logs) from history
by Jochen Bick
Hi,
I would like to use the info (log) HISAT is showing, for another tool.
e.g.:
26781724 reads; of these:
26781724 (100.00%) were unpaired; of these:
2445411 (9.13%) aligned 0 times
20399099 (76.17%) aligned exactly 1 time
3937214 (14.70%) aligned >1 times
90.87% overall alignment rate
[bam_sort_core] merging from 21 f
Is it possible to access these logs?
Cheers Jochen
2 years, 5 months
Gateways 2018 Call for Participation: Extended to May 16
by Katherine Lawrence
The deadline for the Gateways 2018 Call for Participation has been
extended to Wednesday, May 16, 2018.
Gateways 2018 (September 25-27, at the University of Texas at Austin) is
accepting submissions of papers, demos, tutorials, and panels (2-4
pages) on the topic of science or engineering gateways.
Topics may include their design, use, impact, development processes,
sustainability, best practices, or any other aspect that you think
fellow gateway creators or users will find interesting to learn. We also
welcome educational topics directed toward the next generation of
gateway creators.
Submissions of papers, demos, tutorials, and panels are due May 16,
2018. There will be no further extension of this deadline. The poster
session deadline (open to all) is August 1, 2018.
Full submission details can be found here:
https://sciencegateways.org/gateways2018/call
2 years, 5 months
tool_conf.xml
by Nancy Ide
Hello
I’m trying to include some of the tools available from usegalaxy.org in the tool menu of our instance of Galaxy. When I look at the list of tools in the Galaxy tool_conf.xml.main file, it does not seem to line up with what you see in the menu at usegalaxy.org. I’ve searched through various versions etc. on GitHub, but no luck—they all seem to be identical and do not seem to correspond with the menu at usegalaxy.org (I am looking specifically at the Text Manipulation part of the menu). Any clues on where I should look?
Thanks,
Nancy Ide
-----------------------------------------------------
Nancy Ide
Professor of Computer Science
Department of Computer Science
Vassar College
Poughkeepsie, New York 12604-0520
USA
tel: (+1 845) 437 5988
fax: (+1 845) 437 7498
email: ide(a)cs.vassar.edu
http://www.cs.vassar.edu/~ide
-----------------------------------------------------
2 years, 5 months
Galaxy Cleanup
by Previti
Hi everybody,
I have a hopefully basic questions regarding temporary output files.
A user was generating >4TB output files via the program bedtools
intersectbed. The jobs were cancelled, but I still see the (temporary)
output files that were generated.
Does Galaxy clean these up? Does it do this regularly?
Would a quota prevent something like this from happening again or would
the jobs have to finish before being "counted"?
Thanks and best regards,
Christopher
--
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician
German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661
christopher.previti(a)dkfz.de <http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de/>
Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537
Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.
2 years, 5 months
fonts on headless compute nodes
by Matthias Bernt
Dear list,
I found that some tools (eg FastQC and MetaProSIP) have problems with
missing fonts (the former is java and the latter caused by R) on our
compute nodes which do not have xorg and no fonts installed.
On the cluster head nodes several font related packages are installed:
dejavu-fonts-common-2.33-1.el6.noarch
xorg-x11-fonts-Type1-7.2-11.el6.noarch
liberation-fonts-common-1.05.1.20090721-5.el6.noarch
liberation-sans-fonts-1.05.1.20090721-5.el6.noarch
dejavu-sans-fonts-2.33-1.el6.noarch
urw-fonts-2.4-11.el6.noarch
ghostscript-fonts-5.50-23.2.el6.noarch
xorg-x11-fonts-misc-7.2-11.el6.noarch
It seems that conda no fonts are available. Is there somebody with
similar problems. Any hint which of the packages is/are actually needed.
Maybe our admins could install them...
I also found for MetaProSIP that several xorg-... packages are needed.
Should such dependencies be handled by the conda packages or by the
Galaxy tool.
Best,
Matthias
--
-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
m.bernt(a)ufz.de, www.ufz.de
Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
2 years, 5 months
Samtools running on submit-node and as a slurm job
by Previti
Dear all,
We found something strange with our Galaxy instance...
Our Galaxy server is configured as a submit node using slurm to submit
to our cluster.
This works fine for the most part, but today we saw a samtools index job
running "locally" on the submit node instead of a cluster node.
Is there any way of stopping this behavior? The submit nodes are not
configured for data-intensive computation and everything runs
significantly slower.
I was running bamtools (splitting a bam-file into mapped into unmapped)
at the time and the output bam-files were being indexed.
Thanks and best regards,
Christopher
--
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician
German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661
christopher.previti(a)dkfz.de <http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de/>
Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537
Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.
2 years, 5 months
