May 2018 - galaxy-dev - lists.galaxyproject.org

Monitoring Dashboard for Galaxy
by evan clark 03 Sep '18

03 Sep '18

I remember there previously being an additional tool that ran on a different port from galaxy that allowed for monitoring of performance and jobs running. Is this tool still packaged with galaxy and if so how can it be activated.

3 3

Job Script Integrity
by John Letaw 04 Jun '18

04 Jun '18

Hi all, I’m installing via GalaxyKickStart… I’m getting the following error: galaxy.jobs.runners ERROR 2017-11-06 19:14:05,263 (19) Failure preparing job Traceback (most recent call last): File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py", line 175, in prepare_job modify_command_for_container=modify_command_for_container File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py", line 209, in build_command_line container=container File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py", line 84, in build_command externalized_commands = __externalize_commands(job_wrapper, external_command_shell, commands_builder, remote_command_params) File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py", line 143, in __externalize_commands write_script(local_container_script, script_contents, config) File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/util/job_script/__init__.py", line 112, in write_script _handle_script_integrity(path, config) File "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/util/job_script/__init__.py", line 147, in _handle_script_integrity raise Exception("Failed to write job script, could not verify job script integrity.") Exception: Failed to write job script, could not verify job script integrity. galaxy.model.metadata DEBUG 2017-11-06 19:14:05,541 Cleaning up external metadata files galaxy.model.metadata DEBUG 2017-11-06 19:14:05,576 Failed to cleanup MetadataTempFile temp files from /home/exacloud/lustre1/galaxydev/galaxyuser/database/jobs/000/19/metadata_out_HistoryDatasetAssociation_16_I8bhLX: No JSON object could be decoded I would like to further understand what it means to not verify integrity of a job script. Does this just mean there is a permissions error? Ownership doesn’t match up? Thanks, John

2 1

argparse -> tool xml advice
by Matthias Bernt 28 May '18

28 May '18

Dear list, just a question for advice: I plan to wrap a python script (with lots of options) which uses the argparse library. I was wondering if somebody has an idea to generate the tool xml (semi-)automatically? For instance, by subclassing the formatter, parsing the python code, or parsing the command line... Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

3 8

Conda problem
by David Lee 25 May '18

25 May '18

Hi, I hope somebody can help me with this. I've been maintaining a Galaxy installation for the past few months that has suddenly developed a problem that I can't undo. I think it occurred after installing samtools and sambamba packages which I have subsequently uninstalled but which has made no difference. Every tool I try to run fails with this error: "Conda dependency failed to build job environment. This is most likely a limitation in conda. You can try to shorten the path to the job_working_directory." Also, in the paster.log file I'm seeing this error that I've never seen before: "CondaError: RuntimeError('EnforceUnusedAdapter called with url https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2018.4.16-0… command is using a remote connection in offline mode.\n',)" Any suggestions? Regards, David

2 3

The Galaxy release 18.05 is out
by Martin Čech 23 May '18

23 May '18

Dear Community, We are happy to release Galaxy 18.05. Many thanks to committers and you, the community. Enjoy the on time release and please let us know if you face any issues. The release notes are available at https://docs.galaxyproject.org/en/release_18.05/releases/18.05_announce.html Thanks for using Galaxy! Martin, on behalf of the Galaxy committers

1 0

Workflow renaming problem for tool with multiple outputs (Galaxy version 17.09)
by Previti 23 May '18

23 May '18

Hi everybody, I'd like to fix this bug in our local galaxy instance...it's with regard to the erroneous renaming of output files, when there are multiple outputs available. The relevant ticket is the following: https://github.com/galaxyproject/galaxy/pull/5021 Unfortunately a quick update to the newest Galaxy version is not possible for us at the moment. Any tips on how I could do this? Thanks and best regards, Christopher Previti -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

1 0

Purging "Monitor installing repositories"
by Previti 23 May '18

23 May '18

Dear all, I've been having an issue with the "Monitor installing repositories" page. There is a whole list of tools that were not installed properly (usually they got interrupted for some reason) and they aren't being removed. Is there a way to purge this list manually? Thanks and best regards, Christopher Previti <http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/monitor_repository…> -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

1 0

Text file busy and conda
by John Letaw 16 May '18

16 May '18

Hi all, I’m receiving the following error trying to access tools autoinstalled via conda: Fatal error: Exit code 126 () ~/galaxy/database/jobs_directory/000/38/tool_script.sh: line 41: ~/galaxy/database/dependencies/_conda/envs/mulled-v1-bb83f93efd111e042823e53ddfa74c32d81ba74cceca9445dfddfc9e940ff738/bin/samtools: Text file busy So, something is happening too fast I guess, meaning a process is attempting access while this is being created. Any ideas on how I can diagnose this? Not seeing any other errors floating around the logs. Maybe we are dealing with a lustre file locking issue? Thanks, John

2 2

Trackster ownership issue
by Previti 16 May '18

16 May '18

Dear all, I'm having a strange issue with Trackster in our Galaxy implementation. Users seem to be able to upload files ("Add Datasets") from*all *histories of all users, regardless of ownership. ...is this something that has been observed before...? How do I stop this from happening? Thanks and best regards, Christopher -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

2 5

V18.01 'View all histories' doesn't work
by Ping Luo 15 May '18

15 May '18

I just installed Galaxy V18.01. I got an error when clicking 'View all histories' in the history tool bar The error says 'The requested URL /api/histories was not found on this server'. Is there anyone experiencing the same problem? Ping

3 5

Re: [galaxy-dev] Tool to download galaxy datasets locally
by Nicola Soranzo 09 May '18

09 May '18

Dear Evan, you need to setup an environment variable for the tool to specify the directory where users are expected to save their files. One way to do that is: 1) create a directory called EXPORT_DIR_PREFIX in your tool dependencies directory (which by default is `database/dependencies/` ) 2) in this directory create a file called env.sh with a content similar to the following template: EXPORT_DIR_PREFIX=/PATH/WHERE/TO/SAVE/FILES; export EXPORT_DIR_PREFIX 3) create a symlink called `default` inside the EXPORT_DIR_PREFIX directory pointing to the same directory, e.g.: $ cd TOOL_DEPENDENCIES_DIR/EXPORT_DIR_PREFIX/ $ ln -s . default Alternatively, you could define the EXPORT_DIR_PREFIX environment variable in your config/job_conf.xml for the destination where the export_to_cluster tool is going to run (for an example, see `_JAVA_OPTIONS` in config/job_conf.xml.sample_advanced ), but this would define the environment variable also for all the other tools that use that destination (normally not a problem, but worth to mention). Cheers, Nicola On 05/05/18 15:04, Evan Clark wrote: > Have you ever seen this error with export to cluster? > > export_to_cluster.py: error: Directory prefix cannot be empty > > Nicola Soranzo wrote: >> >> Hi Evan, >> you can have a look at >> https://toolshed.g2.bx.psu.edu/view/earlhaminst/export_to_cluster/ >> <https://toolshed.g2.bx.psu.edu/view/earlhaminst/export_to_cluster/b97775e9f…> >> >> Hope that helps! >> >> Cheers, >> Nicola >> >> On 27/03/18 16:03, evan clark wrote: >>> >>> Has anyone attempted to write a tool that allows copying of galaxy >>> datasets locally, i.e. without downloading via http. We have scenario >>> where there are some additional tools available only via command line >>> and it would be very useful to be able to directly copy galaxy >>> datasets from the datasets folder. >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> https://lists.galaxyproject.org/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/ >>

2 1

ELIXIR Workshop for Galaxy training material and skills improvement
by Nicola Soranzo 08 May '18

08 May '18

Dear all, we'd like to invite you to the "ELIXIR Workshop for Galaxy training material and skills improvement", which will be held 21-23 May at the Earlham Institute in Norwich, UK: http://www.earlham.ac.uk/elixir-workshop-galaxy-training-material-and-skill… The Galaxy Training Network (GTN) has been working for several years with GOBLET and the ELIXIR Training Platform to create material and deliver training for scientists, developers and system administrators. A selection of 72 tutorials, developed by more than 60 contributors, is already available at http://training.galaxyproject.org/ In this workshop, we intend to improve the participants’ understanding of learning principles, training techniques and best practices for materials preparation. We will then work together to expand the existing collection of Galaxy training materials by covering more topics relevant to the ELIXIR use cases, add missing annotation and metadata and make the materials easily accessible. The workshop will start with a 1-day "Train the Trainer" course, similar to the Carpentries' Instructor Training and the ELIXIR-EXCELERATE Train the Trainer. Here Galaxy trainers will develop an insight into different learning styles, understand what makes a good trainer, and learn new approaches to training from experienced trainers. This course will help them to better contribute and shape the materials during the hackathon. This workshop is kindly sponsored by ELIXIR-UK and the ELIXIR Training Platform. Registration closes soon, hurry up! https://www.eiseverywhere.com/galaxy We look forward to seeing many of you in Norwich! Nicola -- Nicola Soranzo, Ph.D. Data Infrastructure & Algorithms group Earlham Institute Norwich Research Park, Norwich, NR4 7UZ, UK http://www.earlham.ac.uk/nicola-soranzo

1 0

GCCBOSC 2018 Early Registration Ends this Friday (May 11)
by Dave Clements 07 May '18

07 May '18

Hi All, Just a reminder that early registration for the Galaxy Community's biggest annual gathering ends this Friday. You can also register for housing and childcare too. https://galaxyproject.org/events/gccbosc2018/register/ https://galaxyproject.org/events/gccbosc2018/venue/#housing https://galaxyproject.org/events/gccbosc2018/childcare/ Hope to see you there, Dave C on behalf of the GCCBOSC 2018 Organizing Committee -- https://gccbosc2018.sched.com/ https://galaxyproject.org/ https://usegalaxy.org/

1 0

Best practices for utilising Unique Molecular Indexing (UMI's)
by Brooks, Tony 04 May '18

04 May '18

Hi We are currently seeing a number of methods that are utilising the power of unique molecular indexing. Unfortunately, there is no consensus on how libraries should be configured, and therefore no consensus for how to deal with them within Galaxy. Often libraries that have the UMI placed directly downstream of the first (i7) index, such as ones using the IDT xGen adapter set (https://www.idtdna.com/pages/products/next-generation-sequencing/adapters/x…) Sometimes UMI's exist in place of the second (i5) index (https://www.neb.com/nebnext-direct/nebnext-direct-for-target-enrichment) In both cases, the recommended workflows are convoluted and all the necessary tools do not currently exist in the toolshed (so that the datasets need to be taken out of galaxy, processed and reloaded). It is possible to use bcl2fastq to output the UMI as an additional fastq file, but this would then require me to create a dataset triplicate (not pair) which afaik we can't do (yet). A quick Google/toolshed search had me find UMI-Tools & Je-Suite which both exist in Galaxy. Both these tools assume the UMI is "in-line" (i.e. at the beginning of the read 1 or read2 - not its own read), extract/remove the UMI and place it in the read header, where it is then used further down the line to dedup the bam file. Does anyone know of any tools that would take the UMI from a separate fastq and use it to tag the headers of actual read data. Or alternatively, a tool that will paste the UMI tag onto the 5' end of the read fastq? And whether these steps can be done within Galaxy, or maybe prior to fastq upload? Anyone have a method/workflow for UMI's? Thanks in advance Tony

2 6

May Galaxy News: GCCBOSC, Training Material Workshop, New pubs, jobs, ...
by Dave Clements 02 May '18

02 May '18

Hello all, The May 2018 Galactic News <https://galaxyproject.org/galaxy-updates/2018-05/> highlights what's going on in the Galaxy this month, including: - Lots of GCCBOSC 2018 news <https://galaxyproject.org/galaxy-updates/2018-04/#gccbosc-2018>: - *Early registration ends May 11 <https://galaxyproject.org/galaxy-updates/2018-05/#early-registration-ends-m…>* - Galaxy talks have been posted to the schedule <https://galaxyproject.org/galaxy-updates/2018-05/#gcc2018-schedule-posted> . - BOSC deadline for late-breaking lightning talks, demos and posters *is May 17 <https://galaxyproject.org/galaxy-updates/2018-05/#bosc-deadline-for-late-br…>* - Please welcome *Google Cloud <https://galaxyproject.org/galaxy-updates/2018-05/#gigascience-and-oxford-un…>* and *Advanced HPC <https://galaxyproject.org/galaxy-updates/2018-05/#advanced-hpc>* as GCCBOSC sponsors. *Both are offering goodies for conference participants.* - Other event news: - ELIXIR Workshop for Galaxy training material and skills improvement <https://galaxyproject.org/galaxy-updates/2018-05/#elixir-workshop-for-galax…> is 21-23 May. Register right now. - Galaxy Africa Report <https://galaxyproject.org/galaxy-updates/2018-05/#galaxy-africa-report> - Other Upcoming events <https://galaxyproject.org/galaxy-updates/2018-05/#upcoming-events> (on three continents in 6 countries) - New: - 132 publications <https://galaxyproject.org/galaxy-updates/2018-05/#publications> - Openings in 10 groups on 2 continents <https://galaxyproject.org/galaxy-updates/2018-05/#whos-hiring> - Public Galaxy Server News <https://galaxyproject.org/galaxy-updates/2018-05/#public-galaxy-server-news> : IRProfiler <https://galaxyproject.org/galaxy-updates/2018-05/#irprofiler> - ToolShed contributions <https://galaxyproject.org/galaxy-updates/2018-05/#toolshed-contributions> - Releases <https://galaxyproject.org/galaxy-updates/2018-05/#releases> of BioBlend and Starforge See the newsletter <https://galaxyproject.org/galaxy-updates/2018-05/> for the full story, and we hope to see you in Portland! Dave C on behalf of, well, everyone -- https://gccbosc2018.sched.com/ https://galaxyproject.org/ https://usegalaxy.org/

1 0

extract extra info (logs) from history
by Jochen Bick 02 May '18

02 May '18

Hi, I would like to use the info (log) HISAT is showing, for another tool. e.g.: 26781724 reads; of these: 26781724 (100.00%) were unpaired; of these: 2445411 (9.13%) aligned 0 times 20399099 (76.17%) aligned exactly 1 time 3937214 (14.70%) aligned >1 times 90.87% overall alignment rate [bam_sort_core] merging from 21 f Is it possible to access these logs? Cheers Jochen

4 7