May 2014 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Job execution order mixed-up
by Jean-Francois Payotte 04 Jul '14

04 Jul '14

Hi John, Thank you for your answer and for trying to help. This is greatly appreciated! I didn't really made any progress in tracking down this error, and hopefully this weird behaviour will not happen anymore with the November 4th, distribution. But here are my answers to your questions, in case it would ring a bell: Has this behaviour been reported with any other workflow? It has been reported with 2 different workflows as of now. These 2 workflows doesn't have anything in common, except that they are huge (one of them has 37 steps, producing a total of about 110 datasets). Are you running Galaxy as a single process or multiple processes? If multiple processes, how many web, handler and manager processes do you have and are they all on the same machine? We are running Galaxy in multiple processes with 5 web servers, 3 job handlers and no manager (I believe the manager was rendered obsolete in one of the latest Galaxy distributions). All these processes are run on the same machine. Have you made any modifications to Galaxy that could result in this behaviour? No. What is the value of track_jobs_in_database in your universe_wsgi.ini configuration file? We never touched this part of the configuration file and the line still reads: "#track_jobs_in_database = None". After reading your answer, I've decided to modify this line to: "track_jobs_in_database = True" Unfortunately, running one of the faulty workflows several times (5x), I noticed that one of them was still showing this strange behaviour where some jobs were executed before their inputs were ready. Do you think this issue could be related to the fact that we are using Galaxy with the multiple processes configuration? We implemented this configuration some time ago because some of our users were complaining about the slow responsiveness of the web interface. Would you recommend using Galaxy without the multiple processes configuration? (Lets say if updating to November 4th distribution doesn't fix this issue) I guess you are probably using the multiple processes configuration as well on Galaxy main? Thanks again for your help! Jean-François ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Posted by John Chilton on Nov 09, 2013; 2:50pm Hello Jean-François, Have you made any progress tracking down this error? This appears very serious, but to tell you the truth I have no clue what could cause it. The distribution you are using is pretty old at this point I feel like if it was a bug the exhibited under relatively standard parameter combinations someone else would have reported it by now. Can you tell me some things: has this been reported with any other workflows? Is there anything special about this workflow? Can you rebuild the workflow and see if the error occurs again? Additional questions if the problem is not restricted to the workflow: are you running Galaxy as a single process or multiple processes? If multiple processes, how many web, handler, and manager processes do you have? Are they all on the same machine? Have you made any modifications to Galaxy that could result in this behavior? What is the value of track_jobs_in_database in your universe_wsgi.ini configuration file? -John On Thu, Nov 7, 2013 at 10:34 AM, Jean-Francois Payotte <[hidden email]> wrote: Dear Galaxy mailing-list, Once again I come seeking for your help. I hope someone already had this issue or will have an idea on where to look to solve it. :) One of our users reported having workflows failing because some steps were executed before all their inputs where ready. You can find a screenshot attached, where we can see that step (42) "Sort on data 39" has been executed while step (39) is still waiting to run (gray box). This behaviour has been reproduced with at least two different Galaxy tools (one custom, and the sort tool which comes standard with Galaxy). This behaviour seems to be a little bit random, as running two times a workflow where this issue occurs, only one time did some steps were executed in the wrong order. I could be wrong, but I don't think this issue is grid-related as, from my understanding, Galaxy is not using SGE job dependencies functionality. I believe all jobs stays in some internal queues (within Galaxy) until all input files are ready, and only then the job is submitted to the cluster. Any help or any hint on what to look at to solve this issue would be greatly appreciated. We have updated our Galaxy instance to August 12th distribution on October 1st, and I believe we never experienced this issue before the update. Many thanks for your help, Jean-François

5 5

Sort error while running Cuffdiff
by TD 01 Jul '14

01 Jul '14

Hi All -- I'm a novice when it comes to all things programming, but I recently installed my own instance of Galaxy because I'm working with very large datasets. Everything seems to be running OK, but when I attempt to run Cuffdiff (w/ ref. annotation and bias correction from UCSC genome) on output from Tophat2, I get the following error: "*Error: sort order of reads in BAMs must be the same*." I'm perplexed, because I downloaded this output from Tophat2 that I ran on the public server, before putting it through Cuffdiff on my local instance. When I tried running the same files (prior to download) in Cuffdiff on the public server, they ran just fine. So, why do the files run properly when I run them publicly, but not locally? Am I missing some metadata? Is my install not complete? Does the ordering of bam files change when you download them? I've had a hard time finding solutions, so any tips are greatly appreciated. Cheers!

2 1

Announcement: Galaxy Docker builds
by Björn Grüning 20 Jun '14

20 Jun '14

Dear Galaxy Devs and Users, I'm pleased to announce the Galaxy Docker Image project! The Galaxy Docker Image is an easy distributable full-fledged Galaxy installation, that can be used for testing, teaching and presenting new tools and features. One of the main goals is to make the access to entire tool suites as easy as possible. Usually, this includes the setup of a public available webservice that needs to be maintained, or that the Tool-user needs to either setup a Galaxy Server by its own or to have Admin access to a local Galaxy server. With docker, tool developers can create their own Image with all dependencies and the user only needs to run it within docker. Ideally suited to make your reviewers happy :) Currently, I have three test images to play with. The first one, bgruening/galaxy-stable is the main Image that will do all the heavy lifting of setting up PostgreSQL, Apache and Galaxy. bgruening/deepTools and bgruening/chemicaltoolbox are proof of concepts how easy it is to create an own Image build on top of the main Galaxy Image. https://index.docker.io/u/bgruening/galaxy-deeptools/ https://index.docker.io/u/bgruening/galaxy-chemicaltoolbox/ How to use it: docker run -d -p 8080:80 bgruening/galaxy-deeptools docker run -d -p 8080:80 -p 8000:8000 bgruening/galaxy-chemicaltoolbox For a more detailed description please read: https://github.com/bgruening/docker-recipes/tree/master/galaxy I would be very much interested in any kind of feedback. Especially, in two technical questions: I'm not sure if a "trusted build" from the docker index is the best way to go. Trusted Builds in docker are super convenient. It is a post-commit hook from github. Every time something is pushed, the image will be rebuild. The problem is that I can't tag my images with "trusted builds" and unless I have missed something, that forces me to create new repositories every time a new Galaxy release is out. A better way would be, one repository with different tags: galaxy-stable, galaxy-feb-2014 and so on. Every Galaxy release will be just a diff between the old and the new. But this is not possible with "trusted builds". One other trick I used is to remove the mercurial history, that saves 180MB. I'm not sure if this is wise, because it probably kills a few use cases. Currently, the aim of this project is a little bit different from being a developer environment. So I decided to remove it. Please ping me if this was a bad decision :) Thanks for reading! Bjoern

2 2

Using Mesos to Enable distributed computing under Galaxy?
by Kyle Ellrott 17 Jun '14

17 Jun '14

I think one of the aspects where Galaxy is a bit soft is the ability to do distributed tasks. The current system of split/replicate/merge tasks based on file type is a bit limited and hard for tool developers to expand upon. Distributed computing is a non-trival thing to implement and I think it would be a better use of our time to use an already existing framework. And it would also mean one less API for tool writers to have to develop for. I was wondering if anybody has looked at Mesos ( http://mesos.apache.org/) You can see an overview of the Mesos architecture at https://github.com/apache/mesos/blob/master/docs/Mesos-Architecture.md The important thing about Mesos is that it provides an API for C/C++, Java/Scala and Python to write distributed frameworks. There are already implementations of frameworks for common parallel programming systems such as: - Hadoop (https://github.com/mesos/hadoop) - MPI ( https://github.com/apache/mesos/blob/master/docs/Running-torque-or-mpi-on-m… ) - Spark (http://spark-project.org) And you can find example Python framework at https://github.com/apache/mesos/tree/master/src/examples/python Integration with Galaxy would have three parts: 1) Add a system config variable to Galaxy called 'MESOS_URL' that is then passed to tool wrappers and allows them to contact the local mesos infrastructure (assuming the system has been configured) or pass a null if the system isn't available. 2) Write a tool runner that works as a mesos framework to executes single cpu jobs on the distributed system. 3) For instances where mesos is not available at a system wide level (say they only have access to an SGE based cluster), but the user wants to run distributed jobs, write a wrapper that can create a mesos cluster using the existing queueing system. For example, right now I run a Mesos system under the SGE queue system. I'm curious to see what other people think. Kyle

5 8

sorting of files in data libraries
by Sarah Diehl 11 Jun '14

11 Jun '14

Hi everyone, I uploaded a bunch of files to the Galaxy data libraries with the option "Upload files from filesystem paths" and "Link to files without copying to Galaxy". Afterwards I edited the names that are displayed in Galaxy. However, now the files are not alphabetically sorted anymore. I tried moving them around, but the order stayed the same. Is this on purpose and is there a way to change it? Thanks, Sarah

2 1

Join Paired-End reads (version 1.0.3)
by Julien Daligault 11 Jun '14

11 Jun '14

Hello, I am interested by a wrapper in the galaxy-dev.jgi-psf.org : Join Paired-End reads (version 1.0.3) (https://galaxy-dev.jgi-psf.org/tool_runner?tool_id=jgi_rnd_merge_pe) I didn't find it in the tool shed, where is it possible to access to this wrapper ? Regards, Julien -- Julien Daligault Plateforme Génomique-Transcriptomique (583B) INSTITUT JACQUES MONOD CNRS-Université Paris Diderot Bât. Buffon - 15 rue Hélène Brion 75205 Paris Cedex 13 - France tel : +33 1 57 27 81 31 daligault(a)ijm.univ-paris-diderot.fr

2 1

Docker based tools
by Kyle Ellrott 07 Jun '14

07 Jun '14

I know a few people have talked about using docker to package and distribute Galaxy tools. The most recent version now runs on all major linux distributions, so it's a bit more inclusive now (but there probably won't ever be a windows or mac version). One strategy would be for tool configs to define an option Docker archive name and possibly an alternate command line to execute if the docker archive is being used. How far has that work gotten, and what are people thinking about it? Kyle

3 2

Manual Tool and dependency installation
by Ravi Alla 05 Jun '14

05 Jun '14

Hi All, Due to the nature of our cluster we are not allowed to have compilers on our webserver, which is where our galaxy instance runs. Because of this tool dependencies and tools that need to be compiled fail to install through the galaxy tool shed webpage. Is there a way for me to install dependencies and tools manually? For example I am trying to install the GATK2 wrapper. This depends on samtools-0.1.19 from iuc which in turn requires ncurses-5.9. Is there instructions somewhere as to where to install these tools such that they are tied to the GATK2 wrapper? Pardon me if this is a naive question, but I am really out of sorts here. Thank you Ravi

3 2

Galaxy customized tool config file
by Vipin TS 04 Jun '14

04 Jun '14

Hello dev-team, With the current state, is it possible to write an interactive page for a tool in Galaxy? In my case, I m considering to build a tool page with JavaScript embed in XML file, my expectation is to behave the Galaxy tool page as google translate page with option instant translation on (display the same text content in the next box). I have checked the Admin/Tools/ToolConfigSyntax, not sure whether I missed anything related this. any comments ? thanks, Vipin galaxy.cbio.mskcc.org

2 3

Uploading files to galaxy from a folder
by Kandalaft, Iyad 02 Jun '14

02 Jun '14

Hi Everyone I'm throwing this out there for some feedback and recommendations. Objective: Facilitate transferring large files (> 2GB) from an HPC cluster (and its associated fast tier storage) to galaxy for my clients. I enabled the FTP upload option in galaxy but it involves users learning to copy files over FTP. So, I created a galaxy folder in each users' home directory on the HPC Cluster that symbolically links to the FTP upload folder for galaxy. Hence, users can use either FTP to upload files (drag and drop in windows) or simply copy files into this folder from an ssh session on the cluster. The problem with that strategy was that galaxy had to be the owner of the file (similar to the ProFTPd configuration that sets the UID and GID of uploads files to galaxy's UID/GID). Otherwise, galaxy threw errors when it tried deleting the original file from the FTP upload folder. I could have added the galaxy user to the same group as all user but this meant that users would have to ensure the correct permissions are set on files so that galaxy can read and delete the file thereafter. The alternative involved modifying the upload.py tool to chown/chmod files that were being uploaded. Upload.py now sudo executes an external script that sets ownership to the galaxy user and corrects the permissions if required (see attachment for code modification). The galaxy user has sudo rights on this script and the script restricts chown/chmod to the ftp folder path for security reasons. I was planning to clean up the code and make it production ready by adding an option in universe_wsgi.ini for this "feature", but I thought I would check with the galaxy devs first. Am I taking the wrong approach? Is there a better alternative? As an alternative, I thought about locating the handler code for dataset.type == file and possibly making it support the SETGID sticky bits on folders. In that case, the FTP upload folder would have the sticky bit set for UID and can assume the role of the user to upload that file. Your input is much appreciated. Iyad Kandalaft Bioinformatics Application Developer Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada KW Neatby Bldg | éd. KW Neatby 960 Carling Ave| 960, avenue Carling Ottawa, ON | Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel: Iyad.Kandalaft(a)agr.gc.ca<mailto:Iyad.Kandalaft@agr.gc.ca> Telephone | Téléphone 613- 759-1228 Facsimile | Télécopieur 613-759-1701 Government of Canada | Gouvernement du Canada

3 5