November 2013 - galaxy-dev - lists.galaxyproject.org

SLURM and hidden success
by Andrew Warren 11 Nov '13

11 Nov '13

Hello all, We are in the process of switching from SGE to SLURM for our galaxy setup. We are currently experiencing a problem where jobs that are completely successful (no text in their stderr file and the proper exit code) are being hidden after the job completes. Any job that fails or has some text in the stderr file is not hidden (note: hidden not deleted; they can be viewed by selecting 'Unhide Hidden Datasets'). Our drmaa.py is at changeset 10961:432999eabbaa Our drmaa egg is at drmaa = 0.6 And our SLURM version is 2.3.5 And we are currently passing no parameters for default_cluster_job_runner = drmaa:/// We have the same code base on both clusters but only observe this behavior when using SLURM. Any pointers or advice would be greatly appreciated. Thanks, Andrew

2 2

DRMAA/SGE job handling regression?
by Peter Cock 11 Nov '13

11 Nov '13

Hello all, On our main Galaxy tracking galaxy-dist using DRMAA/SGE, jobs submitted to the cluster and queued and waiting (qw) are correctly shown in Galaxy as "grey" pending entries in the history. With my test instance tracking galaxy-central (along with a new visual look and new icons), such jobs are wrongly shown as "yellow" (running). Is this a general regression affecting other people? There also seem to be issues where killing a job in Galaxy just hides it but it remains running (yellow once you tick show deleted datasets, and running on SGE too). This was working properly on galaxy-dist (the job was killed on the cluster, and shown as red if you ticked show deleted datasets). Thanks, Peter

1 0

Using input dataset names in output dataset names
by Peter Cock 11 Nov '13

11 Nov '13

Hi all, I'd like to change the output dataset labelling in Galaxy file format conversion tools. e.g. If the input is history entry 1 (e.g "My Genes") then the output from tabular_to_fasta.xml is currently named "FASTA-to-Tabular on data 1". I would prefer this was "FASTA-to-Tabular on data My Genes" or better "My Genes (as tabular)". I've just done this for my BLAST XML to tabular tool, using the .display_name trick: https://github.com/peterjc/galaxy_blast/commit/31e31c4b5deadd60828ce6e6a381… Would a pull request doing this to the built-in conversion tools be favourably received? Alternatively, would it be preferable to simply reused the input dataset's name unchanged for simple format conversion tools (without text about the conversion)? Related to this, would people prefer if the $on_string in the case of a single input file was the input file's name (e.g. "My Genes") rather than "data 1"? (When there are multiple input files, $on_string needs to be kept short). Regards, Peter

4 8

set_environment_for_install problem, seeking for ideas
by Bjoern Gruening 09 Nov '13

09 Nov '13

Hi Greg, I'm right now in implementing a setup_perl_environment and stumbled about a tricky problem (that is not only related to perl but also for ruby, python and R). The Problem: Lets assume a perl package (A) requires a xml parser written in C/C++ (Z). (Z) is a dependency that I can import but as far as I can see there is no way to call set_environment_for_install before setup_perl_environment, because setup_perl_environment defines an installation type. I would like to discuss that issue to get a few ideas. I can think about these solutions: - hackish solution: I can call install_environment.add_env_shell_file_paths( action_dict[ 'env_shell_file_paths' ] ) inside of the setup_*_environment path and remove it from action type afterwards - import all env.sh variables from every (package) definition. Regardless if set_environment_for_install is set or not. I must admit, I do not understand why set_environment_for_install is actually needed. I think we can assume that if I specify a <package name="R_3_0_1" version="3.0.1"> <repository name="package_r_3_0_1" owner="iuc" prior_installation_required="True" /> </package> I want the ENV vars sourced. Furthermore, that can solve an other issue: Namely, the need of ENV vars from a package definition in the same file. Lets imagine package P has dependency D and you want to download compile both in one tool_dependencies.xml file. You can either do it in one <package> definition or you need to split them up in 2 tool_dependencies.xml files, rigth? Maybe we can just assume a strict order in a tool_dependencies.xml file, where every ENV vars are sourced for the following one? Does that make sense? Any other ideas? Thanks, Bjoern

5 13

Job execution order mixed-up
by Jean-Francois Payotte 09 Nov '13

09 Nov '13

Dear Galaxy mailing-list, Once again I come seeking for your help. I hope someone already had this issue or will have an idea on where to look to solve it. :) One of our users reported having workflows failing because some steps were executed before all their inputs where ready. You can find a screenshot attached, where we can see that step (42) "Sort on data 39" has been executed while step (39) is still waiting to run (gray box). This behaviour has been reproduced with at least two different Galaxy tools (one custom, and the sort tool which comes standard with Galaxy). This behaviour seems to be a little bit random, as running two times a workflow where this issue occurs, only one time did some steps were executed in the wrong order. I could be wrong, but I don't think this issue is grid-related as, from my understanding, Galaxy is not using SGE job dependencies functionality. I believe all jobs stays in some internal queues (within Galaxy) until all input files are ready, and only then the job is submitted to the cluster. Any help or any hint on what to look at to solve this issue would be greatly appreciated. We have updated our Galaxy instance to August 12th distribution on October 1st, and I believe we never experienced this issue before the update. Many thanks for your help, Jean-François

2 1

multiple output tool in workflow
by Jennifer Jackson 09 Nov '13

09 Nov '13

Hi Jun, There is probably a problem with the tool design itself, but that may be what you are asking how to solve. I wouldn't think this is a problem with workflows at first pass. Is this your own tool? Or a tool from the tool shed (the repo developer is usually the one to make changes, unless you want to try)? This is the primary tool development wiki, the " <output> tag set" is where I would double check the tool first. http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax I am moving this over to the galaxy-dev(a)bx.psu.edu mailing list since it is a tool development question. Jen Galaxy team On 11/8/13 10:19 AM, Jun Fan wrote: > > Hi all, > > I am trying to creating a workflow from history. One of the tool > used generates multiple outputs in the format of gff3, fasta and sam. > Gff3 will be visualized in IGV and the fasta file is doing further > BLAST analysis. Now the problem is that the automatically generated > workflow does not connect the having-multiple-output tool and the > BLAST tool. I failed even I tried to connect these two tools in the > workflow by hand. I am guessing this is due to only the main output > type (gff3) is recognized in the workflow. How could I solve this problem? > > Best regards and have a nice weekend! > > Jun > > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

3 3

Trouble getting proftpd to use Galaxy postgresql authentication
by Hazard, E. Starr 08 Nov '13

08 Nov '13

Folks, I am new to using proftpd/Galaxy on a local cluster I am having lots of issues. Here is one. I cannot connect to the proftpd server to upload files. ./proftpd --version ProFTPD Version 1.3.4d ( so proftpd is compiled and running) I am attempting to connect from an iMAC (OSX 10.9) to a LINUX machine where the proftp server is running ftp 123.45.678.123 Connected to 123.45.678.123. 220 ProFTPD 1.3.4d Server (Public Galaxy FTP by ProFTPD server installation) [128.23.191.200] Name (123.45.678.123:hazards): hazards 331 Password required for hazards I enter a password and I get “Abort trap 6” or “421 service not available, remote server closed connection” ( this latter reply from an older mac) I presume that this means that the authentication failed. Here are the last lines of my proftpd.conf file # Do not authenticate against real (system) users AuthPAM off # Set up mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1 SQLPasswordEngine on SQLPasswordEncoding hex # Set up mod_sql to authenticate against the Galaxy database SQLEngine on SQLBackend postgres SQLConnectInfo galaxydb(a)xxx.musc.edu dbuser dbpassword SQLAuthTypes SHA1 SQLAuthenticate users # An empty directory in case chroot fails SQLDefaultHomedir /shared/app/ProFFTPd-1.3.4d/default # Define a custom query for lookup that returns a passwd-like entry. UID and GID should match your Galaxy user. SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email,password,'581','582','/shared/app/Galaxy/galaxy_dist/database/files/%U','/bin/bash' FROM galaxy_user WHERE email='%U’" I have a user named galaxy on the system and a Galaxy-user named galaxy defined within Galaxy and an owner of the postgresql database. With respect to the following line, If my Galaxy postgresql database is called “galaxydb”, are the dbuser and dbpassword to be the postgresql owner of the database? Or the system user named galaxy who has actually started the instance of galaxy with run.sh command. SQLConnectInfo galaxydb(a)xxx.musc.edu galaxy Galaxy2013 With respect to this line SQLNamedQuery LookupGalaxyUser SELECT "email,password,'581','582’,…… What are the appropriate UID and GID to apply here? I have a system user “galaxy” who starts Galaxy with “run.sh”. This user’s UID and GID are 581 and 582 respectively but I am not able to login for FTP transfer If I use Chrome and paste " ftp://123.45.678.123” I get “unable to load webpage because server sent no data “error code ERR_EMPTY_RESPONSE" If I paste "ftp://123.45.678.123/ /path/to/file" And press execute in my local Galaxy GUI I get a "tool error” with no Std_Out and no Std_error So I cannot use FTP via ProFTPD 1.3.4d Server to upload files. How can I test postgresql authentication via Galaxy/proftpd to get at the problem? Starr

1 0

Managing Data Locality
by Paniagua, Eric 08 Nov '13

08 Nov '13

Dear Galaxy Developers, I administer a Galaxy instance at Cold Spring Harbor Laboratory, which servers around 200 laboratory members. While our initial hardware purchase has scaled well for the last 3 years, we are finding that we can't quite keep up with rising the demand for compute-intensive jobs, such as mapping. We are hesitant to consider buying more hardware to support the load, since we can't expect that solution to scale. Rather, we are attempting to set up Galaxy to queue jobs (especially mappers) out to the lab's HPCC to accommodate the increasing load. While there is a good number of technical challenges involved in this strategy, I am only writing to ask about one: data locality. Normally, all Galaxy datasets are stored directly on the private server hosting our Galaxy instance. The HPCC cannot mount our Galaxy server's storage (ie: for the purpose of running jobs reading/writing datasets) for security reasons. However, we can mount a small portion of the HPCC file system to our Galaxy server. Storage on the HPCC is at a premium, so we can't afford to just let newly created (or copied) datasets just sit there. It follows that we need a mechanism for maintaining temporary storage in the (restricted) HPCC space which allows for transfer of input datasets to the HPCC (so they will be visible to jobs running there) and transfer of output datasets back to persistent storage on our server. I am in the process of analyzing when/where/how exact path names are substituted into tool command lines, looking for potential hooks to facilitate the staging/unstaging of data before/after job execution on the HPCC. I have found a few places where I might try to insert logic for handling this case. Before modifying too much of Galaxy's core code, I would like to know if there is a recommended method for handling this situation and whether other members of the Galaxy community have implemented fixes or workarounds for this or similar data locality issues. If you can offer either type of information, I shall be most grateful. Of course, if the answer were that there were no recommended or known technique, then that would be valuable information too. Thank you in advance, Eric Paniagua

2 3

Trying to map fastq reads with bowtie for illumina to the reference genome sacCer3
by Frank Tedeschi 08 Nov '13

08 Nov '13

Hello! I have SacCer uploaded to galaxy in the fasta format however to use it as a mapping genome with bowtie for illumina it says it must be in fastq format. Do you know a way that I can convert this file or what I can do to make this work? Thank you very much for your time! Sincerely, Frank

1 0

tool of installing tool shed repositories
by ngsflow＠hygenomics.com 08 Nov '13

08 Nov '13

Hi, there: I'm currently trying to migrate all repositories of main tool shed on http://toolshed.g2.bx.psu.edu/ to local, but got some problems. I'm wondering whether there is an existing tool can automatically do the job? thanks rgds, Ray ngsflow(a)hygenomics.com

3 3