August 2013 - galaxy-dev - lists.galaxyproject.org

GATK - Depth of Coverage Error
by Can Holyavkin 15 Aug '13

15 Aug '13

Hi Everyone, Recently, I tried to use GATK - Depth of Coverage tool with my own .bam file on Galaxy for the first time. However it gave following error for each output. (see screenshot) > Depth of Coverage on data 4 and data 1 (output cumulative coverage proportions sample) > error: > An error occurred with this dataset: /bin/sh: 1: samtools: not found Traceback (most recent call last): File "/home/can/galaxy-dist/tools/gatk/gatk_wrapper.py", line 126, in if __name__=="__main__": __main__() File "/home/can/galaxy-dist/tools/gatk/gatk_wrapper.py", line 9 I searched the mailing list, but I couldn't find similar problem that posted before. How can I solve it? Should I install GATK seperately on Galaxy? or anything else? Can Holyavkin PhD Student Istanbul Technical University Molecular Biology and Genetics Dept.

1 0

edgeR Tool Shed
by Richard Kuo 14 Aug '13

14 Aug '13

Hello, I have some questions about the edgeR (fcaramia) and digital_dge (fubar) repositories in tool shed. What is the difference between the two respositories in terms of being able to run edgeR? What is the proper way to install R so that the tool wrappers can use edgeR? What versions of R and edgeR do each of the repositories work with? Similaraly why is it that the repositories do not have information about which versions of R and edgeR are compatible? I have tried both repositories and have installed R and loaded the edgeR library but the tools do not seem to work. There are no errors coming up but the outputs are empty. The input file works with edgeR ran the traditional way through R (without Galaxy). Thank you, Richard

2 1

Setting torque queues
by Scott Hazelhurst 14 Aug '13

14 Aug '13

[Apologies for my previous email : finger slippage meant I sent before completing the email] Dear dev I would be grateful for some assistance in setting up our galaxy server to submit jobs to our torque server.. We are using release_2013.04.01 I have tried going through the install information but clearly I¹ve misunderstood something, and unfortunately most examples come from earlier versions Here is my job_conf.xml file ‹ this is just a temporary file while I set things up. My intention is to run clustalw on our cluster using the torque queue WitsLong, assuming our torque server is galaxy.bioinf.wits.ac.za Below that is the output I get in the log. From the torque log it complains that there is no default queue specified. Any help would be appreciated. Thanks Scott <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> </plugins> <handlers default="handlers"> <handler id="main" tags="handlers,pbs"/> </handlers> <destinations default="WitsLong"> <destination id="local" runner="local"/> <destination id="WitsLong" runner="pbs"> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> <param id="-p">128</param> </destination> </destinations> <tools> <tool id="foo" handler="trackster_handler"> <param id="source">trackster</param> </tool> <tool id="clustalw" destination="WitsLong"/> </tools> </job_conf> galaxy.jobs DEBUG 2013-04-30 15:28:13,481 (33) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/33 galaxy.jobs.handler DEBUG 2013-04-30 15:28:13,527 (33) Dispatching to pbs runner 146.141.75.218, 146.141.1.81 - - [30/Apr/2013:15:28:12 +0200] "POST /galaxy/tool_runner/index HTTP/1.1" 200 - "http://galaxy.bioinf.wits.ac.za/galaxy/tool_runner/rerun?id=9752b387803d3e1 e" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-04-30 15:28:14,213 (33) Persisting job destination (destination id: WitsLong) Exception AssertionError: AssertionError('State <sqlalchemy.orm.state.InstanceState object at 0x7f79682e5150> is not present in this identity map',) in <bound method InstanceState._cleanup of <sqlalchemy.orm.state.InstanceState object at 0x7f79682e5150>> ignored galaxy.jobs.handler INFO 2013-04-30 15:28:14,522 (33) Job dispatched galaxy.tools DEBUG 2013-04-30 15:28:15,189 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2013-04-30 15:28:15,192 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:15,957 (33) submitting file /home/galaxy/galaxy-dist/database/pbs/33.sh galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:15,958 (33) command is: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/000/dataset_2.dat" -o "/home/galaxy/galaxy-dist/database/files/000/dataset_60.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/000/dataset_61.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:15,963 (33) pbs_submit failed (try 1/5), PBS error 15039: Route rejected by all destinations galaxy.jobs DEBUG 2013-04-30 15:28:16,400 (34) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/34 galaxy.jobs.handler DEBUG 2013-04-30 15:28:16,416 (34) Dispatching to pbs runner 146.141.75.218, 146.141.1.81 - - [30/Apr/2013:15:28:13 +0200] "POST /galaxy/tool_runner/index HTTP/1.1" 200 - "http://galaxy.bioinf.wits.ac.za/galaxy/tool_runner/rerun?id=9752b387803d3e1 e" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-04-30 15:28:16,668 (34) Persisting job destination (destination id: WitsLong) galaxy.jobs.handler INFO 2013-04-30 15:28:16,689 (34) Job dispatched galaxy.tools DEBUG 2013-04-30 15:28:16,937 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2013-04-30 15:28:16,938 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:17,160 (34) submitting file /home/galaxy/galaxy-dist/database/pbs/34.sh galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:17,161 (34) command is: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/000/dataset_2.dat" -o "/home/galaxy/galaxy-dist/database/files/000/dataset_62.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/000/dataset_63.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:17,162 (34) pbs_submit failed (try 1/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:17,971 (33) pbs_submit failed (try 2/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:19,166 (34) pbs_submit failed (try 2/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:19,975 (33) pbs_submit failed (try 3/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:21,170 (34) pbs_submit failed (try 3/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:21,978 (33) pbs_submit failed (try 4/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:23,173 (34) pbs_submit failed (try 4/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:23,982 (33) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:25,176 (34) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs ERROR 2013-04-30 15:28:25,984 (33) All attempts to submit job failed galaxy.jobs.runners.pbs ERROR 2013-04-30 15:28:27,179 (34) All attempts to submit job failed <table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;"> <tr> <td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td> </tr> </table

2 1

Upstart script to manage a multi instance load balanced installation
by Seth Sims 14 Aug '13

14 Aug '13

After some work i've created an Upstart script which can manage a load balanced galaxy configuration as described in the wiki. I thought that I would put it on this list for other people to use. The script parses universe_wsgi.ini just like run.sh and spawns all of the servers it finds. It comes in two pieces galaxy.conf and galaxy-worker.conf. Once you place them both in /etc/init and make the proper edits for the environment a server can be started with "sudo start galaxy". The configuration starts the server at boot time and puts all of the instances under the management of upstart which deals with pids, logging to syslog and respawning if an instance crashes. I have just gotten this working reasonably well but have done basically no testing so there are bugs to be found. Any comments are welcome if anyone knows a better way to do something here. - Seth *galaxy.conf* ---------------------------- author "Seth Sims <seth.sims(a)gmail.com>" version "0.0.1 test" description "galaxy master process. Fetches eggs and spawns all of the servers it finds configured" start on started network-services # make sure that any eggs we download are at least owned by the galaxy group. # we cannot use setuid in this script because only root can issue the "start galaxy-worker" # command. But this way the galaxy instances should still be able to use their eggs. setgid galaxy # put galaxy root directory here chdir /srv/galaxy-dist/ pre-start script date echo "checking python version" python ./scripts/check_python.py [ $? -ne 0 ] && exit 1 echo "pre-fetching tossing out expired eggs" python ./scripts/check_eggs.py -q if [ $? -eq 0 ]; then echo "Some eggs are out of date, attempting to fetch..." python ./scripts/fetch_eggs.py if [ $? -eq 0 ]; then echo "Fetch Successful." else echo "Fetch failed." fi fi echo "starting servers" SERVERS=`sed -n 's/^\[server:$.*$\]/\1/ p' universe_wsgi.ini | xargs echo` for SERVER in ${SERVERS} ; do echo "starting server ${SERVER}" start galaxy-worker SERVER_NAME=${SERVER} done end script post-stop script SERVERS=`sed -n 's/^\[server:$.*$\]/\1/ p' universe_wsgi.ini | xargs echo` date echo "stopping galaxy servers". for SERVER in ${SERVERS} ; do echo "stopping ${SERVER}" stop galaxy-worker SERVER_NAME=${SERVER} done end script --------------------------- *galaxy-worker* author "Seth Sims <seth.sims(a)gmail.com>" version "0.0.1 test" description "Starts a galaxy server instance. This is run from the galaxy.conf file" instance $SERVER_NAME #make sure we are running as the galaxy user setuid galaxy setgid galaxy #put the galaxy root directory here chdir /srv/galaxy-dist/ #having multiple instances of galaxy using the same egg directory was causing a race #condition that was stopping the instances from starting correctly. So give each instance #its own directory under /tmp env PYTHON_EGG_CACHE=/tmp/${SERVER_NAME}_egg/ respawn script exec python ./scripts/paster.py serve universe_wsgi.ini --server-name=${SERVER_NAME} end script

2 4

Is there a way to re-run a failed job and check the "re-associated" checkbox from the API?
by Anthonius deBoer 14 Aug '13

14 Aug '13

Hi, I was very excited to see the re-run option and associate with paused jobs option working very well.. I was hoping to be able to re-run a job like this from the API, but cannot seem to find any API calls that corresponds to this here: http://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.… Is this not implemented yet, or just not documented? Thanks Thon

1 0

Jobs remain in queue until restart
by Thon Deboer 14 Aug '13

14 Aug '13

Hi, I have noticed that from time to time, the job queue seems to be "stuck" and can only be unstuck by restarting galaxy. The jobs seem to be in the queue state and the python job handler processes are hardly ticking over and the cluster is empty. When I restart, the startup procedure realizes all jobs are in the a "new state" and it then assigns a jobhandler after which the jobs start fine.. Any ideas? Thon P.S I am using the june version of galaxy and I DO set limits on my users in job_conf.xml as so: (Maybe it is related? Before it went into dormant mode, this user had started lots of jobs and may have hit the limit, but I assumed this limit was the number of running jobs at one time, right?) <?xml version="1.0"?> <job_conf> <plugins workers="4">  <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/> <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="8"/> <plugin id="cli" type="runner" load="galaxy.jobs.runners.cli:ShellJobRunner" workers="2"/> </plugins> <handlers default="handlers">  <handler id="handler0" tags="handlers"/> <handler id="handler1" tags="handlers"/> <handler id="handler2" tags="handlers"/> <handler id="handler3" tags="handlers"/>  </handlers> <destinations default="regularjobs">  <destination id="local" runner="local"/> <destination id="regularjobs" runner="drmaa" tags="cluster">  <param id="nativeSpecification">-V -q long.q -pe smp 1</param> </destination> <destination id="longjobs" runner="drmaa" tags="cluster,long_jobs">  <param id="nativeSpecification">-V -q long.q -pe smp 1</param> </destination> <destination id="shortjobs" runner="drmaa" tags="cluster,short_jobs">  <param id="nativeSpecification">-V -q short.q -pe smp 1</param> </destination> <destination id="multicorejobs4" runner="drmaa" tags="cluster,multicore_jobs">  <param id="nativeSpecification">-V -q long.q -pe smp 4</param> </destination>  <destination id="dynamic" runner="dynamic">  <param id="type">python</param> <param id="function">interactiveOrCluster</param> </destination> </destinations> <tools>  <tool id="bwa_wrapper" destination="multicorejobs4"/> </tools> <limits>  </limits> </job_conf>

6 7

Galaxy database check script
by graham etherington (TSL) 14 Aug '13

14 Aug '13

Hi, I've been getting some errors on our local Galaxy instance which haven't been solved by updating to the latest code. Some of them involve sql-type errors. My sys-admin informed me that "There are a couple of SQL errors being reported, i.e. text attempting to be inserted into an integer field within the sql statements." I'm also getting 'NoneType' object has no attribute 'x' errors [where 'x' can be a number of different attributes]. Are there any scripts in Galaxy that I can run to test the database. I'm wondering if there's some sort of corruption with the database. Many thanks, Graham Dr. Graham Etherington Bioinformatics Support Officer, The Sainsbury Laboratory, Norwich Research Park, Norwich NR4 7UH. UK Tel: +44 (0)1603 450601

2 1

Duplicated entries in "Manage installed tool shed repositories" table
by Peter Cock 14 Aug '13

14 Aug '13

Hi Greg, I'm seeing something strange on a system running the August stable release of galaxy-dist, $ sudo -u galaxy hg head changeset: 10393:d05bf67aefa6 branch: stable tag: tip user: Nate Coraor <nate(a)bx.psu.edu> date: Mon Aug 12 11:55:41 2013 -0400 summary: Added tag release_2013.08.12 for changeset 1ae95b3aa98d changeset: 10390:009088d5e76f user: Ross Lazarus <ross.lazarus(a)gmail.com> date: Sat Aug 10 11:28:28 2013 +1000 summary: Patch for missing options in FromDataTableOutputActionOption when these are used in an output filter 1. Run Galaxy 2. Log in as an administrator 3. Click on Admin (top menu) 4. Click on "Manage installed tool shed repositories" (left menu) 5. Observe duplicate entries: Names of repositories for which updates are available are highlighted in yellow. Description Owner Revision Installation Status Tool shed Galaxy wrappers for Emboss version 5.0.0 tools devteam c159a6ded912 Installed toolshed.g2.bx.psu.edu Galaxy applicable data formats used by Emboss tools. devteam a89163f31369 Installed toolshed.g2.bx.psu.edu Glimmer is a system for finding genes in microbial DNA, especially the genomes of bacteria, archaea, and viruses. bgruening b1ad88bbc5fa Installed toolshed.g2.bx.psu.edu Gene calling with glimmer and a trainingset of know genes. bgruening 22bfc027a718 Installed toolshed.g2.bx.psu.edu Contains a tool dependency definition that downloads and compiles version 1.61 of the biopython package. biopython b482933a6de5 Installed toolshed.g2.bx.psu.edu NCBI BLAST+ 2.2.26 (binaries only) iuc c85236cd100d Installed testtoolshed.g2.bx.psu.edu NCBI BLAST+ 2.2.26 (binaries only) iuc 9ff9bfdcd04f New testtoolshed.g2.bx.psu.edu NCBI BLAST+ 2.2.26 (binaries only) iuc 0e536576294c New testtoolshed.g2.bx.psu.edu Contains a tool dependency definition that downloads and compiles version 2.4 of the freetype font library. iuc 8761091302c4 Installed toolshed.g2.bx.psu.edu Contains a tool dependency definition that downloads and compiles version 1.7 of the the python numpy package iuc 74c21f9bdc39 Installed toolshed.g2.bx.psu.edu As you can see I have been trying the NCBI BLAST+ package from the TEST tool shed, http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Why are there these extra entries (older revisions) labelled with an "Installation Status" of "New"? Are they relics from failed installation attempts? Thanks, Peter

2 1

Fwd: Map with BWA results in correpted SAM file
by Moritz Juchler 14 Aug '13

14 Aug '13

Hey Folks, Is there really nobody that can help Geert and me? Thats quite important to me right now. This is obviously not something specific to me, as Geert has the exact same error. I meanwhile tried to replace the id with "1" but it still doesnt work. Best Moritz ______________________________ Von: galaxy-dev-bounces(a)lists.bx.psu.eduIm Auftrag vonGeert Vandeweyer Gesendet: Donnerstag, 8. August 2013 17:18:59 (UTC+01:00) Amsterdam, Berlin, Bern, Rom, Stockholm, Wien An: galaxy-dev(a)lists.bx.psu.edu Betreff: Re: [galaxy-dev] Map with BWA results in correpted SAM file to add on this : we have similar issues (sam-to-bam conversion fails with similar errors). it seems to be related to the BWA output getting messed up, with (part of) columns missing or duplicated on some lines. I have not found a systematic pattern in the errors, they seem to happen rather random. On 08/08/2013 05:06 PM, Moritz Juchler wrote: Dear Galaxy Community, I have a local instance and installed 0.5.9-r16 BWA and the toolshed wrapper. The mapping is successful. I then use the Filter Sam Tool on the sam file from the alignment, but it spits out this error: Dataset 26: Filter SAM on data 24 Tool execution generated the following error message: Traceback (most recent call last): File "/home/trr/galaxy-dist/tools/samtools/sam_bitwise_flag_filter.py", line 148, in <module> if __name__ == "__main__": main() File "/home/trr/galaxy-dist/tools/samtools/sam_bitwise_flag_filter.py", line 137, in main flags = int( fields[flag_col] ) ValueError: invalid literal for int() with base 10: 'RG:Z:lane712s006433' I have the same workflow online and did the exact same steps on the same fastq files. Is there anything I am missing? Is there any information I can provide to answer this question? Best Moritz ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be<mailto:geert.vandeweyer@ua.ac.be> http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

3 3

outputs label attribute
by Jun Fan 14 Aug '13

14 Aug '13

Dear all, I have managed to use something like <outputs> <data format="fasta" name="output" label="#echo os.path.splitext (str ($input.name))[0]#-ORF.fasta"/> </outputs> to display the wanted label for the dataset in the history. However when I applied the same code to other tool wrappers, some weird things happened: the wrapper can still be loaded in the web page, but when I click the execute button to submit the job, nothing happened except the web browser giving the status "wait for localhost". I cannot think of any reason for that. The main difference I can think of between the successful wrapper and the failed ones is that in the failed ones I have to do some file manipulations as galaxy always stores its dataset with the file suffix .dat which is not acceptable in the tool, e.g. proteowizard. I have to rename it with the correct suffix and move the final result to the output file. Additional related questions: 1. How to display the selected value of a conditional element in the label? $conditional.value? 2. How to display different strings according to the conditional selection? Similar to command element to use #if #elsif #end if? 3. When using <data name="output" format="csv"> <change_format> <when input="type" value="html" format="html"/> </change_format> </data>, could I use label attribute in the data and when elements? Look forward to your kind answers. Many thanks in advance. Best regards! Jun

2 3