- galaxy-dev - lists.galaxyproject.org

Multiple output files do not appear in history
by Christoph Malisi 15 Aug '13

15 Aug '13

Hey, I'm writing a XML for a tool that produces a number of output files. As the number is not known before the tool is run, I used the strategy outlined here: http://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_… First, I tried my tool definition in a galaxy instance in a virtual machine to not interrupt our production instance (we have a local instance set up). Here, everything worked fine. When trying the same code in our production galaxy instance, it didn't: Regardless of how many output files were produced, only one appears in the history. (the first one, corresponding to "output1" in the references wiki entry). When looking at the temp directory provided to the tool by galaxy ("$__new_file_dir__" in the wiki, it always points to "galaxy-dist/database/tmp/"), the additional output files have been created by my tool and are named correctly (e.g. primary_123_output2_visible_fastq, primary_123_output3_visible_fastq, …) Both instances are new, based on release_2013.06.03. Is there any setting in galaxy somewhere that could prevent this from working? Thanks for your help Chris PS: I wrote a little python script that just creates multiple output files, and the discrepancy between the two instances is reproducible. Here is the code for the python script and the corresponding XML: ####### import subprocess import argparse import os def main(): parser = argparse.ArgumentParser() parser.add_argument('-i', type=int) parser.add_argument('output1') parser.add_argument('output1_id') parser.add_argument('out_dir') args = parser.parse_args() print 'Wrapper arguments:', args with open(args.output1, 'w') as f: f.write('output1') for i in range(2, args.i+1): name = 'output%i'%i file_type = 'text' fname = '%s_%s_%s_%s_%s' % ('primary', args.output1_id, name, 'visible', file_type) with open(os.path.join(args.out_dir, fname), 'w') as f: f.write('output%i'%i) if __name__ == "__main__": main() ############################ ########################### <tool name="MultiOutTest" id="multiouttest"> <description>Multiple Outputs test</description> <command interpreter="python"> multiout.py -i $how_many $output1 $output1.id $__new_file_path__ </command> <inputs> <param name="how_many" type="integer" value="2" label="How many output files?"/> </inputs> <outputs> <data name="output1" format="txt"/> </outputs> <help> This is some help text. </help> </tool> #################################

2 2

Job listing exception
by Kyle Ellrott 15 Aug '13

15 Aug '13

I'm getting an exception when trying to look at the admin job management screen. It looks like SQLAlchemy doesn't like non-ascii characters. Any ideas about what to do? Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128) URL: http://pk.kilokluster.ucsc.edu:8079/admin/jobs File '/inside/depot4/galaxy/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/framework/__init__.py', line 221 in decorator return func( self, trans, *args, **kwargs ) File '/inside/depot4/galaxy/lib/galaxy/web/base/controllers/admin.py', line 1053 in jobs for job in jobs: File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/query.py', line 2341 in instances File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3204 in fetchall File '/inside/depot4/galaxy/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py', line 3171 in _fetchall_impl UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 404: ordinal not in range(128)

2 2

Display data in browser
by Gromobir 15 Aug '13

15 Aug '13

Hello list, I'm currently trying to create a new tool, which has an .xhtml file as its output. After finishing the job, I would like to be able to click on the "eye icon" aka "Display data in browser" and display the resulting .xhtml file in the frame to the left of my tool history. Unfortunately I have no idea yet how I could accomplish this. Any help is highly appreciated. Best regards, Gromobir

2 1

Searching on Tool Shed gives multiple revisions
by Peter Cock 15 Aug '13

15 Aug '13

Hi all, Using the left hand menu items "Search for valid tools" and "Search for workflows" gives multiple revisions - whereas searching via the prominent search box on the welcome page does not (only the latest revisions are shown). I would prefer only to get the latest revision - otherwise the search results are a bit confusing and the old revisions are noise drowning out the main information. Regards, Peter

2 2

Test Tool Shed down?
by Lionel Guy 15 Aug '13

15 Aug '13

Good morning Galaxy, It appears that the test tool shed is down, returning a 502 Bad Gateway message. The main tool shed is fine. I'm not sure that this is the right place to announce that, but since I don't know who is in charge of the servers, I give it a shot... Cheers, Lionel

3 4

GATK - Depth of Coverage Error
by Can Holyavkin 15 Aug '13

15 Aug '13

Hi Everyone, Recently, I tried to use GATK - Depth of Coverage tool with my own .bam file on Galaxy for the first time. However it gave following error for each output. (see screenshot) > Depth of Coverage on data 4 and data 1 (output cumulative coverage proportions sample) > error: > An error occurred with this dataset: /bin/sh: 1: samtools: not found Traceback (most recent call last): File "/home/can/galaxy-dist/tools/gatk/gatk_wrapper.py", line 126, in if __name__=="__main__": __main__() File "/home/can/galaxy-dist/tools/gatk/gatk_wrapper.py", line 9 I searched the mailing list, but I couldn't find similar problem that posted before. How can I solve it? Should I install GATK seperately on Galaxy? or anything else? Can Holyavkin PhD Student Istanbul Technical University Molecular Biology and Genetics Dept.

1 0

edgeR Tool Shed
by Richard Kuo 15 Aug '13

15 Aug '13

Hello, I have some questions about the edgeR (fcaramia) and digital_dge (fubar) repositories in tool shed. What is the difference between the two respositories in terms of being able to run edgeR? What is the proper way to install R so that the tool wrappers can use edgeR? What versions of R and edgeR do each of the repositories work with? Similaraly why is it that the repositories do not have information about which versions of R and edgeR are compatible? I have tried both repositories and have installed R and loaded the edgeR library but the tools do not seem to work. There are no errors coming up but the outputs are empty. The input file works with edgeR ran the traditional way through R (without Galaxy). Thank you, Richard

2 1

Setting torque queues
by Scott Hazelhurst 14 Aug '13

14 Aug '13

[Apologies for my previous email : finger slippage meant I sent before completing the email] Dear dev I would be grateful for some assistance in setting up our galaxy server to submit jobs to our torque server.. We are using release_2013.04.01 I have tried going through the install information but clearly I¹ve misunderstood something, and unfortunately most examples come from earlier versions Here is my job_conf.xml file ‹ this is just a temporary file while I set things up. My intention is to run clustalw on our cluster using the torque queue WitsLong, assuming our torque server is galaxy.bioinf.wits.ac.za Below that is the output I get in the log. From the torque log it complains that there is no default queue specified. Any help would be appreciated. Thanks Scott <?xml version="1.0"?>  <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/> <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner"/> </plugins> <handlers default="handlers"> <handler id="main" tags="handlers,pbs"/> </handlers> <destinations default="WitsLong"> <destination id="local" runner="local"/> <destination id="WitsLong" runner="pbs"> <param id="Resource_List">walltime=72:00:00,nodes=1:ppn=8</param> <param id="-p">128</param> </destination> </destinations> <tools> <tool id="foo" handler="trackster_handler"> <param id="source">trackster</param> </tool> <tool id="clustalw" destination="WitsLong"/> </tools> </job_conf> galaxy.jobs DEBUG 2013-04-30 15:28:13,481 (33) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/33 galaxy.jobs.handler DEBUG 2013-04-30 15:28:13,527 (33) Dispatching to pbs runner 146.141.75.218, 146.141.1.81 - - [30/Apr/2013:15:28:12 +0200] "POST /galaxy/tool_runner/index HTTP/1.1" 200 - "http://galaxy.bioinf.wits.ac.za/galaxy/tool_runner/rerun?id=9752b387803d3e1 e" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-04-30 15:28:14,213 (33) Persisting job destination (destination id: WitsLong) Exception AssertionError: AssertionError('State <sqlalchemy.orm.state.InstanceState object at 0x7f79682e5150> is not present in this identity map',) in <bound method InstanceState._cleanup of <sqlalchemy.orm.state.InstanceState object at 0x7f79682e5150>> ignored galaxy.jobs.handler INFO 2013-04-30 15:28:14,522 (33) Job dispatched galaxy.tools DEBUG 2013-04-30 15:28:15,189 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2013-04-30 15:28:15,192 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:15,957 (33) submitting file /home/galaxy/galaxy-dist/database/pbs/33.sh galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:15,958 (33) command is: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/000/dataset_2.dat" -o "/home/galaxy/galaxy-dist/database/files/000/dataset_60.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/000/dataset_61.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:15,963 (33) pbs_submit failed (try 1/5), PBS error 15039: Route rejected by all destinations galaxy.jobs DEBUG 2013-04-30 15:28:16,400 (34) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/34 galaxy.jobs.handler DEBUG 2013-04-30 15:28:16,416 (34) Dispatching to pbs runner 146.141.75.218, 146.141.1.81 - - [30/Apr/2013:15:28:13 +0200] "POST /galaxy/tool_runner/index HTTP/1.1" 200 - "http://galaxy.bioinf.wits.ac.za/galaxy/tool_runner/rerun?id=9752b387803d3e1 e" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:20.0) Gecko/20100101 Firefox/20.0" galaxy.jobs DEBUG 2013-04-30 15:28:16,668 (34) Persisting job destination (destination id: WitsLong) galaxy.jobs.handler INFO 2013-04-30 15:28:16,689 (34) Job dispatched galaxy.tools DEBUG 2013-04-30 15:28:16,937 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2013-04-30 15:28:16,938 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:17,160 (34) submitting file /home/galaxy/galaxy-dist/database/pbs/34.sh galaxy.jobs.runners.pbs DEBUG 2013-04-30 15:28:17,161 (34) command is: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/000/dataset_2.dat" -o "/home/galaxy/galaxy-dist/database/files/000/dataset_62.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/000/dataset_63.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:17,162 (34) pbs_submit failed (try 1/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:17,971 (33) pbs_submit failed (try 2/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:19,166 (34) pbs_submit failed (try 2/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:19,975 (33) pbs_submit failed (try 3/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:21,170 (34) pbs_submit failed (try 3/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:21,978 (33) pbs_submit failed (try 4/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:23,173 (34) pbs_submit failed (try 4/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:23,982 (33) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs WARNING 2013-04-30 15:28:25,176 (34) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs ERROR 2013-04-30 15:28:25,984 (33) All attempts to submit job failed galaxy.jobs.runners.pbs ERROR 2013-04-30 15:28:27,179 (34) All attempts to submit job failed <table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;"> <tr> <td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td> </tr> </table

2 1

Upstart script to manage a multi instance load balanced installation
by Seth Sims 14 Aug '13

14 Aug '13

After some work i've created an Upstart script which can manage a load balanced galaxy configuration as described in the wiki. I thought that I would put it on this list for other people to use. The script parses universe_wsgi.ini just like run.sh and spawns all of the servers it finds. It comes in two pieces galaxy.conf and galaxy-worker.conf. Once you place them both in /etc/init and make the proper edits for the environment a server can be started with "sudo start galaxy". The configuration starts the server at boot time and puts all of the instances under the management of upstart which deals with pids, logging to syslog and respawning if an instance crashes. I have just gotten this working reasonably well but have done basically no testing so there are bugs to be found. Any comments are welcome if anyone knows a better way to do something here. - Seth *galaxy.conf* ---------------------------- author "Seth Sims <seth.sims(a)gmail.com>" version "0.0.1 test" description "galaxy master process. Fetches eggs and spawns all of the servers it finds configured" start on started network-services # make sure that any eggs we download are at least owned by the galaxy group. # we cannot use setuid in this script because only root can issue the "start galaxy-worker" # command. But this way the galaxy instances should still be able to use their eggs. setgid galaxy # put galaxy root directory here chdir /srv/galaxy-dist/ pre-start script date echo "checking python version" python ./scripts/check_python.py [ $? -ne 0 ] && exit 1 echo "pre-fetching tossing out expired eggs" python ./scripts/check_eggs.py -q if [ $? -eq 0 ]; then echo "Some eggs are out of date, attempting to fetch..." python ./scripts/fetch_eggs.py if [ $? -eq 0 ]; then echo "Fetch Successful." else echo "Fetch failed." fi fi echo "starting servers" SERVERS=`sed -n 's/^\[server:$.*$\]/\1/ p' universe_wsgi.ini | xargs echo` for SERVER in ${SERVERS} ; do echo "starting server ${SERVER}" start galaxy-worker SERVER_NAME=${SERVER} done end script post-stop script SERVERS=`sed -n 's/^\[server:$.*$\]/\1/ p' universe_wsgi.ini | xargs echo` date echo "stopping galaxy servers". for SERVER in ${SERVERS} ; do echo "stopping ${SERVER}" stop galaxy-worker SERVER_NAME=${SERVER} done end script --------------------------- *galaxy-worker* author "Seth Sims <seth.sims(a)gmail.com>" version "0.0.1 test" description "Starts a galaxy server instance. This is run from the galaxy.conf file" instance $SERVER_NAME #make sure we are running as the galaxy user setuid galaxy setgid galaxy #put the galaxy root directory here chdir /srv/galaxy-dist/ #having multiple instances of galaxy using the same egg directory was causing a race #condition that was stopping the instances from starting correctly. So give each instance #its own directory under /tmp env PYTHON_EGG_CACHE=/tmp/${SERVER_NAME}_egg/ respawn script exec python ./scripts/paster.py serve universe_wsgi.ini --server-name=${SERVER_NAME} end script

2 4

Is there a way to re-run a failed job and check the "re-associated" checkbox from the API?
by Anthonius deBoer 14 Aug '13

14 Aug '13

Hi, I was very excited to see the re-run option and associate with paused jobs option working very well.. I was hoping to be able to re-run a job like this from the API, but cannot seem to find any API calls that corresponds to this here: http://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.… Is this not implemented yet, or just not documented? Thanks Thon

1 0