May 2013 - galaxy-dev - lists.galaxyproject.org

tool_dependencies inside tool_dependencies
by Björn Grüning 23 May '13

23 May '13

Hi, is there a general rule to handle dependencies inside of tool_dependencies.xml? Lets assume I write a matplotlib orphan tool_dependencies.xml file. matplotlib depends on numpy. Numpy has already a orphan definition. Is there a way to include numpy as dependency inside the matplotlib-definition, so that I did not need to fetch and compile numpy inside of matplotlib? I tried to specify it beforehand but that did not work. Thanks! Bjoern

6 15

writing <command> tag in xml file for a C++ tool
by vijayalakshmi 23 May '13

23 May '13

Hi all, I want to include a new tool to galaxy. The tool is written in C++. How do I write the xml file to execute a cpp program? The tool that I am trying to integrate is https://code.google.com/p/bedtools/ I have the class file to be executed and the command to execute it. For example, 'IntersectBed' is my class file. The command to execute it is $intersectBed -a reads.bed -b genes.bed (reads.bed and genes.bed are input files provided by the user) How should I write the <command> tag in the xml file so as to execute the above command? Since C++ is complied as opposed to Python which is interpreted, I am not sure how to write the <command> tag. I am a newbie. I appreciate any guidance offered! Thanks, VJ.

2 1

Hard time getting UCSC to work with our server
by Shane Sturrock 23 May '13

23 May '13

I'm having a pretty torrid time with this because we've had to implement active directory authentication for Apache. It works and users can log in with their AD accounts and that is all fine. The problems arise when I try and get UCSC to pick up a BAM file. The first problem is that although the Galaxy server is configured to for UCSC as a display server, putting the following into the Apache conf as recommended here http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy doesn't work. 1 <Location "/root/display_as"> 2 Satisfy Any 3 Order deny,allow 4 Deny from all 5 Allow from hgw1.cse.ucsc.edu 6 Allow from hgw2.cse.ucsc.edu 7 Allow from hgw3.cse.ucsc.edu 8 Allow from hgw4.cse.ucsc.edu 9 Allow from hgw5.cse.ucsc.edu 10 Allow from hgw6.cse.ucsc.edu 11 Allow from hgw7.cse.ucsc.edu 12 Allow from hgw8.cse.ucsc.edu 13 </Location> All that happens is you get 401 Authorization Required errors indicating that the Apache server isn't letting the UCSC servers non-authenticated access. Now, if I add the Allow from hgw etc into the <Location "/"> block then it gets through and then I hit the next problem which is that although I've mapped some read data against dm3 I'm getting failures inside Galaxy itself related to serving the bai file UCSC is requesting. Here's the galaxy_access log: 192.168.240.21 - shane [23/May/2013:15:22:24 +1200] "GET /display_application/d3a4d6a5256f2d9a/ucsc_bam/main HTTP/1.1" 302 406 hgw4.cse.ucsc.edu - - [23/May/2013:15:22:24 +1200] "GET /dataset/display_application?app_action=param&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=track&link_name=main&dataset_id=d3a4d6a5256f2d9a HTTP/1.0" 200 311 hgw4.cse.ucsc.edu - - [23/May/2013:15:22:29 +1200] "HEAD /dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a HTTP/1.0" 200 - hgw4.cse.ucsc.edu - - [23/May/2013:15:22:31 +1200] "HEAD /dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a.bai HTTP/1.0" 500 - hgw4.cse.ucsc.edu - - [23/May/2013:15:22:30 +1200] "GET /dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a HTTP/1.0" 200 472000 Note the 500 when requesting the bai file. This is what my manager.log is showing when this happens. 128.114.119.134 - - [23/May/2013:15:22:29 +1300] "HEAD /dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a HTTP/1.1" 200 3301004 "-" "genome.ucsc.edu/net.c" ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 60484) Traceback (most recent call last): File "/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 618, in __init__ self.finish() File "/usr/lib64/python2.6/SocketServer.py", line 661, in finish self.wfile.flush() File "/usr/lib64/python2.6/socket.py", line 303, in flush self._sock.sendall(buffer(data, write_offset, buffer_size)) error: [Errno 32] Broken pipe ---------------------------------------- 128.114.119.134 - - [23/May/2013:15:22:30 +1300] "GET /dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a HTTP/1.1" 200 3301004 "-" "genome.ucsc.edu/net.c" 128.114.119.134 - - [23/May/2013:15:22:31 +1300] "HEAD /dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a.bai HTTP/1.1" 500 - "-" "genome.ucsc.edu/net.c" Error - <type 'exceptions.TypeError'>: Non-hexadecimal digit found ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 60486) Traceback (most recent call last): File "/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 618, in __init__ self.finish() File "/usr/lib64/python2.6/SocketServer.py", line 661, in finish self.wfile.flush() File "/usr/lib64/python2.6/socket.py", line 303, in flush self._sock.sendall(buffer(data, write_offset, buffer_size)) error: [Errno 32] Broken pipe ---------------------------------------- ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 60485) Traceback (most recent call last): File "/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 618, in __init__ self.finish() File "/usr/lib64/python2.6/SocketServer.py", line 661, in finish self.wfile.flush() File "/usr/lib64/python2.6/socket.py", line 303, in flush self._sock.sendall(buffer(data, write_offset, buffer_size)) error: [Errno 32] Broken pipe ---------------------------------------- And here's the text version in the browser (edited out our server's hostname though): URL: https://<ourgalaxyserver>/dataset/display_application?app_action=data&user_id=207edd50b8d72f8b&app_name=ucsc_bam&action_param=galaxy_d3a4d6a5256f2d9a.bam&link_name=main&dataset_id=d3a4d6a5256f2d9a.bai File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 125 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 182 in handle_request body = method( trans, **kwargs ) File '/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/dataset.py', line 727 in display_application data, user = decode_dataset_user( trans, dataset_id, user_id ) File '/home/galaxy/galaxy-dist/lib/galaxy/datatypes/display_applications/util.py', line 22 in decode_dataset_user dataset_id = trans.security.decode_id( dataset_hash ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/security/__init__.py', line 60 in decode_id return int( self.id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( "!" ) ) File '/home/galaxy/galaxy_env/lib64/python2.6/encodings/hex_codec.py', line 42 in hex_decode output = binascii.a2b_hex(input) TypeError: Non-hexadecimal digit found I've been banging my head against this all day :-( Shane Sturrock

1 0

Test Tool Shed still running tests marked as skip?
by Peter Cock 22 May '13

22 May '13

Hi Greg, There may be an issue with the new feature to mark a specific revision to skip tool tests. I ticked this earlier this week on this repository, yet the tests appear to have been run overnight (and failed as expected): http://testtoolshed.g2.bx.psu.edu/view/peterjc/tmhmm_and_signalp/2b35b5c4b7… Revision: 18:2b35b5c4b7f4 Regards, Peter

2 1

<command> tag in xml file for a C++ tool
by vijayalakshmi 22 May '13

22 May '13

Hi all, I want to include a new tool to galaxy. The tool is written in C++. How do I write the xml file to execute a cpp program? The tool that I am trying to integrate is https://code.google.com/p/bedtools/ I have the class file to be executed and the command to execute it. For example, 'IntersectBed' is my class file. The command to execute it is $intersectBed -a reads.bed -b genes.bed (reads.bed and genes.bed are input files provided by the user) How should I write the <command> tag in the xml file so as to execute the above command? Since C++ is complied as opposed to Python which is interpreted, I am not sure how to write the <command> tag. I am a newbie. I appreciate any guidance offered! Thanks, VJ.

3 2

nginx Proxy to VM problem
by Michael E. Cotterell 22 May '13

22 May '13

Hey everyone, I'm having a weird login problem (think it's cookie related) after setting up Galaxy on a virtual machine and trying to proxy_pass to it using nginx. It says I'm logged in, but when I get redirected to the front page I'm no longer logged in. When I visit subsequent pages I'm also not logged in. On the server, here is the relevant part of my nginx configuration (I'm going to setup static serving after I get the login issue resolved): upstream galaxy_app { server 192.168.1.11:8080; } server { ... location /galaxy { proxy_pass http://galaxy_app; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } ... } The dots just mean that I have other things in there for the forwarding I do with other VMs. On the VM, in my universe_wsgi.ini file, I have made sure to set the following (just showing the parts I think are relevant): [server:main] port = 8080 host = 0.0.0.0 [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy [app:main] filter-with = proxy-prefix cookie_path = /galaxy I should note that I might end up changing this configuration to use a CNAME at some point (thus avoiding the prefix issue), but for now I'm totally lost as to why I'm not able to login using my current configuration. Any ideas? Thanks! Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/

1 2

location of .loc-files/data tables -- bug or feature?
by Christian Hundsrucker 22 May '13

22 May '13

Hello, Galaxy developers! I do a bit of tweaking to a local galaxy instance. Thus, I just realized that subdirectories are not supported for the location of .loc-files/data tables. E.g. tool-data/somesub/my.loc is not recognized even if this location is correctly linked in the tool_data_table_conf.xml (<file path="tool-data/somesub/my.loc" />). Via the Galaxy-Admin interface and 'Server', 'View data tables registry' the .loc file is marked as a "Missing index file" - in the example the displayed location would be "tool-data/my.loc" (without the 'somesub' subdirectory). As soon as I store 'my.loc' directly in the tool-data directory galaxy runs smoothly again, even (!) if I keep the subdirectory in the file path in tool_data_table_conf.xml. Is galaxy meant to behave like this? If so, a word about it in the documentation would be really helpful! Best, Christian version: changeset: 8796:961ae35ba612 branch: stable tag: tip parent: 8794:1c7174911392 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Feb 08 11:23:04 2013 -0500

2 2

workflow startup issues
by Geert Vandeweyer 22 May '13

22 May '13

Hi, After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes. I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either. Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation. best, Geert -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

2 5

Re: [galaxy-dev] Irregular failures on cluster (pbs_python?)
by Fields, Christopher J 21 May '13

21 May '13

On May 21, 2013, at 2:13 PM, Christopher Fields <cjfields(a)illinois.edu> wrote: > On May 20, 2013, at 3:08 PM, Nate Coraor <nate(a)bx.psu.edu> wrote: >> ... >> Hi Chris, >> >> We're disconnecting under all normal conditions and most error conditions - it looks like only a few conditions would not properly disconnect: >> >> - If pbs_submit() fails 5 times in a row >> - If an exception is raised anywhere in the queue_job() method after pbs_connect() is called >> - If the call to pbs_statjob() in check_all_jobs() or check_single_job() raises an exception (if that's even possible) >> >> For the exceptions, you would see that an exception was caught in the log file, so you should be able to determine if this is happening. >> >> For the pbs_submit() case, you'd see the message "All attempts to submit job failed". >> >> You may want to move the call to pbs_connect() in queue_job() so that it occurs immediately prior to the call to pbs_submit() and see if that makes a difference. The reason we connect so early on is to avoid writing out the job's files if the PBS server doesn't exist anyway. > > Yes, seeing the pbs_submit() case. Lowering the # handlers does seem to help, but we still seem to run into this after Sorry, saw a unicorn. Meant, 'we still seem to run into this after a period of time on the cluster'. Coffee… :P chris

1 0

Irregular failures on cluster (pbs_python?)
by Fields, Christopher J 21 May '13

21 May '13

I've been seeing this error popping up quite a bit recently (we're using Torque 3.0.5), which is giving a general 'cluster failure' Galaxy error on jobs: galaxy.jobs.runners.pbs WARNING 2013-05-19 18:56:44,073 (10588) pbs_submit failed (try 1/5), PBS error 15033: No free connections This just recently started springing up after we had been using the cluster for over a year, not sure why it would start acting up now. This appears to be related to a bug/defect with pbs_python, which doesn't seem to have been fixed yet (I posted a query whether this has been addressed): https://oss.trac.surfsara.nl/pbs_python/ticket/29 Restarting helps, but are there any other recommended workarounds? Is the only solution recompiling Torque with NCONNECTS? chris

2 4