March 2011 - galaxy-dev - lists.galaxyproject.org

Galaxy job runner queue management questions
by Ry4an Brase 16 Mar '11

16 Mar '11

As use of our Galaxy installation is picking up, we're getting a lot of requests for greater fairness and transparency in the Galaxy job runner area. As I understand things the primary tool Galaxy gives us to affect processing order and wait times with our torque-based setup is the ability to map specific tools to varying queues or to keep them on a local-runner. On one end of the spectrum I could see a simple division of small/fast/light jobs on local and big/heavy/slow job on a single cluster queue. On the other extreme one could set up a queue per tool and use sophisticated queue management stuff on the torque side of things to balance capacity across tools, users, expected processing time, etc. How are other sites handling this? -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institute http://www.msi.umn.edu

3 3

Dutch Galaxy Hackathon
by Freek de Bruijn 16 Mar '11

16 Mar '11

Hi everyone, Galaxy is in use for quite some time now in The Netherlands. In April, NBIC (Netherlands Bioinformatics Centre, www.nbic.nl) is organizing a Hackathon and Galaxy is one of the projects that have been selected. A group of around 8 developers will work on a number of subjects for two days. We are planning to learn more about Galaxy and improve (our use of) it. Potential subjects are: - Call another workflow within a workflow - Loop over files in a directory - Support interactive visualization - Improving NBIC Galaxy UI - Making the access to NGS/Proteomics pipelines easier - Customization of the tool panel - Improving tool installation We would really appreciate suggestions and other input from the Galaxy Team. Thanks in advance! More information is available on our Hackathon wiki: https://wiki.nbic.nl/index.php/NBIC_Galaxy_Hackathon_project. Best regards, Freek de Bruijn -- Freek de Bruijn Scientific programmer | BioAssist Netherlands Bioinformatics Centre (www.nbic.nl) E-mail: freek.de.bruijn(a)nbic.nl Mobile: +31 6 30521044 Skype: freek.de.bruijn

1 0

username or email address from tool
by Glen Beane 16 Mar '11

16 Mar '11

I am implementing a new tool and I would like to reference the galaxy username and/or email address of the galaxy user running the tool (we require all of our users log in). Can this information be referenced in a tool .xml file? -- Glen L. Beane Senior Software Engineer The Jackson Laboratory (207) 288-6153

2 1

Running a workflow programatically
by Darren Brown 15 Mar '11

15 Mar '11

Hello folks, I am using a workflow that needs to be run many times on many different inputs. I have hacked around and figured out how to pull multiple inputs from a history, but I am a little baffled on how to run a galaxy workflow programatically. I have searched around fairly exhaustively and am wondering if this is something that anyone else has come across and accomplished. And links or pointers? Take care, Darren

2 6

Fwd: Galaxy process
by SHAUN WEBB 15 Mar '11

15 Mar '11

Further to my email below. I have a data library that contains many ~117Gb of NGS data, uploaded via file system path. This library was always slow to open (about 10s) but now takes several minutes or not at all. Thanks for any help on this. I am still experiencing a memory leak that I can't pinpoint and it is only dissipated by restarting the server. At the moment my debugging level is set to INFO. Is there anything I can change in the universe file to try to trace this. Thanks! Shaun ----- Forwarded message from swebb1(a)staffmail.ed.ac.uk ----- Date: Wed, 09 Mar 2011 10:15:13 +0000 From: SHAUN WEBB <swebb1(a)staffmail.ed.ac.uk> Subject: Galaxy process To: galaxy dev <galaxy-dev(a)bx.psu.edu> Hi, since making the last update I have found some new warnings in my paster.log, it also seems as though the galaxy process starts to gather memory and eventually hang (35% of 64G memory). I've posted the entries below. If anyone could help me understand what is going on that would be great. Thanks. Shaun Webb paste.httpserver.ThreadPool INFO 2011-03-09 09:49:49,962 No idle tasks, and only 0 busy tasks; adding 5 more workers paste.httpserver.ThreadPool INFO 2011-03-09 09:49:58,754 No idle tasks, and only 4 busy tasks; adding 1 more workers paste.httpserver.ThreadPool INFO 2011-03-09 09:51:47,301 Culling 6 extra workers (5 idle workers present) paste.httpserver.ThreadPool INFO 2011-03-09 09:55:17,163 No idle tasks, and only 0 busy tasks; adding 5 more workers 129.215.14.72 - - [09/Mar/2011:09:48:40 +0100] "GET /history HTTP/1.1" 500 - "http://bifx-core.bio.ed.ac.uk:8080/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" paste.httpserver.ThreadPool INFO 2011-03-09 10:13:16,956 Culling 5 extra workers (7 idle workers present) 212.183.140.59 - - [09/Mar/2011:10:13:17 +0100] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.102 Safari/534.13" paste.httpserver.ThreadPool INFO 2011-03-09 10:14:35,715 No idle tasks, and only 2 busy tasks; adding 3 more workers paste.httpserver.ThreadPool WARNING 2011-03-09 10:15:15,104 Thread 140283224094464 hung (working on task for 3096 seconds) ---------------------------------------- Exception happened during processing of request from ('212.183.140.59', 10871) Traceback (most recent call last): File "/usr/lib/python2.6/SocketServer.py", line 281, in _handle_request_noblock self.process_request(request, client_address) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 1037, in process_request lambda: self.process_request_in_thread(request, client_address)) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 617, in add_task self.kill_hung_threads() File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 778, in kill_hung_threads self.kill_worker(worker.thread_id) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 705, in kill_worker killthread.async_raise(thread_id, SystemExit) File "/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/util/killthread.py", line 22, in async_raise raise ValueError("invalid thread id") ValueError: invalid thread id ---------------------------------------- -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ----- End forwarded message ----- -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

5 10

parallel computing
by Yanji Xu 15 Mar '11

15 Mar '11

Dear Sir/Madam, Does Galaxy support parallel computing? Thanks, Yanji

2 1

Database cleanup and user management
by Sebastian J. Schultheiss 14 Mar '11

14 Mar '11

Dear Galaxy Developers, we are experimenting with cleanup strategies for our production galaxy server that is used by the public. How are you treating the user- uploaded data on galaxy-main? We would like to see how much space is used by an individual user and evenutally be able to remove them after some time of inactivity (>180 days?), ideally preceded by 1-2 warning emails. Can the current cleanup scripts handle that? Thanks for your help! -- Sebastian

3 10

Connecting Galaxy to multiple Remote Clusters?
by Joe Cruz 14 Mar '11

14 Mar '11

Hello, I've read through past emails that connecting Galaxy to a remote cluster is possible but I wanted to ask if it was possible to setup Galaxy as a standalone webserver and have it use multiple remote clusters (multiple head nodes each with their own compute nodes) and if so what the best strategy to implement this is? Also on one of these clusters we are not able to mount an NFS directory onto the compute nodes, since we do not own/manage it. Your help is greatly appreciated. Thanks, Joe

2 1

testing updates on Galaxy CloudMan
by Isabelle Phan 12 Mar '11

12 Mar '11

I am at step 2 of Galaxy CloudMan deployment. "Now, manipulate Galaxy as desired." How do I test that my updates are working before proceeding to step 3? I assumed I could just type 'run.sh'. This starts Galaxy but: - fasta files do not display in the middle pane - png images generated by e.g. boxplot do not display in the middle pane - saving images does not work (get empty file) I need to know how people are customizing their cloud instance of galaxy. If it is NOT possible to test modifications in the usual way PLEASE would somebody simply confirm? I really would appreciate a response. Isabelle -- Isabelle Phan, DPhil Seattle Biomedical Research Institute +1(206)256 7113

2 3

Upload files from filesystem paths - backwards checkboxes
by Peter Cock 11 Mar '11

11 Mar '11

Hi all, Via the Galaxy admin interface, I'm currently importing some BAM files into a library from the local file system. Since they are big, I don't want Galaxy to copy them. So I looked at the "Copy data into Galaxy?" option and made sure it wasn't ticked. Galaxy did copy the file :( I looked at the interface again, and see you have: > Copy data into Galaxy? > [check box] No > Normally data uploaded with this tool is copied into Galaxy's "files" > directory so any later changes to the data will not affect Galaxy. > However, this may not be desired (especially for large NGS > datasets), so use of this option will force Galaxy to always > read the data from its original path. You've got a tick box labelled "No" so it does the opposite to the question above it. That is a HORRIBLE interface. I can't be the first person to be tricked by it, can I? I would urge you to remove the work "No" and flip the meaning (and default) to match. But that could confuse existing admin users used to clicking from previous habit. How about making it a select parameter instead, with options "Copy files (default)" and "Link to files" which is very explicit? Likewise for the "Preserve directory structure?" setting, I'd remove the "No" caption and flip the setting, or just replace the backwards tick box with a select parameter offering "Preserve directory structure" and "Place in selected folder" (or something like that). Pretty please? How about if I write and test the patch? Peter

3 5