February 2012 - galaxy-dev - lists.galaxyproject.org

uploads stuck in history
by Jorrit Boekel 08 Mar '12

08 Mar '12

Dear list, I have stumbled on some strange behaviour when uploading files to galaxy via the upload.py tool. At times, the upload seems to be stalled in history and is never actually performed, followed by a seemingly infinite history update (see log below). My system is Ubuntu 11.10 and runs Python 2.7.2. I find the behaviour in both my own modified galaxy install (based on galaxy-dist), and in a fresh clone from galaxy-central. I have tried to upload different files, and all seem to sometimes trigger the behaviour, but not all the time. A restart of galaxy sometimes sorts things out. Common for the debug messages is that it seems there is never a job id generated as in "galaxy.tools.actions.upload_common INFO 2012-02-22 10:06:36,186 tool upload1 created job id 2". Has anyone seen similar things or can it be a problem with my system? cheers, jorrit --------------Debug messa: galaxy.web.framework DEBUG 2012-02-22 09:47:43,730 Error: this request returned None from get_history(): http://localhost:8080/ 127.0.0.1 - - [22/Feb/2012:09:47:43 +0200] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:44 +0200] "GET /root/tool_menu HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:44 +0200] "GET /history HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:44 +0200] "POST /root/user_get_usage HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:47:50 +0200] "GET /tool_runner?tool_id=upload1 HTTP/1.1" 200 - "http://localhost:8080/root/tool_menu" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "POST /tool_runner/upload_async_create HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "GET /tool_runner/upload_async_message HTTP/1.1" 200 - "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "GET /history HTTP/1.1" 200 - "http://localhost:8080/tool_runner/upload_async_message" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:11 +0200] "POST /root/user_get_usage HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:13 +0200] "GET /tool_runner?tool_id=upload1 HTTP/1.1" 200 - "http://localhost:8080/root/tool_menu" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:15 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:19 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:23 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:27 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:31 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:35 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:39 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:43 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2" 127.0.0.1 - - [22/Feb/2012:09:48:47 +0200] "POST /root/history_item_updates HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"

6 10

Running SICER
by Vidya .H.K 07 Mar '12

07 Mar '12

Hi, I am trying to run SICER-peak calling tool in galaxy. I am not able run the analysis, as an error message " SICER is not available for the genome" is frequently flagged. Can any one suggest me a soultion for the same. Regards VIDYA

2 1

toolsectionlabel error (problem configuring toolshed hotplug?)
by Bossers, Alex 07 Mar '12

07 Mar '12

Somehow I seem to miss something when configuring our latest up-to-date galaxy_central version to use the galaxy toolshed. When as admin I want to install a tool I can browse the given 2 repositories and select view tools. When I select install to local galaxy it drops an error on 2 tools I tested thus far: --------------------------- Error Traceback: View as: Interactive | Text | XML (full) ⇝ AttributeError: 'ToolSectionLabel' object has no attribute 'name' URL: http://galaxy02:8080/galaxy/admin_toolshed/install_repository?tool_shed_url… Module weberror.evalexception.middleware:364 in respond view >> app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ view >> environ, self.app) Module paste.wsgilib:539 in intercept_output view >> app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ view >> return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ view >> return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ view >> body = method( trans, **kwargs ) Module galaxy.web.framework:173 in decorator view >> return func( self, trans, *args, **kwargs ) Module galaxy.web.controllers.admin_toolshed:322 in install_repository view >> tool_panel_section_select_field = build_tool_panel_section_select_field( trans ) Module galaxy.web.controllers.admin_toolshed:804 in build_tool_panel_section_select_field view >> options.append( ( tool_section.name, tool_section.id ) ) AttributeError: 'ToolSectionLabel' object has no attribute 'name' --------------------------- The steps I took: Updated the universe to load the extra shed_tool_conf.xml file. Set the toolpath to "../toolshedtools" in shed_tool_conf.xml Gave the directory read write permissions for galaxy user:dev_group Restarted galaxy. Any additional configs to make somewhere? In the original tool_conf.xml file? Alex

4 7

babel egg: epiphany locale setting fix
by Jorrit Boekel 07 Mar '12

07 Mar '12

Dear list, I encountered problems when using galaxy with ubuntu's epiphany browser. It seems that when using this browser, HTTP_ENVIRON's locale setting is sometimes 'en-us', but sometimes ' en-us'. That's a whitespace before the locale. I solved the problem by adding a simple .strip() in eggs/babel/core.py on line 763: lang = parts.pop(0).lower().strip() I assume the egg is not maintained by the galaxy team, so maybe I should forward this to Babel developers, but if anyone runs into the same problem, here's a solution. cheers, jorrit

2 1

Potential issue with eggs ?
by Michael Burrell (TOC) 07 Mar '12

07 Mar '12

Good evening all, I am seeing the following issue when attempting to start galaxy. Traceback (most recent call last): File "./scripts/paster.py", line 34, in <module> command.run() File "/home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run result = self.command() File "/home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command relative_to=base, global_conf=vars) File "/home/home/galaxy/software/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp **kw) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 224, in loadobj global_conf=global_conf) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 248, in loadcontext global_conf=global_conf) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 278, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 413, in get_context section) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 458, in _context_from_explicit value = import_string(found_expr) File "/home/home/galaxy/software/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 18, in import_string return pkg_resources.EntryPoint.parse("x="+s).load(False) File "/home/home/galaxy/software/galaxy-central/lib/pkg_resources.py", line 1954, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File "/home/home/galaxy/software/galaxy-central/lib/galaxy/web/buildapp.py", line 19, in <module> from galaxy import config, jobs, util, tools File "/home/home/galaxy/software/galaxy-central/lib/galaxy/jobs/__init__.py", line 4, in <module> from galaxy import util, model EOFError: EOF read where object expected Removing PID file paster.pid Would this suggest that the eggs are bad? What would be the best way to remedy the situation Thanks Michael.

3 2

Possible Galaxy memory leak?
by Joshua Gross 06 Mar '12

06 Mar '12

Hi all-- Our group is running a local instance of Galaxy on a Linux box. We have been noticing a memory issue, where if we leave the server running for a few days, soon all available memory (~8gb) will be used up to the point where the tools fail because they are unable to allocate new memory. While we haven't done any rigorous testing, this issue seems to be independent of the specific tools we run, and actually seems to still appear if we do not run any tools at all. The process "python" is what is taking up all the memory, and the issue is not instantaneous but happens over time. Restarting the server frees the memory immediately. Is this a known issue? If so, is there a known fix for it? Any insight on this would be much appreciated, as it is complicating our efforts to automate long work-flows. Thank you so much for your time Josh Gross, Western Washington University.

4 3

can the sample tracking system transfer (copy) data to a dataset library not in private Galaxy server data directory tree and then just link files?
by Leandro Hermida 06 Mar '12

06 Mar '12

Dear all, Is there a way with the Galaxy sample tracking system to have the dataset transfer feature copy data to a dataset library outside the private Galaxy server data directory tree and then the dataset library just link (like you can do with upload files feature when managing data libraries)? The reason for this is that it is common to have such huge data also available outside of Galaxy on the filesystem for bioinformatians/computational biologists to access directly without having to have two copies. regards, Leandro

2 1

Defining Job Runners Dynamically
by John Chilton 05 Mar '12

05 Mar '12

Hello All, I just issued a pull request that augments Galaxy to allow defining job runners dynamically at runtime (https://bitbucket.org/galaxy/galaxy-central/pull-request/12/dynamic-job-run…) Whether it makes the cut or not, I thought I would describe enhancements here in case anyone else would find it useful. There a couple use cases we hope this will help us address for our institution - one is dynamically switching queues based on user (we have a very nice shared memory resource that can only be used by researchers with NIH funding) and the other is inspecting input sizes to give more accurate max walltimes to pbs (a small number of cufflinks jobs for instance take over three days on our cluster but defining max walltimes in excess of that for all jobs could result in our queue sitting idle around our monthly downtimes). You might also imagine using this to dynamically switch queues entirely based on input sizes or parameters, or alter queue priorities based on the submitting user or input sizes/parameters. There are two steps to use this - you must add a line in universe.ini and define a function to compute the true job runner string in the new file lib/galaxy/jobs/rules.py. This first step is similar to what you would do to statically assign a tool to a particular job runner. If you would like to dynamically assign a job runner for cufflinks you would start by adding a line like one of the following to universe.ini cufflinks = dynamic:///python -or- cufflinks = dynamic:///python/compute_runner If you use the first form, a function called cufflinks must be defined in rules.py, adding the extra argument after python/ lets you specify a particular function by name (compute_runner in this example). This second option could let you assign job runners with the same function for multiple tools. The only other step is to define a python function in rules.py that produces a string corresponding to a valid job runner such as "local:///" or "pbs:///queue/-l walltime=48:00:00/". If the functions defined in this file take in arguments, these arguments should have names from the follow list: job_wrapper, user_email, app, job, tool, tool_id, job_id, user. The plumbing will map these arguments to the implied galaxy object. For instance, job_wrapper is the JobWrapper instance for the job that gets passed to the job runner, user_email is the user's email address or None, app is the main application configuration object used throughout the code base that can be used for instance to get values defined in universe.ini, job, tool, and user are model objects, and job_id and tool_id the relevant ids. If you are writing a function that routes a certain list of users to a particular queue or increases their priority, you will probably only need to take in one argument - user_email. However, if you are going to look at input file sizes you may want to take in an argument called job and use the following piece of code to find the input size for input named "input1" in the tool xml. inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) inp_data.update( [ ( da.name, da.dataset ) for da in job.input_library_datasets ] ) input1_file = inp_data[ "input1" ].file_name input1_size = os.path.getsize( input1_file ) This whole concept works for a couple of small tests on my local machine, but there are certain aspects of the job runner code that makes me feel there may be corner cases I am not seeing where this approach may not work - so your millage may vary. -John ------------------------------------------------ John Chilton Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 E-Mail: chilton(a)msi.umn.edu

3 2

snpEff: html report is not displaying after update
by Jennifer Jackson 05 Mar '12

05 Mar '12

Hello Raj, This tool is from the Tool Shed and is maintained by the tool author. To contact them about functionality or problems, go to the Tool Shed at http://toolshed.g2.bx.psu.edu/, locate the repository (search by 'snpEff'), click on the name, in the the top far right corner locate the pull down menu "Repository Actions". Select "Contact repository owner" to send them a message. There have been many updates to Galaxy recently and it is possible that one of these had an impact on this tool. I am going to send your email to galaxy-dev(a)bx.psu.edu, to see if anyone else has had problems with this tool since the last distribution update. You may be asked to provide more information (be as specific as you can): the hg pull #, any customizations, python version, etc. Regards, Jen Galaxy team On 2/14/12 8:28 PM, Praveen Raj Somarajan wrote: > Hi All, > > I updated galaxy recently to the latest version. Everything looks fine, > except snpEff report html view. It was displaying properly (all tables > and summary values) before the update, but the summary values are not > displaying after the update. A sample screen-shot is attached for your > reference. Could you please figure out this issue? > > When I ran the same on command line, the reports were generated > correctly. I assume, something (datatypes or preview) has changed by the > update. Please let me know the work around on this? > > Secondly, as we know, snpEff also generates a gene-wise annotation file > along with other results, but somehow we cannot access this file through > Galaxy. Though we see the link in the html report, but it seems the path > is broken. > > Let me know your suggestions. > > Best, > > Raj > > > ------------------------------------------------------------------------ > This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended > solely for the use of the addressee(s). If you are not the intended > recipient, please notify the sender by e-mail and delete the original > message. Further, you are not to copy, disclose, or distribute this > e-mail or its contents to any other person and any such actions that are > unlawful. This e-mail may contain viruses. Ocimum Biosolutions has taken > every reasonable precaution to minimize this risk, but is not liable for > any damage you may sustain as a result of any virus in this e-mail. You > should carry out your own virus checks before opening the e-mail or > attachment. > The information contained in this email and any attachments is > confidential and may be subject to copyright or other intellectual > property protection. If you are not the intended recipient, you are not > authorized to use or disclose this information, and we request that you > notify us by reply mail or telephone and delete the original message > from your mail system. > > OCIMUMBIO SOLUTIONS (P) LTD > > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/Support

2 1

Problem with python ./scripts/make_egg_packager.py
by Jose William Valero Velasquez 05 Mar '12

05 Mar '12

Hi, I'm writing you because i'm trying to install galaxy in a server running suse behind a proxy, so i have to build the eggs like if the server were offline, in the guide says that i should execute the command ./scripts/make_egg_packager.py but i get the following error, and i don't really understand whats happening: File "scripts/make_egg_packager.py", line 21, in <module> c = Crate() TypeError: __init__() takes at least 2 arguments (1 given) I'd really apreciate your help to figure out how to solve this issue. William Valero Universidad Nacional de Colombia.

2 1