February 2011 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] [galaxy-user] Peep view for history elements broken on IE?
by Leandro Hermida 22 Feb '11

22 Feb '11

Hi everyone, I posted a similar thread back in August http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-August/003224.html There are a couple web UI issues with Galaxy when using IE, like as mentioned in this thread in the history pane data items the link is missing and you cannot expand them to "peep" at what's there. -Leandro On Thu, Jan 27, 2011 at 6:07 PM, Kanwei Li <kanwei(a)gmail.com> wrote: > Weird.. the proxy shouldn't be relevant. Could you open the javascript > error console and see what the error is? > > On Thu, Jan 27, 2011 at 8:57 AM, Peter <peter(a)maubp.freeserve.co.uk> > wrote: > > On Wed, Jan 26, 2011 at 7:49 PM, Kanwei Li <kanwei(a)gmail.com> wrote: > >> Hi Peter, > >> > >> Is this still an issue? The error you describe sounds like a jStore > >> problem but I tested it in IE7 and it works. > >> > >> Thanks, > >> > >> Kanwei > > > > Hi Kanwei, > > > > I just tried the public Galaxy at http://usegalaxy.org aka > > http://main.g2.bx.psu.edu/ and also ttp://test.g2.bx.psu.edu/ which > > says it tracks the galaxy-central repo, and for both the history peep > > links don't work. This is using Internet Explorer version 7.0.5730.13, > > which I believe is the current release (from back in 2007), from two > > different Windows XP machines at our institute (which are accessing > > the internet via a proxy - perhaps relevant?). > > > > Peter > > > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user >

3 9

Import files from local file system
by Steve Taylor 22 Feb '11

22 Feb '11

Hi, I noticed the recent update to Galaxy that allowed users to put data on the galaxy FTP server and then import it into their history. Would it be possible to use a similar mechanism to import data from the galaxy host file system but not use an FTP daemon? In our case we we would like to mount a volume to the galaxy file server and allow people to cp or mv data into it and then be able to import the data into their history. For example, In the context of the example at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP could you set up a directory called '/home/nate/galaxy_dist/database/localfs/' instead of '/home/nate/galaxy_dist/database/ftp/' which contained email named directories and the files in these in these would be available to import via the upload UI? Or is it all tied into FTP? Thanks for any advice, Steve

5 10

sorting bam
by vasu punj 21 Feb '11

21 Feb '11

Can I sort (chr) bam file in Glaxay or I have to go back to samtools please?

2 1

undelete data library
by Davide Cittaro 21 Feb '11

21 Feb '11

Hi there, is there a way to undelete a folder within a data library? I've deleted one (with some files) by mistake :-( d /* Davide Cittaro, PhD Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

2 6

Re: [galaxy-dev] [galaxy-user] operating on, and getting at, large files in galaxy...
by Nate Coraor 21 Feb '11

21 Feb '11

Nick Schurch wrote: > Hi all, > > I've recently encountered a few problems when trying to use Galaxy which are > really driving me away from using it as a bioinformatics platform for NGS. I > was wonderinf if there are any simple solutions that I've missed... Hi Nick, We've had some internal discussion and proposed some solutions which would hopefully make Galaxy more useful for your environment. > Firstly, It seems that while there are a few solutions for getting large > files (a few GB) into a local install of galaxy without going through HTTP, > many tools that operate on these files produces multiple, uncompressed large > files which quickly eat up the disk allocation. This is particularly > significant in a workflow that has multiple processing steps which each > leave behind a large file. With no way to compress or archive files produced > by intermediate steps in a workflow, and no desire to delete them since I > may need to go back to them and they can take hours to re-run, the only two > remaining options seem to be to save them and then delete them. We've dealt with this locally by implementing compression in the underlying filesystem (ZFS), but this requires a fileserver that runs Solaris (or a derivative) or FreeBSD. Btrfs also supports compression but I would be a bit more wary of losing my data with btrfs since it is less mature and can't recover corrupted filesystems. Fusecompress would also be an option. We would strongly recommend performing regular backups regardless of any filesystem-level choice. Unfortunately this is a tricky problem to solve within Galaxy itself. While some tools can operate on compressed files directly, many cannot and so compressing all outputs could prove to be very CPU intensive and a waste of time if the next step will have to decompress the file. There has been some discussion of how to implement transparent compression and other complex underlying data management directly in Galaxy, but any work on it is not likely to commence soon. > And this brings me to the second problem. Getting large files out of Galaxy. > The only way to save large files from Galaxy (that I can see) is the save > icon, which downloads the file via http. This take *ages* for a large file > and also causes big headaches for my firefox browser. I've taken a quick > peek at the Galaxy file system to see if I could just copy a file, but its > almost completely indecipherable if you want to find out what file in the > file system corresponds to a file saved from a tool. Is there some way to > get the location of a particular file on the galaxy file system, that I can > just copy? This is certainly something we can implement and will be working on fairly soon. There have been quite a few requests to integrate more tightly with environments where Galaxy users exist as system users. There's an issue in our tracker which you can follow here: https://bitbucket.org/galaxy/galaxy-central/issue/106/ --nate > > -- > Cheers, > > Nick Schurch > > Data Analysis Group (The Barton Group), > School of Life Sciences, > University of Dundee, > Dow St, > Dundee, > DD1 5EH, > Scotland, > UK > > Tel: +44 1382 388707 > Fax: +44 1382 345 893 > > > > -- > Cheers, > > Nick Schurch > > Data Analysis Group (The Barton Group), > School of Life Sciences, > University of Dundee, > Dow St, > Dundee, > DD1 5EH, > Scotland, > UK > > Tel: +44 1382 388707 > Fax: +44 1382 345 893 > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

2 1

listing the same tool twice in tool_conf.xml
by Branden Timm 21 Feb '11

21 Feb '11

Hi All, My group is trying to create some custom sections in the Tools pane, and have run into an issue. We have a custom section, within which there are three different labels and several tools under each label. We are trying list the same tool under two different labels within the same section. However, the second occurrence of the tool is not displayed. Is it possible to list the same tool twice under the same section? -- Branden Timm btimm(a)glbrc.wisc.edu

3 2

Re: [galaxy-dev] [galaxy-user] Galaxy does not find my executables
by Nate Coraor 21 Feb '11

21 Feb '11

Hi Florent, et. al., I've moved the discussion over to galaxy-dev since it pertains to local instances. Florent Angly wrote: > > Indeed, this workaround works! Thank you. > I added these lines to the beginning of my run.sh: > PATH="/opt/local/bin:/opt/local/sbin:$PATH" > export $PATH > > Now, Galaxy can find Grinder, Velvet, etc.... > > Could Galaxy detect and use ~/.profile automatically if it exists? > Alternatively, maybe there should be a place in the configuration > file universe_wsgi.ini to specify which file to read environment > variables from? Is it worth filing an enhancement request in the bug > tracker? Unfortunately, this is really more of an issue with the way shells are designed. Galaxy can't make too many assumptions about how you want the environment to be set up, especially since it's run under a variety of shells. What's going to happen is dependent entirely upon how the parent shell to Galaxy is started. By default if invoked in non-login, non-interactive mode (which would likely be the case for most managed startup methods like launchd, rc scripts, etc.), bash will not read *any* startup files. This behavior can be changed by specifying the path to a file to be sourced in the $BASH_ENV environment variable in the calling environment. However, when bash is called as 'sh', even $BASH_ENV is not available. For more details, see the "INVOCATION" section of the bash(1) man page. Since the public site runs on Solaris, we solve this by setting $PATH and any other relevant variables within SMF. Similar methods should work (setting the $PATH in an init script or in the LaunchDaemon plist) for other startup methods. Although modifying run.sh as mentioned does work, modifications could conflict with future changes from the upstream source repository. --nate P.S. I'll add a plug for zsh - its invocation is much less confusing and ~/.zshenv is *always* read regardless of invocation. > > Again, thanks for the help. > > Florent

3 5

Problems with run_functional_tests.sh and history?
by Peter 21 Feb '11

21 Feb '11

Hi all, I'm having trouble with the Galaxy functional tests. On our Linux server things seem to work, but I'm having trouble on my Mac. This affects multiple tests. For example, using a test for my own tool wrapper, on Linux my test passes. On my Mac, the tool is missing the required binary so I expect the test to fail with error level 127 and a message to stderr. However, I get the following cryptic error: $ ./run_functional_tests.sh -id tmhmm2 ... functional_tests.py INFO 2010-10-28 11:57:01,129 Embedded web server started functional_tests.py INFO 2010-10-28 11:57:01,185 Functional tests will be run against localhost:8980 nose.plugins.manager DEBUG 2010-10-28 11:57:01,210 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2010-10-28 11:57:01,215 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML TMHMM 2.0 ( tmhmm2 ) > Test-1 ... Error - <type 'exceptions.TypeError'>: 'NoneType' object is not callable URL: http://localhost:8980/history File '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.7.2-py2.5.egg/paste/exceptions/errormiddleware.py', line 144 in __call__ app_iter = self.application(environ, sr_checker) File '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.7.2-py2.5.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.7.2-py2.5.egg/paste/httpexceptions.py', line 636 in __call__ return self.application(environ, start_response) File '/Users/xxx/repositories/galaxy-central/lib/galaxy/web/framework/base.py', line 154 in __call__ return body( environ, start_response ) File '/Users/xxx/repositories/galaxy-central/lib/galaxy/web/framework/__init__.py', line 668 in render template.render_context( context ) File '/Users/xxx/repositories/galaxy-central/eggs/Mako-0.2.5-py2.5.egg/mako/template.py', line 146 in render_context runtime._render_context(self, self.callable_, context, *args, **kwargs) File '/Users/xxx/repositories/galaxy-central/eggs/Mako-0.2.5-py2.5.egg/mako/runtime.py', line 381 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/Users/xxx/repositories/galaxy-central/eggs/Mako-0.2.5-py2.5.egg/mako/runtime.py', line 414 in _exec_template callable_(context, *args, **kwargs) File '/Users/xxx/repositories/galaxy-central/database/compiled_templates/root/history.mako.py', line 55 in render_body __M_writer(u'\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n\n<html>\n\n<head>\n<title>') File '/Users/xxx/repositories/galaxy-central/lib/galaxy/web/framework/__init__.py', line 665 in write response_write( d.encode( 'utf-8' ) ) TypeError: 'NoneType' object is not callable CGI Variables ------------- CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html; */*' HTTP_ACCEPT_ENCODING: 'identity' HTTP_CONNECTION: 'close' HTTP_COOKIE: '$Version=1; galaxysession=eb142648ac45b770a711a07bbd8189c4a80d9868f9ceff96f0635391728a47dfe5161bca180a2aae; $Path="/"' HTTP_HOST: 'localhost:8980' PATH_INFO: '/history' REMOTE_ADDR: '127.0.0.1' REQUEST_METHOD: 'GET' SERVER_NAME: '127.0.0.1' SERVER_PORT: '8980' SERVER_PROTOCOL: 'HTTP/1.1' Configuration ------------- static_enabled: False use_translogger: False WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0xbfd2e10> paste.cookies: (<SimpleCookie: galaxysession='eb142648ac45b770a711a07bbd8189c4a80d9868f9ceff96f0635391728a47dfe5161bca180a2aae'>, '$Version=1; galaxysession=eb142648ac45b770a711a07bbd8189c4a80d9868f9ceff96f0635391728a47dfe5161bca180a2aae; $Path="/"') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0xbfd3110> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0xccf1f10> paste.recursive.forward: <paste.recursive.Forwarder from /> paste.recursive.include: <paste.recursive.Includer from /> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> paste.recursive.script_name: '' paste.throw_errors: True webob._parsed_query_vars: (MultiDict([]), '') wsgi process: 'Multithreaded' ------------------------------------------------------------ ERROR ====================================================================== ERROR: TMHMM 2.0 ( tmhmm2 ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 160, in test_tool self.do_it( td ) File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 24, in do_it self.new_history() File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 291, in new_history self.check_history_for_string('Your history is empty') File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 215, in check_history_for_string self.visit_page( "history" ) File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 1040, in visit_page tc.code( 200 ) File "/Users/xxx/repositories/galaxy-central/eggs/twill-0.9-py2.5.egg/twill/commands.py", line 133, in code should_be)) TwillAssertionError: code is 500 != 200 ---------------------------------------------------------------------- Ran 1 test in 0.824s FAILED (errors=1) functional_tests.py INFO 2010-10-28 11:57:02,079 Shutting down functional_tests.py INFO 2010-10-28 11:57:02,079 Shutting down embedded web server ... ############################################################################### Here is my suggested fix: diff -r 82743fcbf23c lib/galaxy/web/framework/__init__.py --- a/lib/galaxy/web/framework/__init__.py Thu Oct 28 11:41:18 2010 +0100 +++ b/lib/galaxy/web/framework/__init__.py Thu Oct 28 11:56:16 2010 +0100 @@ -660,6 +660,9 @@ ## return template.render( **data ) def render( environ, start_response ): response_write = start_response( self.response.wsgi_status(), self.response.wsgi_headeritems() ) + if response_write is None: + log.error( "Error: this request returned None from start_response: %s" % self.request.browser_url ) + return [] class StreamBuffer( object ): def write( self, d ): response_write( d.encode( 'utf-8' ) ) ############################################################################### With the patch, I get: $ ./run_functional_tests.sh -id tmhmm2 ... functional_tests.py INFO 2010-10-28 11:52:37,742 Embedded web server started functional_tests.py INFO 2010-10-28 11:52:37,800 Functional tests will be run against localhost:9383 nose.plugins.manager DEBUG 2010-10-28 11:52:37,825 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2010-10-28 11:52:37,829 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML TMHMM 2.0 ( tmhmm2 ) > Test-1 ... galaxy.web.framework ERROR 2010-10-28 11:52:38,685 Error: this request returned None from start_response: http://localhost:9383/history ERROR ====================================================================== ERROR: TMHMM 2.0 ( tmhmm2 ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 160, in test_tool self.do_it( td ) File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 24, in do_it self.new_history() File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 291, in new_history self.check_history_for_string('Your history is empty') File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 217, in check_history_for_string tc.find(subpatt) File "/Users/xxx/repositories/galaxy-central/eggs/twill-0.9-py2.5.egg/twill/commands.py", line 239, in find raise TwillAssertionError("no match to '%s'" % (what,)) TwillAssertionError: no match to 'Your' -------------------- >> begin captured logging << -------------------- galaxy.web.framework: ERROR: Error: this request returned None from start_response: http://localhost:9383/history --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 1 test in 0.821s FAILED (errors=1) functional_tests.py INFO 2010-10-28 11:52:38,691 Shutting down functional_tests.py INFO 2010-10-28 11:52:38,691 Shutting down embedded web server ... ############################################################################### I'm still puzzled about what is going wrong - this affects other Galaxy tests as well - not just mine. Is there something obvious wrong with the setup on this machine? Actually using Galaxy running on this Mac seems to work. Thanks, Peter

2 6

pbs_python vs drmaa
by Ryan Golhar 21 Feb '11

21 Feb '11

I've configured my local instance of galaxy to run with drmaa. Unfortunately, there is a problem that after some time, galaxy loses its ability to talk to Torque, hence Galaxy thinks long-running jobs have failed and its unable to submit new jobs. I have no idea how this stuff works underneath so I thought I'd try setting up pbs_python instead, but I get an error when I run the command given onthe Config/Cluster Wiki (galaxy_env)[galaxy@genomes galaxy-dist]$ LIBTORQUE=/campus/apps/amd64_linux26/packages/torque-2.4.8/lib/libtorque.so python scripts/scramble.py pbs_python fetch_one(): Using existing source, remove to download again: /home/galaxy/galaxy-dist/scripts/scramble/archives/pbs_python-4.1.0.tar.gz unpack_source(): Removing old build directory at: /home/galaxy/galaxy-dist/scripts/scramble/build/py2.6-linux-x86_64-ucs2/pbs_python unpack_source(): Unpacked to: /home/galaxy/galaxy-dist/scripts/scramble/build/py2.6-linux-x86_64-ucs2/pbs_python copy_build_script(): Using build script /home/galaxy/galaxy-dist/scripts/scramble/scripts/pbs_python.py run_scramble_script(): Beginning build run_scramble_script(): Executing in /home/galaxy/galaxy-dist/scripts/scramble/build/py2.6-linux-x86_64-ucs2/pbs_python: /home/galaxy/galaxy_env/bin/python -ES scramble.py Traceback (most recent call last): File "scramble.py", line 1, in <module> import os, sys, shutil ImportError: No module named shutil Traceback (most recent call last): File "scripts/scramble.py", line 44, in <module> egg.scramble() File "/home/galaxy/galaxy-dist/lib/galaxy/eggs/scramble.py", line 58, in scramble self.run_scramble_script() File "/home/galaxy/galaxy-dist/lib/galaxy/eggs/scramble.py", line 207, in run_scramble_script raise ScrambleFailure( self, "%s(): Egg build failed for %s %s" % ( sys._getframe().f_code.co_name, self.name, self.version ) ) galaxy.eggs.scramble.ScrambleFailure: run_scramble_script(): Egg build failed for pbs_python 4.1.0 (galaxy_env)[galaxy@genomes galaxy-dist]$ It looks like it can't find shutil. I'm also running under a virtualenv as indicated by the ProductionServer wiki page, and not sure this makes a difference or not. Does anyone know how to correct this?

2 1

question about galaxy gzip support in upload.py
by Ryan Golhar 19 Feb '11

19 Feb '11

In tools/data_source/upload.py, about line 216, galaxy attempts to uncompressed gzip'd files in their existing location. In my case, I'm importing files from the filesystem, and telling galaxy NOT to copy the data into Galaxy. The 'galaxy' user doesn't have write access to these files or folders. Galaxy complains about this when it tries to uncompress the files. Shouldn't galaxy uncompress the files in the database/tmp ($TEMP) folder instead?

1 0