- galaxy-dev - lists.galaxyproject.org

[hg] galaxy 3515: merge
by Greg Von Kuster 18 Mar '10

18 Mar '10

details: http://www.bx.psu.edu/hg/galaxy/rev/6fa986527398 changeset: 3515:6fa986527398 user: fubar: ross Lazarus at gmail period com date: Wed Mar 10 23:59:25 2010 -0500 description: merge

1 0

[hg] galaxy 3514: Fixes for downloading library archives - corre...
by Greg Von Kuster 18 Mar '10

18 Mar '10

details: http://www.bx.psu.edu/hg/galaxy/rev/06dcf56688ea changeset: 3514:06dcf56688ea user: fubar: ross Lazarus at gmail period com date: Wed Mar 10 23:58:35 2010 -0500 description: Fixes for downloading library archives - correct extensions help a lot and removing spaces from ldda.name helps too. diffstat: lib/galaxy/web/controllers/library_common.py | 15 +++++++++------ 1 files changed, 9 insertions(+), 6 deletions(-) diffs (63 lines): diff -r 87da8cd1f091 -r 06dcf56688ea lib/galaxy/web/controllers/library_common.py --- a/lib/galaxy/web/controllers/library_common.py Wed Mar 10 20:59:11 2010 -0500 +++ b/lib/galaxy/web/controllers/library_common.py Wed Mar 10 23:58:35 2010 -0500 @@ -1,4 +1,4 @@ -import os, os.path, shutil, urllib, StringIO, re, gzip, tempfile, shutil, zipfile, copy, glob +import os, os.path, shutil, urllib, StringIO, re, gzip, tempfile, shutil, zipfile, copy, glob, string from galaxy.web.base.controller import * from galaxy import util, jobs from galaxy.datatypes import sniff @@ -1258,7 +1258,6 @@ messagetype = 'error' else: ldda_ids = util.listify( ldda_ids ) - log.debug('## act on multiple got %s' % ldda_ids) if action == 'import_to_history': history = trans.get_history() if history is None: @@ -1312,6 +1311,8 @@ msg = "The selected datasets have been removed from this data library" else: error = False + killme = string.punctuation + string.whitespace + trantab = string.maketrans(killme,'_'*len(killme)) try: outext = 'zip' if action == 'zip': @@ -1325,10 +1326,10 @@ archive.add = lambda x, y: archive.write( x, y.encode('CP437') ) elif action == 'tgz': archive = util.streamball.StreamBall( 'w|gz' ) - outext = 'gz' + outext = 'tgz' elif action == 'tbz': archive = util.streamball.StreamBall( 'w|bz2' ) - outext = 'bz2' + outext = 'tbz2' except (OSError, zipfile.BadZipFile): error = True log.exception( "Unable to create archive for download" ) @@ -1356,12 +1357,13 @@ path = os.path.join( parent_folder.name, path ) parent_folder = parent_folder.parent path += ldda.name + path = path.translate(trantab) while path in seen: path += '_' seen.append( path ) + zpath = os.path.split(path)[-1] # comes as base_name/fname + outfname,zpathext = os.path.splitext(zpath) if is_composite: # need to add all the components from the extra_files_path to the zip - zpath = os.path.split(path)[-1] # comes as base_name/fname - outfname,zpathext = os.path.splitext(zpath) if zpathext == '': zpath = '%s.html' % zpath # fake the real nature of the html file try: @@ -1375,6 +1377,7 @@ flist = glob.glob(os.path.join(ldda.dataset.extra_files_path,'*.*')) # glob returns full paths for fpath in flist: efp,fname = os.path.split(fpath) + fname = fname.translate(trantab) try: archive.add( fpath,fname ) except IOError:

1 0

[hg] galaxy 3513: Remove old Lmap from datatypes_conf.xml.sample...
by Greg Von Kuster 18 Mar '10

18 Mar '10

details: http://www.bx.psu.edu/hg/galaxy/rev/87da8cd1f091 changeset: 3513:87da8cd1f091 user: fubar: ross Lazarus at gmail period com date: Wed Mar 10 20:59:11 2010 -0500 description: Remove old Lmap from datatypes_conf.xml.sample - now deprecated from genetics.py Causing buildbot to barf. diffstat: datatypes_conf.xml.sample | 1 - lib/galaxy/web/controllers/library_common.py | 3 ++- run_functional_tests.sh | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diffs (43 lines): diff -r 53ddb4b728f7 -r 87da8cd1f091 datatypes_conf.xml.sample --- a/datatypes_conf.xml.sample Wed Mar 10 19:48:28 2010 -0500 +++ b/datatypes_conf.xml.sample Wed Mar 10 20:59:11 2010 -0500 @@ -194,7 +194,6 @@  <datatype extension="gg" type="galaxy.datatypes.genetics:GenomeGraphs"/>  - <datatype extension="lmap" type="galaxy.datatypes.genetics:Lmap" display_in_upload="true"/> <datatype extension="malist" type="galaxy.datatypes.genetics:MAlist" display_in_upload="true"/>  <datatype extension="lped" type="galaxy.datatypes.genetics:Lped" display_in_upload="true"> diff -r 53ddb4b728f7 -r 87da8cd1f091 lib/galaxy/web/controllers/library_common.py --- a/lib/galaxy/web/controllers/library_common.py Wed Mar 10 19:48:28 2010 -0500 +++ b/lib/galaxy/web/controllers/library_common.py Wed Mar 10 20:59:11 2010 -0500 @@ -1111,7 +1111,7 @@ # is composite - must return a zip of contents and the html file itself - ugh - should be reversible at upload! # use act_on_multiple_datasets( self, trans, cntrller, library_id, ldda_ids='', **kwd ) since it does what we need kwd['do_action'] = 'zip' - return self.act_on_multiple_datasets( trans, cntrller, library_id, ldda_ids=id, **kwd ) + return self.act_on_multiple_datasets( trans, cntrller, library_id, ldda_ids=[id,], **kwd ) else: mime = trans.app.datatypes_registry.get_mimetype_by_extension( ldda.extension.lower() ) trans.response.set_content_type( mime ) @@ -1258,6 +1258,7 @@ messagetype = 'error' else: ldda_ids = util.listify( ldda_ids ) + log.debug('## act on multiple got %s' % ldda_ids) if action == 'import_to_history': history = trans.get_history() if history is None: diff -r 53ddb4b728f7 -r 87da8cd1f091 run_functional_tests.sh --- a/run_functional_tests.sh Wed Mar 10 19:48:28 2010 -0500 +++ b/run_functional_tests.sh Wed Mar 10 20:59:11 2010 -0500 @@ -1,7 +1,7 @@ #!/bin/sh # A good place to look for nose info: http://somethingaboutorange.com/mrl/projects/nose/ - +export PATH=/usr/local/bin:$PATH rm -f run_functional_tests.log if [ ! $1 ]; then

1 0

[hg] galaxy 3512: Reverted missing python version check for zipf...
by Greg Von Kuster 18 Mar '10

18 Mar '10

details: http://www.bx.psu.edu/hg/galaxy/rev/53ddb4b728f7 changeset: 3512:53ddb4b728f7 user: fubar: ross Lazarus at gmail period com date: Wed Mar 10 19:48:28 2010 -0500 description: Reverted missing python version check for zipfile setup Changed download archive names for both libraries and for history items so they're related to the content - uses data.name rather than one fixed string for all diffstat: lib/galaxy/web/controllers/dataset.py | 25 ++++++++++++++++--------- lib/galaxy/web/controllers/library_common.py | 13 ++++++++----- 2 files changed, 24 insertions(+), 14 deletions(-) diffs (153 lines): diff -r 2af472aa0844 -r 53ddb4b728f7 lib/galaxy/web/controllers/dataset.py --- a/lib/galaxy/web/controllers/dataset.py Wed Mar 10 17:10:43 2010 -0500 +++ b/lib/galaxy/web/controllers/dataset.py Wed Mar 10 19:48:28 2010 -0500 @@ -1,4 +1,4 @@ -import logging, os, string, shutil, re, socket, mimetypes, smtplib, urllib, tempfile, zipfile, glob +import logging, os, string, shutil, re, socket, mimetypes, smtplib, urllib, tempfile, zipfile, glob, sys from galaxy.web.base.controller import * from galaxy.web.framework.helpers import time_ago, iff, grids @@ -11,6 +11,11 @@ pkg_resources.require( "Paste" ) import paste.httpexceptions +if sys.version_info[:2] < ( 2, 6 ): + zipfile.BadZipFile = zipfile.error +if sys.version_info[:2] < ( 2, 5 ): + zipfile.LargeZipFile = zipfile.error + tmpd = tempfile.mkdtemp() comptypes=[] ziptype = '32' @@ -204,6 +209,9 @@ def archive_composite_dataset( self, trans, data=None, **kwd ): # save a composite object into a compressed archive for downloading params = util.Params( kwd ) + valid_chars = '.,^_-()[]0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' + outfname = data.name[0:150] + outfname = ''.join(c in valid_chars and c or '_' for c in outfname) if (params.do_action == None): params.do_action = 'zip' # default msg = util.restore_text( params.get( 'msg', '' ) ) @@ -230,7 +238,7 @@ except (OSError, zipfile.BadZipFile): error = True log.exception( "Unable to create archive for download" ) - msg = "Unable to create archive for %s for download, please report this error" % data.name + msg = "Unable to create archive for %s for download, please report this error" % outfname messagetype = 'error' if not error: current_user_roles = trans.get_current_user_roles() @@ -239,7 +247,7 @@ fname = os.path.split(path)[-1] basename = data.metadata.base_name efp = data.extra_files_path - htmlname = os.path.splitext(data.name)[0] + htmlname = os.path.splitext(outfname)[0] if not htmlname.endswith(ext): htmlname = '%s_%s' % (htmlname,ext) archname = '%s.html' % htmlname # fake the real nature of the html file @@ -276,14 +284,14 @@ messagetype = 'error' if not error: trans.response.set_content_type( "application/x-zip-compressed" ) - trans.response.headers[ "Content-Disposition" ] = "attachment; filename=GalaxyCompositeObject.zip" + trans.response.headers[ "Content-Disposition" ] = "attachment; filename=%s.zip" % outfname return tmpfh else: trans.response.set_content_type( "application/x-tar" ) outext = 'tgz' if params.do_action == 'tbz': outext = 'tbz' - trans.response.headers[ "Content-Disposition" ] = "attachment; filename=GalaxyLibraryFiles.%s" % outext + trans.response.headers[ "Content-Disposition" ] = "attachment; filename=%s.%s" % (outfname,outext) archive.wsgi_status = trans.response.wsgi_status() archive.wsgi_headeritems = trans.response.wsgi_headeritems() return archive.stream @@ -294,7 +302,8 @@ @web.expose def display(self, trans, dataset_id=None, preview=False, filename=None, to_ext=None, **kwd): """Catches the dataset id and displays file contents as directed""" - + composite_extensions = trans.app.datatypes_registry.get_composite_extensions( ) + composite_extensions.append('html') # for archiving composite datatypes # DEPRECATION: We still support unencoded ids for backward compatibility try: dataset_id = int( dataset_id ) @@ -329,8 +338,6 @@ trans.log_event( "Display dataset id: %s" % str( dataset_id ) ) if to_ext: # Saving the file - composite_extensions = trans.app.datatypes_registry.get_composite_extensions( ) - composite_extensions.append('html') if data.ext in composite_extensions: return self.archive_composite_dataset( trans, data, **kwd ) else: @@ -340,7 +347,7 @@ valid_chars = '.,^_-()[]0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' fname = data.name fname = ''.join(c in valid_chars and c or '_' for c in fname)[0:150] - trans.response.headers["Content-Disposition"] = "attachment; filename=GalaxyHistoryItem-%s-[%s]%s" % (data.hid, fname, to_ext) + trans.response.headers["Content-Disposition"] = "attachment; filename=Galaxy%s-[%s]%s" % (data.hid, fname, to_ext) return open( data.file_name ) if os.path.exists( data.file_name ): max_peek_size = 1000000 # 1 MB diff -r 2af472aa0844 -r 53ddb4b728f7 lib/galaxy/web/controllers/library_common.py --- a/lib/galaxy/web/controllers/library_common.py Wed Mar 10 17:10:43 2010 -0500 +++ b/lib/galaxy/web/controllers/library_common.py Wed Mar 10 19:48:28 2010 -0500 @@ -1312,6 +1312,7 @@ else: error = False try: + outext = 'zip' if action == 'zip': # Can't use mkstemp - the file must not exist first tmpd = tempfile.mkdtemp() @@ -1323,8 +1324,10 @@ archive.add = lambda x, y: archive.write( x, y.encode('CP437') ) elif action == 'tgz': archive = util.streamball.StreamBall( 'w|gz' ) + outext = 'gz' elif action == 'tbz': archive = util.streamball.StreamBall( 'w|bz2' ) + outext = 'bz2' except (OSError, zipfile.BadZipFile): error = True log.exception( "Unable to create archive for download" ) @@ -1357,11 +1360,11 @@ seen.append( path ) if is_composite: # need to add all the components from the extra_files_path to the zip zpath = os.path.split(path)[-1] # comes as base_name/fname - zpathext = os.path.splitext(zpath)[-1] + outfname,zpathext = os.path.splitext(zpath) if zpathext == '': zpath = '%s.html' % zpath # fake the real nature of the html file try: - archive.add(ldda.dataset.file_name,zpath) + archive.add(ldda.dataset.file_name,zpath) # add the primary of a composite set except IOError: error = True log.exception( "Unable to add composite parent %s to temporary library download archive" % ldda.dataset.file_name) @@ -1375,7 +1378,7 @@ archive.add( fpath,fname ) except IOError: error = True - log.exception( "Unable to add %s to temporary library download archive" % fname) + log.exception( "Unable to add %s to temporary library download archive %s" % (fname,outfname)) msg = "Unable to create archive for download, please report this error" messagetype = 'error' continue @@ -1402,11 +1405,11 @@ messagetype = 'error' if not error: trans.response.set_content_type( "application/x-zip-compressed" ) - trans.response.headers[ "Content-Disposition" ] = "attachment; filename=GalaxyLibraryFiles.%s" % action + trans.response.headers[ "Content-Disposition" ] = "attachment; filename=%s.%s" % (outfname,outext) return tmpfh else: trans.response.set_content_type( "application/x-tar" ) - trans.response.headers[ "Content-Disposition" ] = "attachment; filename=GalaxyLibraryFiles.%s" % action + trans.response.headers[ "Content-Disposition" ] = "attachment; filename=%s.%s" % (outfname,outext) archive.wsgi_status = trans.response.wsgi_status() archive.wsgi_headeritems = trans.response.wsgi_headeritems() return archive.stream

1 0

Balloon tooltips
by Assaf Gordon 17 Mar '10

17 Mar '10

Hi, I was wondering if there's an easy way to disable the pop-up balloon tooltips (on the save, rerun, view, delete icons) ? The new beautiful icons (or the old text-label) are informative enough - and the constantly appearing/disappearing black tooltip balloons (when you move the mouse over datasets) is a bit annoying. Alternatively, is it possible to add a short delay before showing the tooltip (like in all regular GUI client-side applications) ? Thanks, -gordon

2 1

Feature request: highlight currently displayed history item.
by Pieter Neerincx 17 Mar '10

17 Mar '10

Hi Galaxy team, When working with histories with many items I sometimes get a little lost: which is the history item whose content is displayed in the middle section of my Galaxy window? (The output from different tools usually is different enough to prevent confusion, but when you process multiple samples in the same history it can get confusing.) Would it be possible to highlight the history item whose content is displayed in the center of a Galaxy window. Some other visual clue in middle section of a Galaxy window like a small heading or title bar with the name/tag of the displayed history item would probably also do the trick.... Cheers, Pi --------------------------------------------------------------- Biomolecular Mass Spectrometry & Proteomics group Utrecht University Visiting address: H.R. Kruyt building room O607 Padualaan 8 3584 CH Utrecht The Netherlands Mail address: P.O. box 80.082 3508 TB Utrecht The Netherlands phone: +31 6 143 66 783 email: pieter.neerincx(a)gmail.com skype: pieter.online ---------------------------------------------------------------

2 1

Feature request: conf.d
by Pieter Neerincx 17 Mar '10

17 Mar '10

Hi Galaxy team, As my local instance grows it becomes more difficult each time to upgrade. I now use diff on the various config files to see what changes I should lift over to the new install. (I don't update, but leave my current setup untouched as a backup, create a new install and if that works switch to the new setup using a symlink.) Would it be possible to use a special directory with config files for site specific customizations? I'm thinking of something like Apache's "conf.d" directory or good ol' SRS's "sites" directory. This would allow us to leave the original Galaxy tools + datatypes config files untouched and list our local additions/mods in separate config files making upgrades a bit easier... Cheers, Pi --------------------------------------------------------------- Biomolecular Mass Spectrometry & Proteomics group Utrecht University Visiting address: H.R. Kruyt building room O607 Padualaan 8 3584 CH Utrecht The Netherlands Mail address: P.O. box 80.082 3508 TB Utrecht The Netherlands phone: +31 6 143 66 783 email: pieter.neerincx(a)gmail.com skype: pieter.online ---------------------------------------------------------------

4 5

can no longer import data libraries into history
by Edward Kirton 16 Mar '10

16 Mar '10

please help! our install of galaxy is completely unable to load any data into libraries now! URL: http://galaxy.jgi-psf.org/library_common/act_on_multiple_datasets?library_i… File '/house/groupdirs/mep/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/house/groupdirs/mep/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/house/groupdirs/mep/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/house/groupdirs/mep/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/house/groupdirs/mep/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/house/groupdirs/mep/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 125 in __call__ body = method( trans, **kwargs ) File '/house/groupdirs/mep/galaxy/galaxy-dist/lib/galaxy/web/controllers/library_common.py', line 1384 in act_on_multiple_datasets archive.add( ldda.dataset.file_name, path ) UnboundLocalError: local variable 'archive' referenced before assignment i tried adding a new library to see if i could see it but i get an error symlinking or copying from filesystem paths: Miscellaneous information: Traceback (most recent call last): File "/house/groupdirs/mep/galaxy/galaxy-dist/tools/data_source/upload.py", line 326, in __main__() File "/house/groupdirs/mep/galaxy/galaxy-dist/tools/data_source/upload.py", line 318, in __main__ a error I could upload, but I still get the first error Thanks in advance for any assistance! Ed Kirton

4 4

create multiple historie items
by vill 16 Mar '10

16 Mar '10

I have a perl script that splits a text file with genomic information (forward strand and reverse strand) in two separate files. One file for the forward strand and one file for the reverse strand. I have written a wrapper so that Galaxy can execute this script. When my script outputs one file this files is listed as a history item. When I let my script output the two files only one history item gets created. What I need to now is what extra variables do I need to output from the .xml file so that galaxy finds the two output files and puts both files in as a history item I have been looking at the "interval2maf .py and .xml" I just can't figure out how this works, it seems that the history_id is important and the species is a list array and for every species a file gets created in the databas/tmp directory and when the file is created the tmp dir is deleted. Adding and extra file from the .py into this tmp dir does not end up in the history items, renaming the files seems to end up with no history items at all. Cheers, //Michel

11 23

Galaxy install error: NoSuchTableError: migrate_version
by Patrick T. 16 Mar '10

16 Mar '10

Hi, I've tried to install galaxy on my Ubuntu Linux amd64 2.6.27-11-generic, Python 2.5.2 (r252:60911), >>> sqlalchemy.__version__ '0.5.8', I ran $ hg clone http://bitbucket.org/galaxy/galaxy-central/ 5 days ago And I got this error after calling % sh run.sh Traceback (most recent call last): File "/home/tan/galaxy_dist/lib/galaxy/web/buildapp.py", line 61, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/home/tan/galaxy_dist/lib/galaxy/app.py", line 28, in __init__ create_or_verify_database( db_url, self.config.database_engine_options ) File "/home/tan/galaxy_dist/lib/galaxy/model/migrate/check.py", line 66, in create_or_verify_database db_schema = schema.ControlledSchema( engine, migrate_repository ) File "/home/tan/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/schema.py", line 24, in __init__ self._load() File "/home/tan/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.5.egg/migrate/versioning/schema.py", line 36, in _load self.table = Table(tname, self.meta, autoload=True) File "/var/lib/python-support/python2.5/sqlalchemy/schema.py", line 110, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/var/lib/python-support/python2.5/sqlalchemy/schema.py", line 226, in __init__ _bind_or_error(metadata).reflecttable(self, include_columns=include_columns) File "/var/lib/python-support/python2.5/sqlalchemy/engine/base.py", line 1275, in reflecttable self.dialect.reflecttable(conn, table, include_columns) File "/var/lib/python-support/python2.5/sqlalchemy/databases/sqlite.py", line 343, in reflecttable raise exceptions.NoSuchTableError(table.name) NoSuchTableError: migrate_version /var/lib/python-support/python2.5/sqlalchemy/__init__ says ver 0.4.6 Thanks, Patrick __________________________________________________________________ Connect with friends from any web browser - no download required. Try the new Yahoo! Canada Messenger for the Web BETA at http://ca.messenger.yahoo.com/webmessengerpromo.php

3 4