January 2010 - galaxy-dev - lists.galaxyproject.org

[hg] galaxy 3287: Fixes for GeneTrack. TODO: HDF indexes are han...
by Greg Von Kuster 28 Jan '10

28 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/0126990119e9 changeset: 3287:0126990119e9 user: Dan Blankenberg <dan(a)bx.psu.edu> date: Wed Jan 27 16:33:49 2010 -0500 description: Fixes for GeneTrack. TODO: HDF indexes are handled in a fragile way: GeneTrack should accept an explicit filename for the index instead of using dataset_name.hdf. diffstat: eggs.ini | 4 ++-- lib/galaxy/datatypes/tracks.py | 2 +- tools/visualization/genetrack_indexer.py | 15 +++++++++++++-- tools/visualization/genetrack_indexer.xml | 2 +- tools/visualization/genetrack_peak_prediction.xml | 2 +- 5 files changed, 18 insertions(+), 7 deletions(-) diffs (86 lines): diff -r 57e5f8e4d341 -r 0126990119e9 eggs.ini --- a/eggs.ini Wed Jan 27 16:09:14 2010 -0500 +++ b/eggs.ini Wed Jan 27 16:33:49 2010 -0500 @@ -61,7 +61,7 @@ MySQL_python = _5.0.67_static python_lzo = _static bx_python = _dev_3b9d30e47619 -GeneTrack = _dev_e380f21c704218622155b9d230a44b3c9c452524 +GeneTrack = _dev_86867fc7bf46680cfafbb0749be8410b749c5593 SQLAlchemy = _dev_r6498 ; nose = .dev_r7156749efc58 @@ -84,7 +84,7 @@ decorator = http://pypi.python.org/packages/source/d/decorator/decorator-3.1.2.tar.gz docutils = http://downloads.sourceforge.net/docutils/docutils-0.4.tar.gz elementtree = http://effbot.org/downloads/elementtree-1.2.6-20050316.tar.gz -GeneTrack = http://github.com/ialbert/genetrack-central/tarball/e380f21c704218622155b9d… +GeneTrack = http://github.com/ialbert/genetrack-central/tarball/86867fc7bf46680cfafbb07… lrucache = http://evan.prodromou.name/lrucache/lrucache-0.2.tar.gz Mako = http://www.makotemplates.org/downloads/Mako-0.2.5.tar.gz nose = http://pypi.python.org/packages/source/n/nose/nose-0.11.1.tar.gz diff -r 57e5f8e4d341 -r 0126990119e9 lib/galaxy/datatypes/tracks.py --- a/lib/galaxy/datatypes/tracks.py Wed Jan 27 16:09:14 2010 -0500 +++ b/lib/galaxy/datatypes/tracks.py Wed Jan 27 16:33:49 2010 -0500 @@ -30,7 +30,7 @@ if hda.has_data: # Get the disk file name and data id file_name = hda.dataset.get_file_name() - data_id = hda.dataset.id + data_id = hda.id # Make it secure hashkey = hmac_new( app.config.tool_secret, file_name ) encoded = binascii.hexlify( file_name ) diff -r 57e5f8e4d341 -r 0126990119e9 tools/visualization/genetrack_indexer.py --- a/tools/visualization/genetrack_indexer.py Wed Jan 27 16:09:14 2010 -0500 +++ b/tools/visualization/genetrack_indexer.py Wed Jan 27 16:33:49 2010 -0500 @@ -6,7 +6,7 @@ usage: %prog input output shift """ -import sys +import sys, shutil, os from galaxy import eggs import pkg_resources pkg_resources.require( "GeneTrack" ) @@ -36,4 +36,15 @@ sys.exit(-1) else: tabs2genetrack.transform(inpname=options.inpname, outname=options.outname,\ - format=options.format, shift=options.shift, index=options.index) + format=options.format, shift=options.shift, index=options.index, options=options) + + #HACK ALERT + #output created in job working directory has a different name than the final destination. + #GeneTrack uses dataset_name.hdf for indexes + #This will fix the different name problem, but this is not a secure solution + #This is Exceptionally Fragile as any change to the way that temporary output names are created could break + #GeneTrack should allow explicit definition of the index file, not filename.hdf + if options.workdir and os.path.isdir( options.workdir ): + for filename in os.listdir( options.workdir ): + if filename.endswith( '.dat.hdf' ) and filename.startswith( 'galaxy_dataset_' ): + shutil.move( os.path.join( options.workdir, filename ), os.path.join( options.workdir, filename[ len( 'galaxy_' ): ] ) ) diff -r 57e5f8e4d341 -r 0126990119e9 tools/visualization/genetrack_indexer.xml --- a/tools/visualization/genetrack_indexer.xml Wed Jan 27 16:09:14 2010 -0500 +++ b/tools/visualization/genetrack_indexer.xml Wed Jan 27 16:33:49 2010 -0500 @@ -3,7 +3,7 @@ <description> - creates a visualization from a bed file</description> <command interpreter="python"> - genetrack_indexer.py -i $input -o $output -s $shift -v 0 -f BED -x + genetrack_indexer.py -i $input -o $output -s $shift -v 0 -f BED -x -w $output.files_path </command> <inputs> diff -r 57e5f8e4d341 -r 0126990119e9 tools/visualization/genetrack_peak_prediction.xml --- a/tools/visualization/genetrack_peak_prediction.xml Wed Jan 27 16:09:14 2010 -0500 +++ b/tools/visualization/genetrack_peak_prediction.xml Wed Jan 27 16:33:49 2010 -0500 @@ -3,7 +3,7 @@ <description> - generates peak predictions from an index file</description> <command interpreter="python"> - genetrack_peak_prediction.py -i $input -o $output --level=$level --sigma=$sigma --mode=$mode --exclusion=$exclusion --strand=$strand -v 0 + genetrack_peak_prediction.py -i $input -o $output --level=$level --sigma=$sigma --mode=$mode --exclusion=$exclusion --strand=$strand -v 0 -w $input.extra_files_path </command> <inputs>

1 0

[hg] galaxy 3286: Fix bug when showing all histories in page's h...
by Greg Von Kuster 28 Jan '10

28 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/57e5f8e4d341 changeset: 3286:57e5f8e4d341 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Wed Jan 27 16:09:14 2010 -0500 description: Fix bug when showing all histories in page's history selection grid. diffstat: templates/page/select_histories_grid_async.mako | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (16 lines): diff -r d1bc3b0535d0 -r 57e5f8e4d341 templates/page/select_histories_grid_async.mako --- a/templates/page/select_histories_grid_async.mako Wed Jan 27 15:50:21 2010 -0500 +++ b/templates/page/select_histories_grid_async.mako Wed Jan 27 16:09:14 2010 -0500 @@ -1,8 +1,8 @@ <%namespace file="../grid_base.mako" import="*" /> ## Always show item checkboxes so that users can select histories. -${render_grid_table_body_contents(show_item_checkboxes=True)} +${render_grid_table_body_contents( grid, show_item_checkboxes=True )} ***** ${num_pages} ***** -${render_grid_message()} \ No newline at end of file +${render_grid_message( grid )} \ No newline at end of file

1 0

[hg] galaxy 3285: Merge.
by Greg Von Kuster 28 Jan '10

28 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/d1bc3b0535d0 changeset: 3285:d1bc3b0535d0 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Wed Jan 27 15:50:21 2010 -0500 description: Merge. diffstat: dist-eggs.ini | 24 +++--- lib/galaxy/config.py | 1 + lib/galaxy/security/__init__.py | 6 +- lib/galaxy/util/heartbeat.py | 86 +++++++++++++++++++++++++++- lib/galaxy/visualization/tracks/data/bam.py | 29 +++++++- lib/galaxy/web/controllers/library_common.py | 14 +++- lib/galaxy/web/framework/base.py | 4 + 7 files changed, 141 insertions(+), 23 deletions(-) diffs (280 lines): diff -r acf985b23928 -r d1bc3b0535d0 dist-eggs.ini --- a/dist-eggs.ini Wed Jan 27 15:49:50 2010 -0500 +++ b/dist-eggs.ini Wed Jan 27 15:50:21 2010 -0500 @@ -7,18 +7,18 @@ ; [hosts] -py2.4-linux-i686-ucs2 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.4 -py2.4-linux-i686-ucs4 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.4 -py2.5-linux-i686-ucs2 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.5 -py2.5-linux-i686-ucs4 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.5 -py2.6-linux-i686-ucs2 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.6 -py2.6-linux-i686-ucs4 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.6 -py2.4-linux-x86_64-ucs2 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.4 -py2.4-linux-x86_64-ucs4 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.4 -py2.5-linux-x86_64-ucs2 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.5 -py2.5-linux-x86_64-ucs4 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.5 -py2.6-linux-x86_64-ucs2 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.6 -py2.6-linux-x86_64-ucs4 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.6 +py2.4-linux-i686-ucs2 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.4 +py2.4-linux-i686-ucs4 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.4 +py2.5-linux-i686-ucs2 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.5 +py2.5-linux-i686-ucs4 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.5 +py2.6-linux-i686-ucs2 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.6 +py2.6-linux-i686-ucs4 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.6 +py2.4-linux-x86_64-ucs2 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.4 +py2.4-linux-x86_64-ucs4 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.4 +py2.5-linux-x86_64-ucs2 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.5 +py2.5-linux-x86_64-ucs4 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.5 +py2.6-linux-x86_64-ucs2 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.6 +py2.6-linux-x86_64-ucs4 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.6 py2.4-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.4 py2.5-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.5 py2.6-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.6 diff -r acf985b23928 -r d1bc3b0535d0 lib/galaxy/config.py --- a/lib/galaxy/config.py Wed Jan 27 15:49:50 2010 -0500 +++ b/lib/galaxy/config.py Wed Jan 27 15:50:21 2010 -0500 @@ -97,6 +97,7 @@ raise ConfigurationError( "user_library_import_dir specified in config (%s) does not exist" % self.user_library_import_dir ) self.allow_library_path_paste = kwargs.get( 'allow_library_path_paste', False ) # Configuration options for taking advantage of nginx features + self.apache_xsendfile = kwargs.get( 'apache_xsendfile', False ) self.nginx_x_accel_redirect_base = kwargs.get( 'nginx_x_accel_redirect_base', False ) self.nginx_upload_store = kwargs.get( 'nginx_upload_store', False ) self.nginx_upload_path = kwargs.get( 'nginx_upload_path', False ) diff -r acf985b23928 -r d1bc3b0535d0 lib/galaxy/security/__init__.py --- a/lib/galaxy/security/__init__.py Wed Jan 27 15:49:50 2010 -0500 +++ b/lib/galaxy/security/__init__.py Wed Jan 27 15:50:21 2010 -0500 @@ -138,13 +138,15 @@ intermed.sort() return map( operator.getitem, intermed, ( -1, ) * len( intermed ) ) roles = set() - # If a library has roles associated with the LIBRARY_ACCESS permission, we need to start with them. + # If item has roles associated with the access permission, we need to start with them. access_roles = item.get_access_roles( trans ) for role in access_roles: roles.add( role ) # Each role potentially has users. We need to find all roles that each of those users have. for ura in role.users: - roles.add( ura.role ) + user = ura.user + for ura2 in user.roles: + roles.add( ura2.role ) # Each role also potentially has groups which, in turn, have members ( users ). We need to # find all roles that each group's members have. for gra in role.groups: diff -r acf985b23928 -r d1bc3b0535d0 lib/galaxy/util/heartbeat.py --- a/lib/galaxy/util/heartbeat.py Wed Jan 27 15:49:50 2010 -0500 +++ b/lib/galaxy/util/heartbeat.py Wed Jan 27 15:50:21 2010 -0500 @@ -54,6 +54,9 @@ self.period = period self.fname = fname self.file = None + self.fname_nonsleeping = fname + ".nonsleeping" + self.file_nonsleeping = None + self.nonsleeping_heartbeats = { } # Save process id self.pid = os.getpid() # Event to wait on when sleeping, allows us to interrupt for shutdown @@ -62,6 +65,9 @@ self.file = open( self.fname, "a" ) print >> self.file, "Heartbeat for pid %d thread started at %s" % ( self.pid, time.asctime() ) print >> self.file + self.file_nonsleeping = open ( self.fname_nonsleeping, "a" ) + print >> self.file_nonsleeping, "Non-Sleeping-threads for pid %d thread started at %s" % ( self.pid, time.asctime() ) + print >> self.file_nonsleeping try: while not self.should_stop: # Print separator with timestamp @@ -81,6 +87,7 @@ print >> self.file, "End dump" print >> self.file self.file.flush() + self.print_nonsleeping(threads) # Sleep for a bit self.wait_event.wait( self.period ) finally: @@ -88,9 +95,86 @@ print >> self.file # Cleanup self.file.close() + self.file_nonsleeping.close() def shutdown( self ): self.should_stop = True self.wait_event.set() self.join() - \ No newline at end of file + + def thread_is_sleeping ( self, last_stack_frame ): + """ + Returns True if the given stack-frame represents a known + sleeper function (at least in python 2.5) + """ + _filename = last_stack_frame[0] + _line = last_stack_frame[1] + _funcname = last_stack_frame[2] + _text = last_stack_frame[3] + ### Ugly hack to tell if a thread is supposedly sleeping or not + ### These are the most common sleeping functions I've found. + ### Is there a better way? (python interpreter internals?) + ### Tested only with python 2.5 + if _funcname=="wait" and _text=="waiter.acquire()": + return True + if _funcname=="wait" and _text=="_sleep(delay)": + return True + if _funcname=="accept" and _text[-14:]=="_sock.accept()": + return True + #Ugly hack: always skip the heartbeat thread + #TODO: get the current thread-id in python + # skip heartbeat thread by thread-id, not by filename + if _filename.find("/lib/galaxy/util/heartbeat.py")!=-1: + return True + ## By default, assume the thread is not sleeping + return False + + def get_interesting_stack_frame ( self, stack_frames ): + """ + Scans a given backtrace stack frames, returns a single + quadraple of [filename, line, function-name, text] of + the single, deepest, most interesting frame. + Interesting being: + inside the galaxy source code ("/lib/galaxy"), + prefreably not an egg. + """ + for _filename, _line, _funcname, _text in reversed(stack_frames): + idx = _filename.find("/lib/galaxy/") + if idx!=-1: + relative_filename = _filename[idx:] + return ( relative_filename, _line, _funcname, _text ) + # no "/lib/galaxy" code found, return the innermost frame + return stack_frames[-1] + + def print_nonsleeping( self, threads_object_dict ): + print >> self.file_nonsleeping, "Non-Sleeping threads at %s:" % time.asctime() + print >> self.file_nonsleeping + all_threads_are_sleeping = True + threads = get_current_thread_object_dict() + for thread_id, frame in threadframe.dict().iteritems(): + if thread_id in threads: + object = repr( threads[thread_id] ) + else: + object = "<No Thread object>" + tb = traceback.extract_stack(frame) + if self.thread_is_sleeping(tb[-1]): + if thread_id in self.nonsleeping_heartbeats: + del self.nonsleeping_heartbeats[thread_id] + continue + + # Count non-sleeping thread heartbeats + if thread_id in self.nonsleeping_heartbeats: + self.nonsleeping_heartbeats[thread_id] += 1 + else: + self.nonsleeping_heartbeats[thread_id]=1 + + good_frame = self.get_interesting_stack_frame(tb) + print >> self.file_nonsleeping, "Thread %s\t%s\tnon-sleeping for %d heartbeat(s)\n File %s:%d\n Function \"%s\"\n %s" % \ + ( thread_id, object, self.nonsleeping_heartbeats[thread_id], good_frame[0], good_frame[1], good_frame[2], good_frame[3] ) + all_threads_are_sleeping = False + + if all_threads_are_sleeping: + print >> self.file_nonsleeping, "All threads are sleeping." + print >> self.file_nonsleeping + self.file_nonsleeping.flush() + diff -r acf985b23928 -r d1bc3b0535d0 lib/galaxy/visualization/tracks/data/bam.py --- a/lib/galaxy/visualization/tracks/data/bam.py Wed Jan 27 15:49:50 2010 -0500 +++ b/lib/galaxy/visualization/tracks/data/bam.py Wed Jan 27 15:50:21 2010 -0500 @@ -1,23 +1,42 @@ +""" +Visualization data provider for BAM format. +""" + import pkg_resources; pkg_resources.require( "pysam" ) + from pysam import csamtools from math import floor, ceil, log import logging class BamDataProvider( object ): + """ + Provides access to intervals from a sorted indexed BAM file. + """ def __init__( self, index, original_dataset ): - self.log = logging.getLogger(__name__) + # self.log = logging.getLogger(__name__) self.index = index self.original_dataset = original_dataset def get_data( self, chrom, start, end, **kwargs ): + """ + Fetch intervals in the region + """ start, end = int(start), int(end) - bamfile = csamtools.Samfile(filename=self.original_dataset.file_name, mode='rb', index_filename=self.index.file_name) - - data = bamfile.fetch(start=start, end=end, reference=chrom) + # Attempt to open the BAM file with index + bamfile = csamtools.Samfile( filename=self.original_dataset.file_name, mode='rb', index_filename=self.index.file_name ) + try: + data = bamfile.fetch(start=start, end=end, reference=chrom) + except ValueError, e: + # Some BAM files do not prefix chromosome names with chr, try + # without + if chrom.startswith( 'chr' ): + data = bamfile.fetch( start=start, end=end, reference=chrom[3:] ) + else: + raise + # Encode reads as list of dictionaries results = [] for read in data: payload = { 'uid': str(read.pos) + str(read.seq), 'start': read.pos, 'end': read.pos + read.rlen, 'name': read.seq } - results.append(payload) bamfile.close() return results diff -r acf985b23928 -r d1bc3b0535d0 lib/galaxy/web/controllers/library_common.py --- a/lib/galaxy/web/controllers/library_common.py Wed Jan 27 15:49:50 2010 -0500 +++ b/lib/galaxy/web/controllers/library_common.py Wed Jan 27 15:50:21 2010 -0500 @@ -489,10 +489,18 @@ msg=util.sanitize_text( msg ), messagetype='error' ) ) lddas.append( ldda ) - # If the library is public all roles are legitimate, but if the library is restricted, only those - # roles associated with the LIBRARY_ACCESS permission are legitimate. library = trans.sa_session.query( trans.app.model.Library ).get( trans.security.decode_id( library_id ) ) - roles = trans.app.security_agent.get_legitimate_roles( trans, library ) + # If access to the dataset is restricted, then use the roles associated with the DATASET_ACCESS permission to + # determine the legitimate roles. If the dataset is public, see if access to the library is restricted. If + # it is, use the roles associated with the LIBRARY_ACCESS permission to determine the legitimate roles. If both + # the dataset and the library are public, all roles are legitimate. All of the datasets will have the same + # permissions at this point. + ldda = lddas[0] + if trans.app.security_agent.dataset_is_public( ldda.dataset ): + # The dataset is public, so check access to the library + roles = trans.app.security_agent.get_legitimate_roles( trans, library ) + else: + roles = trans.app.security_agent.get_legitimate_roles( trans, ldda.dataset ) if params.get( 'update_roles_button', False ): current_user_roles = trans.get_current_user_roles() if cntrller=='library_admin' or ( trans.app.security_agent.can_manage_library_item( current_user_roles, ldda ) and \ diff -r acf985b23928 -r d1bc3b0535d0 lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py Wed Jan 27 15:49:50 2010 -0500 +++ b/lib/galaxy/web/framework/base.py Wed Jan 27 15:50:21 2010 -0500 @@ -327,10 +327,14 @@ def send_file( start_response, trans, body ): # If configured use X-Accel-Redirect header for nginx base = trans.app.config.nginx_x_accel_redirect_base + apache_xsendfile = trans.app.config.apache_xsendfile if base: trans.response.headers['X-Accel-Redirect'] = \ base + os.path.abspath( body.name ) body = [ "" ] + elif apache_xsendfile: + trans.response.headers['X-Sendfile'] = os.path.abspath( body.name ) + body = [ "" ] # Fall back on sending the file in chunks else: body = iterate_file( body )

1 0

[hg] galaxy 3284: Ensure that workflows shared via email prior t...
by Greg Von Kuster 28 Jan '10

28 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/acf985b23928 changeset: 3284:acf985b23928 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Wed Jan 27 15:49:50 2010 -0500 description: Ensure that workflows shared via email prior to changeset 3263:fe6e3c197d69 will have a slug for viewing. diffstat: lib/galaxy/web/base/controller.py | 5 ++++- lib/galaxy/web/controllers/workflow.py | 8 ++++++++ 2 files changed, 12 insertions(+), 1 deletions(-) diffs (40 lines): diff -r 69a04e6bd513 -r acf985b23928 lib/galaxy/web/base/controller.py --- a/lib/galaxy/web/base/controller.py Wed Jan 27 08:51:23 2010 -0500 +++ b/lib/galaxy/web/base/controller.py Wed Jan 27 15:49:50 2010 -0500 @@ -146,7 +146,7 @@ self.set_item_slug( sa_session, item ) def set_item_slug( self, sa_session, item ): - """ Set item slug. Slug is unique among user's importable items for item's class. """ + """ Set item slug. Slug is unique among user's importable items for item's class. Returns true if item's slug was set; false otherwise. """ if item.slug is None or item.slug == "": # Replace whitespace with '-' slug_base = re.sub( "\s+", "-", item.name.lower() ) @@ -164,6 +164,9 @@ slug = '%s-%i' % ( slug_base, count ) count += 1 item.slug = slug + return True + + return False """ Deprecated: `BaseController` used to be available under the name `Root` diff -r 69a04e6bd513 -r acf985b23928 lib/galaxy/web/controllers/workflow.py --- a/lib/galaxy/web/controllers/workflow.py Wed Jan 27 08:51:23 2010 -0500 +++ b/lib/galaxy/web/controllers/workflow.py Wed Jan 27 15:49:50 2010 -0500 @@ -118,6 +118,14 @@ .filter( model.StoredWorkflow.deleted == False ) \ .order_by( desc( model.StoredWorkflow.update_time ) ) \ .all() + + # Legacy issue: all shared workflows must have slugs. + slug_set = False + for workflow_assoc in shared_by_others: + slug_set = self.set_item_slug( trans.sa_session, workflow_assoc.stored_workflow ) + if slug_set: + trans.sa_session.flush() + return trans.fill_template( "workflow/list.mako", workflows = workflows, shared_by_others = shared_by_others )

1 0

[hg] galaxy 3283: Patch from Assaf Gordon to support Apache's mo...
by Greg Von Kuster 28 Jan '10

28 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/32c4edacdc99 changeset: 3283:32c4edacdc99 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 27 14:18:14 2010 -0500 description: Patch from Assaf Gordon to support Apache's mod_xsendfile for offloading downloads to Apache. diffstat: dist-eggs.ini | 24 ++++++++++++------------ lib/galaxy/config.py | 1 + lib/galaxy/web/framework/base.py | 4 ++++ 3 files changed, 17 insertions(+), 12 deletions(-) diffs (63 lines): diff -r e8b17a1cfdf9 -r 32c4edacdc99 dist-eggs.ini --- a/dist-eggs.ini Wed Jan 27 13:34:53 2010 -0500 +++ b/dist-eggs.ini Wed Jan 27 14:18:14 2010 -0500 @@ -7,18 +7,18 @@ ; [hosts] -py2.4-linux-i686-ucs2 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.4 -py2.4-linux-i686-ucs4 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.4 -py2.5-linux-i686-ucs2 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.5 -py2.5-linux-i686-ucs4 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.5 -py2.6-linux-i686-ucs2 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.6 -py2.6-linux-i686-ucs4 = basie.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.6 -py2.4-linux-x86_64-ucs2 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.4 -py2.4-linux-x86_64-ucs4 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.4 -py2.5-linux-x86_64-ucs2 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.5 -py2.5-linux-x86_64-ucs4 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.5 -py2.6-linux-x86_64-ucs2 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.6 -py2.6-linux-x86_64-ucs4 = scofield.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.6 +py2.4-linux-i686-ucs2 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.4 +py2.4-linux-i686-ucs4 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.4 +py2.5-linux-i686-ucs2 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.5 +py2.5-linux-i686-ucs4 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.5 +py2.6-linux-i686-ucs2 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs2/bin/python2.6 +py2.6-linux-i686-ucs4 = stegmaier.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-i686-ucs4/bin/python2.6 +py2.4-linux-x86_64-ucs2 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.4 +py2.4-linux-x86_64-ucs4 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.4 +py2.5-linux-x86_64-ucs2 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.5 +py2.5-linux-x86_64-ucs4 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.5 +py2.6-linux-x86_64-ucs2 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs2/bin/python2.6 +py2.6-linux-x86_64-ucs4 = straub.bx.psu.edu /afs/bx.psu.edu/project/pythons/linux-x86_64-ucs4/bin/python2.6 py2.4-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.4 py2.5-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.5 py2.6-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.6 diff -r e8b17a1cfdf9 -r 32c4edacdc99 lib/galaxy/config.py --- a/lib/galaxy/config.py Wed Jan 27 13:34:53 2010 -0500 +++ b/lib/galaxy/config.py Wed Jan 27 14:18:14 2010 -0500 @@ -97,6 +97,7 @@ raise ConfigurationError( "user_library_import_dir specified in config (%s) does not exist" % self.user_library_import_dir ) self.allow_library_path_paste = kwargs.get( 'allow_library_path_paste', False ) # Configuration options for taking advantage of nginx features + self.apache_xsendfile = kwargs.get( 'apache_xsendfile', False ) self.nginx_x_accel_redirect_base = kwargs.get( 'nginx_x_accel_redirect_base', False ) self.nginx_upload_store = kwargs.get( 'nginx_upload_store', False ) self.nginx_upload_path = kwargs.get( 'nginx_upload_path', False ) diff -r e8b17a1cfdf9 -r 32c4edacdc99 lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py Wed Jan 27 13:34:53 2010 -0500 +++ b/lib/galaxy/web/framework/base.py Wed Jan 27 14:18:14 2010 -0500 @@ -327,10 +327,14 @@ def send_file( start_response, trans, body ): # If configured use X-Accel-Redirect header for nginx base = trans.app.config.nginx_x_accel_redirect_base + apache_xsendfile = trans.app.config.apache_xsendfile if base: trans.response.headers['X-Accel-Redirect'] = \ base + os.path.abspath( body.name ) body = [ "" ] + elif apache_xsendfile: + trans.response.headers['X-Sendfile'] = os.path.abspath( body.name ) + body = [ "" ] # Fall back on sending the file in chunks else: body = iterate_file( body )

1 0

Cloning large histories
by Assaf Gordon 27 Jan '10

27 Jan '10

Hello, I tried to clone a large history (334 active datasets) with very big files (more than half are FASTQ/FASTA files, of 400MB to 1.5GB size). The python process was busy for 17 minutes, and of course I got an HTTP 504 "proxy timeout" error. The cloning actually succeeded (after 17 minutes, the history appeared in my histories list), but I was wondering if there's a way to make this faster - because it effectively disables one of the two web-service processes that I have (luckily, I have two now, thanks yesterday's help). I think I noticed that external set_metadata process was called for some/all of the the cloned datasets. Is it possible to disable set-metadata when cloning? the datasets already have a metadata. (Or I could be mistaken and the external set-metadata was not related to the cloning?). Thanks, -gordon

2 1

[hg] galaxy 3282: Some tweaks to the way roles are derived from ...
by Greg Von Kuster 27 Jan '10

27 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/e8b17a1cfdf9 changeset: 3282:e8b17a1cfdf9 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Jan 27 13:34:53 2010 -0500 description: Some tweaks to the way roles are derived from the access permission on an item. diffstat: lib/galaxy/security/__init__.py | 6 ++++-- lib/galaxy/web/controllers/library_common.py | 14 +++++++++++--- 2 files changed, 15 insertions(+), 5 deletions(-) diffs (46 lines): diff -r 13e217878147 -r e8b17a1cfdf9 lib/galaxy/security/__init__.py --- a/lib/galaxy/security/__init__.py Wed Jan 27 12:43:11 2010 -0500 +++ b/lib/galaxy/security/__init__.py Wed Jan 27 13:34:53 2010 -0500 @@ -138,13 +138,15 @@ intermed.sort() return map( operator.getitem, intermed, ( -1, ) * len( intermed ) ) roles = set() - # If a library has roles associated with the LIBRARY_ACCESS permission, we need to start with them. + # If item has roles associated with the access permission, we need to start with them. access_roles = item.get_access_roles( trans ) for role in access_roles: roles.add( role ) # Each role potentially has users. We need to find all roles that each of those users have. for ura in role.users: - roles.add( ura.role ) + user = ura.user + for ura2 in user.roles: + roles.add( ura2.role ) # Each role also potentially has groups which, in turn, have members ( users ). We need to # find all roles that each group's members have. for gra in role.groups: diff -r 13e217878147 -r e8b17a1cfdf9 lib/galaxy/web/controllers/library_common.py --- a/lib/galaxy/web/controllers/library_common.py Wed Jan 27 12:43:11 2010 -0500 +++ b/lib/galaxy/web/controllers/library_common.py Wed Jan 27 13:34:53 2010 -0500 @@ -489,10 +489,18 @@ msg=util.sanitize_text( msg ), messagetype='error' ) ) lddas.append( ldda ) - # If the library is public all roles are legitimate, but if the library is restricted, only those - # roles associated with the LIBRARY_ACCESS permission are legitimate. library = trans.sa_session.query( trans.app.model.Library ).get( trans.security.decode_id( library_id ) ) - roles = trans.app.security_agent.get_legitimate_roles( trans, library ) + # If access to the dataset is restricted, then use the roles associated with the DATASET_ACCESS permission to + # determine the legitimate roles. If the dataset is public, see if access to the library is restricted. If + # it is, use the roles associated with the LIBRARY_ACCESS permission to determine the legitimate roles. If both + # the dataset and the library are public, all roles are legitimate. All of the datasets will have the same + # permissions at this point. + ldda = lddas[0] + if trans.app.security_agent.dataset_is_public( ldda.dataset ): + # The dataset is public, so check access to the library + roles = trans.app.security_agent.get_legitimate_roles( trans, library ) + else: + roles = trans.app.security_agent.get_legitimate_roles( trans, ldda.dataset ) if params.get( 'update_roles_button', False ): current_user_roles = trans.get_current_user_roles() if cntrller=='library_admin' or ( trans.app.security_agent.can_manage_library_item( current_user_roles, ldda ) and \

1 0

[hg] galaxy 3281: A patch from Assaf Gordon to create a second h...
by Greg Von Kuster 27 Jan '10

27 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/13e217878147 changeset: 3281:13e217878147 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 27 12:43:11 2010 -0500 description: A patch from Assaf Gordon to create a second heartbeat log that only shows the non-sleeping threads, to aid in debugging. diffstat: lib/galaxy/util/heartbeat.py | 86 +++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 85 insertions(+), 1 deletions(-) diffs (120 lines): diff -r 98a45b4fe3a2 -r 13e217878147 lib/galaxy/util/heartbeat.py --- a/lib/galaxy/util/heartbeat.py Wed Jan 27 12:15:43 2010 -0500 +++ b/lib/galaxy/util/heartbeat.py Wed Jan 27 12:43:11 2010 -0500 @@ -54,6 +54,9 @@ self.period = period self.fname = fname self.file = None + self.fname_nonsleeping = fname + ".nonsleeping" + self.file_nonsleeping = None + self.nonsleeping_heartbeats = { } # Save process id self.pid = os.getpid() # Event to wait on when sleeping, allows us to interrupt for shutdown @@ -62,6 +65,9 @@ self.file = open( self.fname, "a" ) print >> self.file, "Heartbeat for pid %d thread started at %s" % ( self.pid, time.asctime() ) print >> self.file + self.file_nonsleeping = open ( self.fname_nonsleeping, "a" ) + print >> self.file_nonsleeping, "Non-Sleeping-threads for pid %d thread started at %s" % ( self.pid, time.asctime() ) + print >> self.file_nonsleeping try: while not self.should_stop: # Print separator with timestamp @@ -81,6 +87,7 @@ print >> self.file, "End dump" print >> self.file self.file.flush() + self.print_nonsleeping(threads) # Sleep for a bit self.wait_event.wait( self.period ) finally: @@ -88,9 +95,86 @@ print >> self.file # Cleanup self.file.close() + self.file_nonsleeping.close() def shutdown( self ): self.should_stop = True self.wait_event.set() self.join() - \ No newline at end of file + + def thread_is_sleeping ( self, last_stack_frame ): + """ + Returns True if the given stack-frame represents a known + sleeper function (at least in python 2.5) + """ + _filename = last_stack_frame[0] + _line = last_stack_frame[1] + _funcname = last_stack_frame[2] + _text = last_stack_frame[3] + ### Ugly hack to tell if a thread is supposedly sleeping or not + ### These are the most common sleeping functions I've found. + ### Is there a better way? (python interpreter internals?) + ### Tested only with python 2.5 + if _funcname=="wait" and _text=="waiter.acquire()": + return True + if _funcname=="wait" and _text=="_sleep(delay)": + return True + if _funcname=="accept" and _text[-14:]=="_sock.accept()": + return True + #Ugly hack: always skip the heartbeat thread + #TODO: get the current thread-id in python + # skip heartbeat thread by thread-id, not by filename + if _filename.find("/lib/galaxy/util/heartbeat.py")!=-1: + return True + ## By default, assume the thread is not sleeping + return False + + def get_interesting_stack_frame ( self, stack_frames ): + """ + Scans a given backtrace stack frames, returns a single + quadraple of [filename, line, function-name, text] of + the single, deepest, most interesting frame. + Interesting being: + inside the galaxy source code ("/lib/galaxy"), + prefreably not an egg. + """ + for _filename, _line, _funcname, _text in reversed(stack_frames): + idx = _filename.find("/lib/galaxy/") + if idx!=-1: + relative_filename = _filename[idx:] + return ( relative_filename, _line, _funcname, _text ) + # no "/lib/galaxy" code found, return the innermost frame + return stack_frames[-1] + + def print_nonsleeping( self, threads_object_dict ): + print >> self.file_nonsleeping, "Non-Sleeping threads at %s:" % time.asctime() + print >> self.file_nonsleeping + all_threads_are_sleeping = True + threads = get_current_thread_object_dict() + for thread_id, frame in threadframe.dict().iteritems(): + if thread_id in threads: + object = repr( threads[thread_id] ) + else: + object = "<No Thread object>" + tb = traceback.extract_stack(frame) + if self.thread_is_sleeping(tb[-1]): + if thread_id in self.nonsleeping_heartbeats: + del self.nonsleeping_heartbeats[thread_id] + continue + + # Count non-sleeping thread heartbeats + if thread_id in self.nonsleeping_heartbeats: + self.nonsleeping_heartbeats[thread_id] += 1 + else: + self.nonsleeping_heartbeats[thread_id]=1 + + good_frame = self.get_interesting_stack_frame(tb) + print >> self.file_nonsleeping, "Thread %s\t%s\tnon-sleeping for %d heartbeat(s)\n File %s:%d\n Function \"%s\"\n %s" % \ + ( thread_id, object, self.nonsleeping_heartbeats[thread_id], good_frame[0], good_frame[1], good_frame[2], good_frame[3] ) + all_threads_are_sleeping = False + + if all_threads_are_sleeping: + print >> self.file_nonsleeping, "All threads are sleeping." + print >> self.file_nonsleeping + self.file_nonsleeping.flush() +

1 0

[hg] galaxy 3280: trackster: Support ensembl / 1000 genomes styl...
by Greg Von Kuster 27 Jan '10

27 Jan '10

details: http://www.bx.psu.edu/hg/galaxy/rev/98a45b4fe3a2 changeset: 3280:98a45b4fe3a2 user: James Taylor <james(a)jamestaylor.org> date: Wed Jan 27 12:15:43 2010 -0500 description: trackster: Support ensembl / 1000 genomes style BAM files which don't prefix chromosome numbers with 'chr' diffstat: lib/galaxy/visualization/tracks/data/bam.py | 29 ++++++++++++++++++++++++----- 1 files changed, 24 insertions(+), 5 deletions(-) diffs (51 lines): diff -r 69a04e6bd513 -r 98a45b4fe3a2 lib/galaxy/visualization/tracks/data/bam.py --- a/lib/galaxy/visualization/tracks/data/bam.py Wed Jan 27 08:51:23 2010 -0500 +++ b/lib/galaxy/visualization/tracks/data/bam.py Wed Jan 27 12:15:43 2010 -0500 @@ -1,23 +1,42 @@ +""" +Visualization data provider for BAM format. +""" + import pkg_resources; pkg_resources.require( "pysam" ) + from pysam import csamtools from math import floor, ceil, log import logging class BamDataProvider( object ): + """ + Provides access to intervals from a sorted indexed BAM file. + """ def __init__( self, index, original_dataset ): - self.log = logging.getLogger(__name__) + # self.log = logging.getLogger(__name__) self.index = index self.original_dataset = original_dataset def get_data( self, chrom, start, end, **kwargs ): + """ + Fetch intervals in the region + """ start, end = int(start), int(end) - bamfile = csamtools.Samfile(filename=self.original_dataset.file_name, mode='rb', index_filename=self.index.file_name) - - data = bamfile.fetch(start=start, end=end, reference=chrom) + # Attempt to open the BAM file with index + bamfile = csamtools.Samfile( filename=self.original_dataset.file_name, mode='rb', index_filename=self.index.file_name ) + try: + data = bamfile.fetch(start=start, end=end, reference=chrom) + except ValueError, e: + # Some BAM files do not prefix chromosome names with chr, try + # without + if chrom.startswith( 'chr' ): + data = bamfile.fetch( start=start, end=end, reference=chrom[3:] ) + else: + raise + # Encode reads as list of dictionaries results = [] for read in data: payload = { 'uid': str(read.pos) + str(read.seq), 'start': read.pos, 'end': read.pos + read.rlen, 'name': read.seq } - results.append(payload) bamfile.close() return results

1 0

Galaxy + Apache + XSendFile
by Assaf Gordon 27 Jan '10

27 Jan '10

Hi, I noticed there's a trick to accelerate file downloads for nginx ("X_accel_redirect"), The following patch adds the same feature for Apache's xsendfile. To enable it: 1. Download, Compile, Install mod_xsendfile http://tn123.ath.cx/mod_xsendfile/ 2. Make sure your apache configuration has the correct "LoadModule" for xsendfile. 3. In the apache's configuration relevant to galaxy, add the two directives: XSendFile ON and XSendFileAllowAbove ON Example: ReWriteEngine on ReWriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy(.*) http://127.0.0.1:8080$1 [P] <Location "/galaxy" > order allow,deny allow from all XSendFile on XSendFileAllowAbove on </Location> 4. In "universe_wsgi.ini", add the following parameter: apache_xsendfile = true If all worked well, whenever a user downloads a dataset (via the "save" link), the actual file transfer will be delegated to back apache. Comments are welcomed, -gordon

2 1