January 2009 - galaxy-dev - lists.galaxyproject.org

[hg] galaxy 1704: Improvements to job recovery and clarification...
by Nate Coraor 14 Jan '09

14 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/941266327bc3 changeset: 1704:941266327bc3 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 14 10:49:14 2009 -0500 description: Improvements to job recovery and clarification of the false path's job id 2 file(s) affected in this change: lib/galaxy/jobs/__init__.py lib/galaxy/tools/__init__.py diffs (79 lines): diff -r c259ce9cb50d -r 941266327bc3 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Wed Jan 14 08:50:31 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Wed Jan 14 10:49:14 2009 -0500 @@ -111,19 +111,19 @@ def __check_jobs_at_startup( self ): """ - Checks all jobs that are in the 'running' or 'queued' state in the - database and requeues or cleans up as necessary. Only run as the + Checks all jobs that are in the 'new', 'queued' or 'running' state in + the database and requeues or cleans up as necessary. Only run as the job manager starts. """ model = self.app.model - # Jobs in the NEW state won't be requeued unless we're tracking in the database - if not self.track_jobs_in_database: - for job in model.Job.filter( model.Job.c.state==model.Job.states.NEW ).all(): - log.debug( "no runner: %s is still in new state, adding to the jobs queue" %job.id ) + for job in model.Job.filter( model.Job.c.state==model.Job.states.NEW ).all(): + log.debug( "no runner: %s is still in new state, adding to the jobs queue" %job.id ) + self.queue.put( ( job.id, job.tool_id ) ) + for job in model.Job.filter( (model.Job.c.state == model.Job.states.RUNNING) | (model.Job.c.state == model.Job.states.QUEUED) ).all(): + if job.job_runner_name is None: + log.debug( "no runner: %s is still in queued state, adding to the jobs queue" %job.id ) self.queue.put( ( job.id, job.tool_id ) ) - for job in model.Job.filter( (model.Job.c.state == model.Job.states.RUNNING) | (model.Job.c.state == model.Job.states.QUEUED) ).all(): - if job.job_runner_name is not None: - # why are we passing the queue to the wrapper? + else: job_wrapper = JobWrapper( job, self.app.toolbox.tools_by_id[ job.tool_id ], self ) self.dispatcher.recover( job, job_wrapper ) @@ -383,12 +383,12 @@ if not job.state == model.Job.states.DELETED: for dataset_assoc in job.output_datasets: if self.app.config.outputs_to_working_directory: - false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.id ) ) - if os.path.exists( false_path ): + false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.dataset.id ) ) + try: shutil.move( false_path, dataset_assoc.dataset.file_name ) log.debug( "fail(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) - else: - log.warning( "fail(): Missing output file in working directory: %s" % false_path ) + except ( IOError, OSError ), e: + log.error( "fail(): Missing output file in working directory: %s" % e ) dataset = dataset_assoc.dataset dataset.refresh() dataset.state = dataset.states.ERROR @@ -453,12 +453,13 @@ job.state = 'ok' for dataset_assoc in job.output_datasets: if self.app.config.outputs_to_working_directory: - false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.id ) ) - if os.path.exists( false_path ): + false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.dataset.id ) ) + try: shutil.move( false_path, dataset_assoc.dataset.file_name ) log.debug( "finish(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) - else: - log.warning( "finish(): Missing output file in working directory: %s" % false_path ) + except ( IOError, OSError ): + self.fail( "The job's output dataset(s) could not be read" ) + return for dataset in dataset_assoc.dataset.dataset.history_associations: #need to update all associated output hdas, i.e. history was shared with job running dataset.blurb = 'done' dataset.peek = 'no peek' diff -r c259ce9cb50d -r 941266327bc3 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py Wed Jan 14 08:50:31 2009 -0500 +++ b/lib/galaxy/tools/__init__.py Wed Jan 14 10:49:14 2009 -0500 @@ -1098,7 +1098,7 @@ for name, data in output_datasets.items(): # Write outputs to the working directory (for security purposes) if desired. if self.app.config.outputs_to_working_directory and working_directory is not None: - false_path = os.path.abspath( os.path.join( working_directory, "galaxy_dataset_%d.dat" % data.id ) ) + false_path = os.path.abspath( os.path.join( working_directory, "galaxy_dataset_%d.dat" % data.dataset.id ) ) param_dict[name] = DatasetFilenameWrapper( data, false_path = false_path ) open( false_path, 'w' ).close() else:

1 0

[hg] galaxy 1702: Set the working directory in the wrapper when ...
by Nate Coraor 14 Jan '09

14 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/b868c8d902d2 changeset: 1702:b868c8d902d2 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 14 08:43:00 2009 -0500 description: Set the working directory in the wrapper when recovering jobs at startup. 1 file(s) affected in this change: lib/galaxy/jobs/__init__.py diffs (12 lines): diff -r c3a20d503bd3 -r b868c8d902d2 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Tue Jan 13 12:18:08 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Wed Jan 14 08:43:00 2009 -0500 @@ -125,6 +125,8 @@ if job.job_runner_name is not None: # why are we passing the queue to the wrapper? job_wrapper = JobWrapper( job, self.app.toolbox.tools_by_id[ job.tool_id ], self ) + job_wrapper.working_directory = \ + os.path.join( self.app.config.job_working_directory, str( job.id ) ) self.dispatcher.recover( job, job_wrapper ) def __monitor( self ):

1 0

[hg] galaxy 1703: Move setting of the working directory to the w...
by Nate Coraor 14 Jan '09

14 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/c259ce9cb50d changeset: 1703:c259ce9cb50d user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 14 08:50:31 2009 -0500 description: Move setting of the working directory to the wrapper's init method instead. 1 file(s) affected in this change: lib/galaxy/jobs/__init__.py diffs (37 lines): diff -r b868c8d902d2 -r c259ce9cb50d lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Wed Jan 14 08:43:00 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Wed Jan 14 08:50:31 2009 -0500 @@ -125,8 +125,6 @@ if job.job_runner_name is not None: # why are we passing the queue to the wrapper? job_wrapper = JobWrapper( job, self.app.toolbox.tools_by_id[ job.tool_id ], self ) - job_wrapper.working_directory = \ - os.path.join( self.app.config.job_working_directory, str( job.id ) ) self.dispatcher.recover( job, job_wrapper ) def __monitor( self ): @@ -300,9 +298,13 @@ self.queue = queue self.app = queue.app self.extra_filenames = [] - self.working_directory = None self.command_line = None self.galaxy_lib_dir = None + # With job outputs in the working directory, we need the working + # directory to be set before prepare is run, or else premature deletion + # and job recovery fail. + self.working_directory = \ + os.path.join( self.app.config.job_working_directory, str( self.job_id ) ) def get_param_dict( self ): """ @@ -319,9 +321,6 @@ config files. """ mapping.context.current.clear() #this prevents the metadata reverting that has been seen in conjunction with the PBS job runner - # Create the working directory - self.working_directory = \ - os.path.join( self.app.config.job_working_directory, str( self.job_id ) ) if not os.path.exists( self.working_directory ): os.mkdir( self.working_directory ) # Restore parameters from the database

1 0

[hg] galaxy 1700: Eliminate boilerplate code to track memory_usa...
by Greg Von Kuster 13 Jan '09

13 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/03f1e9e78f74 changeset: 1700:03f1e9e78f74 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Tue Jan 13 11:33:17 2009 -0500 description: Eliminate boilerplate code to track memory_usage, keeping the class and config settings for possible future use. 5 file(s) affected in this change: lib/galaxy/jobs/__init__.py lib/galaxy/tools/actions/__init__.py lib/galaxy/tools/actions/upload.py lib/galaxy/web/controllers/root.py lib/galaxy/web/controllers/user.py diffs (446 lines): diff -r 511c2b1966dc -r 03f1e9e78f74 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Tue Jan 13 10:55:08 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Tue Jan 13 11:33:17 2009 -0500 @@ -246,9 +246,6 @@ job can be dispatched. Otherwise, return JOB_WAIT indicating that input datasets are still being prepared. """ - if self.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() for dataset_assoc in job.input_datasets: idata = dataset_assoc.dataset if not idata: @@ -264,9 +261,6 @@ elif idata.state != idata.states.OK: # need to requeue return JOB_WAIT - if self.app.memory_usage: - m1 = self.app.memory_usage.memory( m0, pretty=True ) - log.info("End of check_if_ready_to_run for job id %d, memory used increased by %s" % ( job.id, m1 ) ) if job.state == model.Job.states.DELETED: return JOB_DELETED return JOB_READY @@ -322,9 +316,6 @@ Prepare the job to run by creating the working directory and the config files. """ - if self.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() mapping.context.current.clear() #this prevents the metadata reverting that has been seen in conjunction with the PBS job runner # Create the working directory self.working_directory = \ @@ -378,9 +369,6 @@ extra_filenames.append( param_filename ) self.param_dict = param_dict self.extra_filenames = extra_filenames - if self.app.memory_usage: - m1 = self.app.memory_usage.memory( m0, pretty=True ) - log.info("End of prepare for job id %d, memory used increased by %s" % ( job.id, m1 ) ) return extra_filenames def fail( self, message, exception=False ): @@ -388,9 +376,6 @@ Indicate job failure by setting state and message on all output datasets. """ - if self.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() job = model.Job.get( self.job_id ) job.refresh() # if the job was deleted, don't fail it @@ -417,16 +402,10 @@ if exception: job.traceback = traceback.format_exc() job.flush() - if self.app.memory_usage: - m1 = self.app.memory_usage.memory( m0, pretty=True ) - log.info("End of fail for job id %d, memory used increased by %s" % ( job.id, m1 ) ) # If the job was deleted, just clean up self.cleanup() def change_state( self, state, info = False ): - if self.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() job = model.Job.get( self.job_id ) job.refresh() for dataset_assoc in job.output_datasets: @@ -440,9 +419,6 @@ job.info = info job.state = state job.flush() - if self.app.memory_usage: - m1 = self.app.memory_usage.memory( m0, pretty=True ) - log.info("End of change_state for job id %d, memory used increased by %s" % ( job.id, m1 ) ) def get_state( self ): job = model.Job.get( self.job_id ) @@ -462,9 +438,6 @@ the output datasets based on stderr and stdout from the command, and the contents of the output files. """ - if self.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() # default post job setup mapping.context.current.clear() job = model.Job.get( self.job_id ) @@ -535,15 +508,9 @@ job.command_line = self.command_line mapping.context.current.flush() log.debug( 'job %d ended' % self.job_id ) - if self.app.memory_usage: - m1 = self.app.memory_usage.memory( m0, pretty=True ) - log.info("End of finish for job id %d, memory used increased by %s" % ( job.id, m1 ) ) self.cleanup() def cleanup( self ): - if self.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() # remove temporary files try: for fname in self.extra_filenames: @@ -552,9 +519,6 @@ os.rmdir( self.working_directory ) except: log.exception( "Unable to cleanup job %d" % self.job_id ) - if self.app.memory_usage: - m1 = self.app.memory_usage.memory( m0, pretty=True ) - log.info("End of cleanup for job id %d, memory used increased by %s" % ( self.job_id, m1 ) ) def get_command_line( self ): return self.command_line diff -r 511c2b1966dc -r 03f1e9e78f74 lib/galaxy/tools/actions/__init__.py --- a/lib/galaxy/tools/actions/__init__.py Tue Jan 13 10:55:08 2009 -0500 +++ b/lib/galaxy/tools/actions/__init__.py Tue Jan 13 11:33:17 2009 -0500 @@ -65,9 +65,6 @@ return input_datasets def execute(self, tool, trans, incoming={}, set_output_hid=True ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = trans.app.memory_usage.memory() out_data = {} # Collect any input datasets from the incoming parameters inp_data = self.collect_input_datasets( tool, incoming, trans ) @@ -201,9 +198,6 @@ for name, dataset in out_data.iteritems(): job.add_output_dataset( name, dataset ) trans.app.model.flush() - if trans.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info("End of tool %s execution for job id %d, memory used increased by %s" % ( tool.id, job.id, m1 ) ) # Some tools are not really executable, but jobs are still created for them ( for record keeping ). # Examples include tools that redirect to other applications ( epigraph ). These special tools must # include something that can be retrieved from the params ( e.g., REDIRECT_URL ) to keep the job diff -r 511c2b1966dc -r 03f1e9e78f74 lib/galaxy/tools/actions/upload.py --- a/lib/galaxy/tools/actions/upload.py Tue Jan 13 10:55:08 2009 -0500 +++ b/lib/galaxy/tools/actions/upload.py Tue Jan 13 11:33:17 2009 -0500 @@ -15,9 +15,6 @@ self.line_count = None def execute( self, tool, trans, incoming={}, set_output_hid = True ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = trans.app.memory_usage.memory() data_file = incoming['file_data'] file_type = incoming['file_type'] dbkey = incoming['dbkey'] @@ -129,9 +126,6 @@ job.flush() log.info( 'job id %d ended ok, file size: %s' % ( job.id, file_size_str ) ) trans.log_event( 'job id %d ended ok, file size: %s' % ( job.id, file_size_str ), tool_id=tool.id ) - if trans.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info("End of tool %s execution for job id %d, memory used increased by %s" % ( tool.id, job.id, m1 ) ) return dict( output=hda ) def upload_empty( self, trans, job, err_code, err_msg ): diff -r 511c2b1966dc -r 03f1e9e78f74 lib/galaxy/web/controllers/root.py --- a/lib/galaxy/web/controllers/root.py Tue Jan 13 10:55:08 2009 -0500 +++ b/lib/galaxy/web/controllers/root.py Tue Jan 13 11:33:17 2009 -0500 @@ -116,9 +116,6 @@ Returns data directly into the browser. Sets the mime-type according to the extension """ - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() if hid is not None: try: hid = int( hid ) @@ -149,9 +146,6 @@ fname = ''.join(c in valid_chars and c or '_' for c in fname)[0:150] trans.response.headers["Content-Disposition"] = "attachment; filename=GalaxyHistoryItem-%s-[%s]%s" % (data.hid, fname, toext) trans.log_event( "Display dataset id: %s" % str(id) ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/display, memory used increased by %s" % m1 ) try: return open( data.file_name ) except: @@ -328,9 +322,6 @@ @web.expose def history_delete( self, trans, id=None, **kwd): """Deletes a list of histories, ensures that histories are owned by current user""" - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() history_names = [] if id: if isinstance( id, list ): @@ -358,18 +349,12 @@ trans.log_event( "History id %s marked as deleted" % str(hid) ) else: return trans.show_message( "You must select at least one history to delete." ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_delete, memory used increased by %s" % m1 ) return trans.show_message( "History deleted: %s" % ",".join(history_names), refresh_frames=['history']) @web.expose def history_undelete( self, trans, id=[], **kwd): """Undeletes a list of histories, ensures that histories are owned by current user""" - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() history_names = [] errors = [] ok_msg = "" @@ -401,34 +386,22 @@ ok_msg = "Histories (%s) have been undeleted." % ", ".join( history_names ) else: errors.append( "You must select at least one history to undelete." ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_undelete, memory used increased by %s" % m1 ) return self.history_available( trans, id=','.join( id ), show_deleted=True, ok_msg = ok_msg, error_msg = " ".join( errors ) ) @web.expose def clear_history( self, trans ): """Clears the history for a user""" - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() history = trans.get_history() for dataset in history.datasets: dataset.deleted = True dataset.clear_associated_files() self.app.model.flush() trans.log_event( "History id %s cleared" % (str(history.id)) ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/clear_history, memory used increased by %s" % m1 ) trans.response.send_redirect( url_for("/index" ) ) @web.expose @web.require_login( "share histories with other users" ) def history_share( self, trans, id=None, email="", **kwd ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() send_to_err = "" if not id: id = trans.get_history().id @@ -455,9 +428,6 @@ trans.log_event( "History share, id: %s, name: '%s': to new id: %s" % (str(history.id), history.name, str(new_history.id)) ) self.app.model.flush() return trans.show_message( "History (%s) has been shared with: %s" % (",".join(history_names),email) ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_share, memory used increased by %s" % m1 ) return trans.fill_template( "/history/share.mako", histories=histories, email=email, send_to_err=send_to_err) @web.expose @@ -466,9 +436,6 @@ """ List all available histories """ - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() if as_xml: trans.response.set_content_type('text/xml') return trans.fill_template( "/history/list_as_xml.mako" ) @@ -476,9 +443,6 @@ id = id.split( "," ) trans.log_event( "History id %s available" % str( id ) ) history_operations = dict( share=self.history_share, rename=self.history_rename, delete=self.history_delete, undelete=self.history_undelete ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_available, memory used increased by %s" % m1 ) if do_operation in history_operations: return history_operations[do_operation]( trans, id=id, show_deleted=show_deleted, ok_msg=ok_msg, error_msg=error_msg, **kwd ) return trans.fill_template( "/history/list.mako", ids=id, @@ -491,9 +455,6 @@ @web.expose def history_import( self, trans, id=None, confirm=False, **kwd ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() msg = "" user = trans.get_user() user_history = trans.get_history() @@ -534,9 +495,6 @@ new_history.flush() trans.set_history( new_history ) trans.log_event( "History imported, id: %s, name: '%s': " % (str(new_history.id) , new_history.name ) ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_import, memory used increased by %s" % m1 ) return trans.show_ok_message( """ History "%s" has been imported. Click <a href="%s">here</a> to begin.""" % ( new_history.name, web.url_for( '/' ) ) ) @@ -551,9 +509,6 @@ if not id: return trans.response.send_redirect( web.url_for( action='history_available' ) ) else: - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() new_history = trans.app.model.History.get( id ) if new_history: galaxy_session = trans.get_galaxy_session() @@ -565,9 +520,6 @@ new_history.flush() trans.set_history( new_history ) trans.log_event( "History switched to id: %s, name: '%s'" % (str(new_history.id), new_history.name ) ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_switch, memory used increased by %s" % m1 ) return trans.show_message( "History switched to: %s" % new_history.name, refresh_frames=['history']) else: @@ -575,22 +527,13 @@ @web.expose def history_new( self, trans ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() trans.new_history() trans.log_event( "Created new History, id: %s." % str(trans.get_history().id) ) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_new, memory used increased by %s" % m1 ) return trans.show_message( "New history created", refresh_frames = ['history'] ) @web.expose @web.require_login( "renames histories" ) def history_rename( self, trans, id=None, name=None, **kwd ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() user = trans.get_user() if not isinstance( id, list ): @@ -627,17 +570,11 @@ change_msg = change_msg + "You must specify a valid name for History: "+cur_names[i]+"" else: change_msg = change_msg + "History: "+cur_names[i]+" does not appear to belong to you." - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_rename, memory used increased by %s" % m1 ) return trans.show_message( "%s" % change_msg, refresh_frames=['history'] ) @web.expose def history_add_to( self, trans, history_id=None, file_data=None, name="Data Added to History",info=None,ext="txt",dbkey="?",**kwd ): """Adds a POSTed file to a History""" - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() try: history = trans.app.model.History.get( history_id ) data = trans.app.model.HistoryDatasetAssociation( name = name, info = info, extension = ext, dbkey = dbkey, create_dataset = True ) @@ -656,9 +593,6 @@ data.set_peek() data.flush() trans.log_event("Added dataset %d to history %d" %(data.id, trans.history.id)) - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/history_add_to, memory used increased by %s" % m1 ) return trans.show_ok_message("Dataset "+str(data.hid)+" added to history "+str(history_id)+".") except: return trans.show_error_message("Adding File to History has Failed") @@ -666,9 +600,6 @@ @web.expose def dataset_make_primary( self, trans, id=None): """Copies a dataset and makes primary""" - if trans.app.memory_usage: - # Keep track of memory usage - m0 = self.app.memory_usage.memory() try: old_data = self.app.model.HistoryDatasetAssociation.get( id ) new_data = old_data.copy() @@ -677,9 +608,6 @@ history = trans.get_history() history.add_dataset(new_data) new_data.flush() - if self.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of root/dataset_make_primary, memory used increased by %s" % m1 ) return trans.show_message( "Secondary dataset has been made primary.", refresh_frames=['history'] ) except: return trans.show_error_message( "Failed to make secondary dataset primary." ) diff -r 511c2b1966dc -r 03f1e9e78f74 lib/galaxy/web/controllers/user.py --- a/lib/galaxy/web/controllers/user.py Tue Jan 13 10:55:08 2009 -0500 +++ b/lib/galaxy/web/controllers/user.py Tue Jan 13 11:33:17 2009 -0500 @@ -70,9 +70,6 @@ @web.expose def login( self, trans, email='', password='' ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = trans.app.memory_usage.memory() email_error = password_error = None # Attempt login if email or password: @@ -88,9 +85,6 @@ trans.handle_user_login( user ) trans.log_event( "User logged in" ) return trans.show_ok_message( "Now logged in as " + user.email, refresh_frames=['masthead', 'history'] ) - if trans.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of user/login, memory used increased by %s" % m1 ) return trans.show_form( web.FormBuilder( web.url_for(), "Login", submit_text="Login" ) .add_text( "email", "Email address", value=email, error=email_error ) @@ -98,22 +92,13 @@ help="<a href='%s'>Forgot password? Reset here</a>" % web.url_for( action='reset_password' ) ) ) @web.expose def logout( self, trans ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = trans.app.memory_usage.memory() # Since logging an event requires a session, we'll log prior to ending the session trans.log_event( "User logged out" ) trans.handle_user_logout() - if trans.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of user/logout, memory used increased by %s" % m1 ) return trans.show_ok_message( "You are no longer logged in", refresh_frames=['masthead', 'history'] ) @web.expose def create( self, trans, email='', password='', confirm='',subscribe=False ): - if trans.app.memory_usage: - # Keep track of memory usage - m0 = trans.app.memory_usage.memory() email_error = password_error = confirm_error = None if email: if len( email ) == 0 or "@" not in email or "." not in email: @@ -139,9 +124,6 @@ mail.write("To: %s\nFrom: %s\nSubject: Join Mailing List\n\nJoin Mailing list." % (trans.app.config.mailing_join_addr,email) ) if mail.close(): return trans.show_warn_message( "Now logged in as " + user.email+". However, subscribing to the mailing list has failed.", refresh_frames=['masthead', 'history'] ) - if trans.app.memory_usage: - m1 = trans.app.memory_usage.memory( m0, pretty=True ) - log.info( "End of user/create, memory used increased by %s" % m1 ) return trans.show_ok_message( "Now logged in as " + user.email, refresh_frames=['masthead', 'history'] ) return trans.show_form( web.FormBuilder( web.url_for(), "Create account", submit_text="Create" )

1 0

[hg] galaxy 1701: Tool to find the least common ancestor added u...
by Greg Von Kuster 13 Jan '09

13 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/c3a20d503bd3 changeset: 1701:c3a20d503bd3 user: guru date: Tue Jan 13 12:18:08 2009 -0500 description: Tool to find the least common ancestor added under Taxonomy section. 2 file(s) affected in this change: tools/taxonomy/lca.py tools/taxonomy/lca.xml diffs (166 lines): diff -r 03f1e9e78f74 -r c3a20d503bd3 tools/taxonomy/lca.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/taxonomy/lca.py Tue Jan 13 12:18:08 2009 -0500 @@ -0,0 +1,145 @@ +#!/usr/bin/env python +#Guruprasad Ananda +""" +This tool provides the SQL "group by" functionality. +""" +import sys, string, re, commands, tempfile, random +#from rpy import * + +def stop_err(msg): + sys.stderr.write(msg) + sys.exit() + +def main(): + try: + inputfile = sys.argv[1] + outfile = sys.argv[2] + except: + stop_err("Syntax error: Use correct syntax: program infile outfile") + group_col = 0 + + tmpfile = tempfile.NamedTemporaryFile() + + try: + """ + The -k option for the Posix sort command is as follows: + -k, --key=POS1[,POS2] + start a key at POS1, end it at POS2 (origin 1) + In other words, column positions start at 1 rather than 0, so + we need to add 1 to group_col. + if POS2 is not specified, the newer versions of sort will consider the entire line for sorting. To prevent this, we set POS2=POS1. + """ + command_line = "sort -f -k " + str(group_col+1) +"," + str(group_col+1) + " -o " + tmpfile.name + " " + inputfile + except Exception, exc: + stop_err( 'Initialization error -> %s' %str(exc) ) + + error_code, stdout = commands.getstatusoutput(command_line) + + if error_code != 0: + stop_err( "Sorting input dataset resulted in error: %s: %s" %( error_code, stdout )) + + prev_item = "" + prev_vals = [] + remaining_vals = [] + skipped_lines = 0 + first_invalid_line = 0 + invalid_line = '' + invalid_value = '' + invalid_column = 0 + fout = open(outfile, "w") + cols = range(1,25) + block_valid = False + + for ii, line in enumerate( file( tmpfile.name )): + if line and not line.startswith( '#' ): + line = line.rstrip( '\r\n' ) + try: + fields = line.split("\t") + item = fields[group_col] + if prev_item != "": + # At this level, we're grouping on values (item and prev_item) in group_col + if item == prev_item: + # Keep iterating and storing values until a new value is encountered. + if block_valid: + for i, col in enumerate(cols): + if col >= 3: + prev_vals[i].append(fields[col].strip()) + if len(set(prev_vals[i])) > 1: + block_valid = False + break + + else: + """ + When a new value is encountered, write the previous value and the + corresponding aggregate values into the output file. This works + due to the sort on group_col we've applied to the data above. + """ + out_list = ['']*25 + out_list[0] = str(prev_item) + out_list[1] = str(prev_vals[0][0]) + out_list[2] = str(prev_vals[1][0]) + out_list[24] = str(prev_vals[23][0]) + #print >> fout, prev_vals + #sys.exit() + for k, col in enumerate(cols): + if col >= 3 and col < 24: + if len(set(prev_vals[k])) == 1: + out_list[col] = prev_vals[k][0] + else: + break + while k < 23: + out_list[k+1] = 'n' + k += 1 + + print >>fout, '\t'.join(out_list) + + block_valid = True + prev_item = item + prev_vals = [] + for col in cols: + val_list = [] + val_list.append(fields[col].strip()) + prev_vals.append(val_list) + + else: + # This only occurs once, right at the start of the iteration. + block_valid = True + prev_item = item #groupby item + for col in cols: #everyting else + val_list = [] + val_list.append(fields[col].strip()) + prev_vals.append(val_list) + + except Exception, exc: + skipped_lines += 1 + if not first_invalid_line: + first_invalid_line = ii+1 + else: + skipped_lines += 1 + if not first_invalid_line: + first_invalid_line = ii+1 + + # Handle the last grouped value + out_list = ['']*25 + out_list[0] = str(prev_item) + out_list[1] = str(prev_vals[0][0]) + out_list[2] = str(prev_vals[1][0]) + out_list[24] = str(prev_vals[23][0]) + for k, col in enumerate(cols): + if col >= 3 and col < 24: + if len(set(prev_vals[k])) == 1: + out_list[col] = prev_vals[k][0] + else: + break + while k < 23: + out_list[k+1] = 'n' + k += 1 + + print >>fout, '\t'.join(out_list) + + if skipped_lines > 0: + msg= "Skipped %d invalid lines starting with line %d. Value '%s' in column %d is not numeric." % ( skipped_lines, first_invalid_line, invalid_value, invalid_column ) + print msg + +if __name__ == "__main__": + main() \ No newline at end of file diff -r 03f1e9e78f74 -r c3a20d503bd3 tools/taxonomy/lca.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/taxonomy/lca.xml Tue Jan 13 12:18:08 2009 -0500 @@ -0,0 +1,12 @@ +<tool id="lca1" name="Least Common Ancestor" version="1.0.0"> + <description></description> + <command interpreter="python"> + lca.py $input1 $out_file1 + </command> + <inputs> + <param format="taxonomy" name="input1" type="data" label="Select taxonomy dataset"/> + </inputs> + <outputs> + <data format="taxonomy" name="out_file1" metadata_source="input1" /> + </outputs> +</tool>

1 0

[hg] galaxy 1697: galaxy.config.resolve_path is not absolute...
by Greg Von Kuster 13 Jan '09

13 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/a6d6c2e4c021 changeset: 1697:a6d6c2e4c021 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Jan 09 17:18:18 2009 -0500 description: galaxy.config.resolve_path is not absolute... 1 file(s) affected in this change: lib/galaxy/config.py diffs (21 lines): diff -r aac0403bbe68 -r a6d6c2e4c021 lib/galaxy/config.py --- a/lib/galaxy/config.py Fri Jan 09 17:08:49 2009 -0500 +++ b/lib/galaxy/config.py Fri Jan 09 17:18:18 2009 -0500 @@ -15,7 +15,7 @@ if not( os.path.isabs( path ) ): path = os.path.join( root, path ) return path - + class ConfigurationError( Exception ): pass @@ -44,7 +44,7 @@ self.cluster_job_queue_workers = int( kwargs.get( "cluster_job_queue_workers", "5" ) ) self.job_scheduler_policy = kwargs.get("job_scheduler_policy", "FIFO") self.job_queue_cleanup_interval = int( kwargs.get("job_queue_cleanup_interval", "5") ) - self.cluster_files_directory = resolve_path( kwargs.get( "cluster_files_directory", "database/pbs" ), self.root ) + self.cluster_files_directory = os.path.abspath( kwargs.get( "cluster_files_directory", "database/pbs" ) ) self.job_working_directory = resolve_path( kwargs.get( "job_working_directory", "database/job_working_directory" ), self.root ) self.outputs_to_working_directory = string_as_bool( kwargs.get( 'outputs_to_working_directory', False ) ) self.admin_pass = kwargs.get('admin_pass',"galaxy")

1 0

[hg] galaxy 1696: Make the PBS script/stdout/stderr directory co...
by Greg Von Kuster 13 Jan '09

13 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/aac0403bbe68 changeset: 1696:aac0403bbe68 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Jan 09 17:08:49 2009 -0500 description: Make the PBS script/stdout/stderr directory configurable. 3 file(s) affected in this change: lib/galaxy/config.py lib/galaxy/jobs/__init__.py lib/galaxy/jobs/runners/pbs.py diffs (93 lines): diff -r 94df4d059b19 -r aac0403bbe68 lib/galaxy/config.py --- a/lib/galaxy/config.py Fri Jan 09 15:59:29 2009 -0500 +++ b/lib/galaxy/config.py Fri Jan 09 17:08:49 2009 -0500 @@ -44,6 +44,7 @@ self.cluster_job_queue_workers = int( kwargs.get( "cluster_job_queue_workers", "5" ) ) self.job_scheduler_policy = kwargs.get("job_scheduler_policy", "FIFO") self.job_queue_cleanup_interval = int( kwargs.get("job_queue_cleanup_interval", "5") ) + self.cluster_files_directory = resolve_path( kwargs.get( "cluster_files_directory", "database/pbs" ), self.root ) self.job_working_directory = resolve_path( kwargs.get( "job_working_directory", "database/job_working_directory" ), self.root ) self.outputs_to_working_directory = string_as_bool( kwargs.get( 'outputs_to_working_directory', False ) ) self.admin_pass = kwargs.get('admin_pass',"galaxy") @@ -94,7 +95,7 @@ return default def check( self ): # Check that required directories exist - for path in self.root, self.file_path, self.tool_path, self.tool_data_path, self.template_path, self.job_working_directory: + for path in self.root, self.file_path, self.tool_path, self.tool_data_path, self.template_path, self.job_working_directory, self.cluster_files_directory: if not os.path.isdir( path ): raise ConfigurationError("Directory does not exist: %s" % path ) # Check that required files exist diff -r 94df4d059b19 -r aac0403bbe68 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Fri Jan 09 15:59:29 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Fri Jan 09 17:08:49 2009 -0500 @@ -1,4 +1,4 @@ -import logging, threading, sys, os, time, subprocess, string, tempfile, re, traceback +import logging, threading, sys, os, time, subprocess, string, tempfile, re, traceback, shutil from galaxy import util, model from galaxy.model import mapping @@ -396,6 +396,13 @@ # if the job was deleted, don't fail it if not job.state == model.Job.states.DELETED: for dataset_assoc in job.output_datasets: + if self.app.config.outputs_to_working_directory: + false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.id ) ) + if os.path.exists( false_path ): + shutil.move( false_path, dataset_assoc.dataset.file_name ) + log.debug( "fail(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) + else: + log.warning( "fail(): Missing output file in working directory: %s" % false_path ) dataset = dataset_assoc.dataset dataset.refresh() dataset.state = dataset.states.ERROR @@ -474,10 +481,10 @@ if self.app.config.outputs_to_working_directory: false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.id ) ) if os.path.exists( false_path ): - os.rename( false_path, dataset_assoc.dataset.file_name ) - log.debug( "Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) + shutil.move( false_path, dataset_assoc.dataset.file_name ) + log.debug( "finish(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) else: - log.warning( "Missing output file in working directory: %s" % false_path ) + log.warning( "finish(): Missing output file in working directory: %s" % false_path ) for dataset in dataset_assoc.dataset.dataset.history_associations: #need to update all associated output hdas, i.e. history was shared with job running dataset.blurb = 'done' dataset.peek = 'no peek' diff -r 94df4d059b19 -r aac0403bbe68 lib/galaxy/jobs/runners/pbs.py --- a/lib/galaxy/jobs/runners/pbs.py Fri Jan 09 15:59:29 2009 -0500 +++ b/lib/galaxy/jobs/runners/pbs.py Fri Jan 09 17:08:49 2009 -0500 @@ -166,8 +166,8 @@ raise Exception( "Connection to PBS server for submit failed" ) # define job attributes - ofile = "%s/database/pbs/%s.o" % (os.getcwd(), job_wrapper.job_id) - efile = "%s/database/pbs/%s.e" % (os.getcwd(), job_wrapper.job_id) + ofile = "%s/%s.o" % (self.app.config.cluster_files_directory, job_wrapper.job_id) + efile = "%s/%s.e" % (self.app.config.cluster_files_directory, job_wrapper.job_id) # If an application server is set, we're staging if self.app.config.pbs_application_server: @@ -201,7 +201,7 @@ script = pbs_symlink_template % (job_wrapper.galaxy_lib_dir, " ".join(job_wrapper.get_input_fnames() + job_wrapper.get_output_fnames()), self.app.config.pbs_stage_path, exec_dir, command_line) else: script = pbs_template % (job_wrapper.galaxy_lib_dir, exec_dir, command_line) - job_file = "%s/database/pbs/%s.sh" % (os.getcwd(), job_wrapper.job_id) + job_file = "%s/%s.sh" % (self.app.config.cluster_files_directory, job_wrapper.job_id) fh = file(job_file, "w") fh.write(script) fh.close() @@ -436,9 +436,9 @@ def recover( self, job, job_wrapper ): """Recovers jobs stuck in the queued/running state when Galaxy started""" pbs_job_state = PBSJobState() - pbs_job_state.ofile = "%s/database/pbs/%s.o" % (os.getcwd(), job.id) - pbs_job_state.efile = "%s/database/pbs/%s.e" % (os.getcwd(), job.id) - pbs_job_state.job_file = "%s/database/pbs/%s.sh" % (os.getcwd(), job.id) + pbs_job_state.ofile = "%s/%s.o" % (self.app.config.cluster_files_directory, job.id) + pbs_job_state.efile = "%s/%s.e" % (self.app.config.cluster_files_directory, job.id) + pbs_job_state.job_file = "%s/%s.sh" % (self.app.config.cluster_files_directory, job.id) pbs_job_state.job_id = str( job.job_runner_external_id ) pbs_job_state.runner_url = job_wrapper.tool.job_runner job_wrapper.command_line = job.command_line

1 0

[hg] galaxy 1698: Fix for data source tools that were broken due...
by Greg Von Kuster 13 Jan '09

13 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/88fe7fbd28da changeset: 1698:88fe7fbd28da user: Greg Von Kuster <greg(a)bx.psu.edu> date: Tue Jan 13 10:34:39 2009 -0500 description: Fix for data source tools that were broken due to the new "outputs_to_working_directory" config setting. Request parameters are now temporarily written to the output dataset if this config setting is either True or False. 1 file(s) affected in this change: lib/galaxy/tools/__init__.py diffs (43 lines): diff -r a6d6c2e4c021 -r 88fe7fbd28da lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py Fri Jan 09 17:18:18 2009 -0500 +++ b/lib/galaxy/tools/__init__.py Tue Jan 13 10:34:39 2009 -0500 @@ -1068,7 +1068,10 @@ wrap_values( input.cases[current].inputs, values ) elif isinstance( input, DataToolParameter ): input_values[ input.name ] = \ - DatasetFilenameWrapper( input_values[ input.name ], datatypes_registry = self.app.datatypes_registry, tool = self, name = input.name ) + DatasetFilenameWrapper( input_values[ input.name ], + datatypes_registry = self.app.datatypes_registry, + tool = self, + name = input.name ) else: input_values[ input.name ] = \ InputValueWrapper( input, input_values[ input.name ], param_dict ) @@ -1085,7 +1088,10 @@ # but this should be considered DEPRECATED, instead use: # $dataset.get_child( 'name' ).filename for name, data in input_datasets.items(): - param_dict[name] = DatasetFilenameWrapper( data, datatypes_registry = self.app.datatypes_registry, tool = self, name = name ) + param_dict[name] = DatasetFilenameWrapper( data, + datatypes_registry = self.app.datatypes_registry, + tool = self, + name = name ) if data: for child in data.children: param_dict[ "_CHILD___%s___%s" % ( name, child.designation ) ] = DatasetFilenameWrapper( child ) @@ -1263,8 +1269,13 @@ if data_type not in app.datatypes_registry.datatypes_by_extension: data_type = 'interval' data = app.datatypes_registry.change_datatype( data, data_type ) - # Store external data source's request parameters temporarily in output file - out = open( data.file_name, 'w' ) + # Store external data source's request parameters temporarily in output file. + # In case the config setting for "outputs_to_working_directory" is True, we must write to + # the DatasetFilenameWrapper object in the param_dict since it's "false_path" attribute + # is the temporary path to the output dataset ( until the job is run ). However, + # even if the "outputs_to_working_directory" setting is False, we can still open the file + # the same way for temporarily storing the request parameters. + out = open( str( param_dict.get( name ) ), 'w' ) for key, value in param_dict.items(): print >> out, '%s\t%s' % ( key, value ) out.close()

1 0

[hg] galaxy 1699: Add new tool config for flymine production ser...
by Greg Von Kuster 13 Jan '09

13 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/511c2b1966dc changeset: 1699:511c2b1966dc user: Greg Von Kuster <greg(a)bx.psu.edu> date: Tue Jan 13 10:55:08 2009 -0500 description: Add new tool config for flymine production server. 4 file(s) affected in this change: tool_conf.xml.main tool_conf.xml.sample tools/data_source/flymine.xml tools/data_source/flymine_test.xml diffs (68 lines): diff -r 88fe7fbd28da -r 511c2b1966dc tool_conf.xml.main --- a/tool_conf.xml.main Tue Jan 13 10:34:39 2009 -0500 +++ b/tool_conf.xml.main Tue Jan 13 10:55:08 2009 -0500 @@ -6,6 +6,7 @@ <tool file="data_source/ucsc_tablebrowser_archaea.xml" /> <tool file="data_source/microbial_import.xml" /> <tool file="data_source/biomart.xml" /> + <tool file="data_source/flymine.xml" /> <tool file="data_source/encode_db.xml" /> <tool file="data_source/epigraph_import.xml" /> </section> diff -r 88fe7fbd28da -r 511c2b1966dc tool_conf.xml.sample --- a/tool_conf.xml.sample Tue Jan 13 10:34:39 2009 -0500 +++ b/tool_conf.xml.sample Tue Jan 13 10:55:08 2009 -0500 @@ -11,6 +11,7 @@ <tool file="data_source/wormbase.xml" /> <tool file="data_source/wormbase_test.xml" /> <tool file="data_source/flymine.xml" /> + <tool file="data_source/flymine_test.xml" /> <tool file="data_source/encode_db.xml" /> <tool file="data_source/epigraph_import.xml" /> <tool file="data_source/epigraph_import_test.xml" /> diff -r 88fe7fbd28da -r 511c2b1966dc tools/data_source/flymine.xml --- a/tools/data_source/flymine.xml Tue Jan 13 10:34:39 2009 -0500 +++ b/tools/data_source/flymine.xml Tue Jan 13 10:55:08 2009 -0500 @@ -7,7 +7,7 @@ <tool name="Flymine" id="flymine" tool_type="data_source" URL_method="post"> <description>server</description> <command interpreter="python">data_source.py $output</command> - <inputs action="http://preview.flymine.org/preview/begin.do" check_values="false" method="get" target="_top"> + <inputs action="http://www.flymine.org" check_values="false" method="get" target="_top"> <display>go to Flymine server $GALAXY_URL</display> <param name="GALAXY_URL" type="baseurl" value="/tool_runner?tool_id=flymine" /> </inputs> diff -r 88fe7fbd28da -r 511c2b1966dc tools/data_source/flymine_test.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/data_source/flymine_test.xml Tue Jan 13 10:55:08 2009 -0500 @@ -0,0 +1,30 @@ +<?xml version="1.0"?> + +<tool name="Flymine test" id="flymine_test" tool_type="data_source" URL_method="post"> + <description>server</description> + <command interpreter="python">data_source.py $output</command> + <inputs action="http://preview.flymine.org/preview/begin.do" check_values="false" method="get" target="_top"> + <display>go to Flymine server $GALAXY_URL</display> + <param name="GALAXY_URL" type="baseurl" value="/tool_runner?tool_id=flymine" /> + </inputs> + <request_param_translation> + <request_param galaxy_name="URL" remote_name="URL" missing="" /> + <request_param galaxy_name="dbkey" remote_name="db" missing="?" /> + <request_param galaxy_name="organism" remote_name="organism" missing="" /> + <request_param galaxy_name="table" remote_name="table" missing="" /> + <request_param galaxy_name="description" remote_name="description" missing="" /> + <request_param galaxy_name="name" remote_name="name" missing="FlyMine query" /> + <request_param galaxy_name="info" remote_name="info" missing="" /> + <request_param galaxy_name="data_type" remote_name="data_type" missing="txt" /> + </request_param_translation> + <uihints minwidth="800"/> + <outputs> + <data name="output" format="txt" /> + </outputs> + <options sanitize="False" refresh="True"/> +</tool> +

1 0

Showing dataset state in History List View
by Assaf Gordon 13 Jan '09

13 Jan '09

Hello, I'd like to suggest a new feature: In the history list view, instead of just showing the 'size' of the history (= how many datasets are in the history) - show the state of each dataset. The reason is that some users run long jobs (or workflows with many steps) in several histories in parallel - and they want to quickly know: 1. which jobs are running, 2. which jobs are completed, 3. Were there any errors Currently, they have to switch to each history, and look at the state of the datasets. With this feature, all one needs to do is look at the history view. Using the same color keys for ok/queued/running/error states, users can quickly know: 1. If there's a grey box - some jobs are still queued. 2. If there are no grey boxes but some yellow boxes - some jobs are still running. 3. if there are no grey boxes and no yellow boxes - all jobs have been completed. 4. If there are red boxes - some jobs failed. Attached pictures illustrate the feature (at different states of jobs). To add this feature: Extract the attached 'list.mako.tar.gz' to GALAXY/templates/history (overriding the current list.mako). You'll also need to add the following function to GALAXY/lib/galaxy/model/__init__.py class History, line ~399 # returns number of datasets matching the requested state # Added by gordon, 24dec2008 def get_dataset_count( self, state ): count = 0 for data in self.datasets: if data.state == state and not data.deleted : count += 1 return count Another feature in this list.mako is the separation of 'switch to' link from the 'delete' and 'rename' links - The 'switch to' is much more important and frequently-used than the other two, and users have been complaining about the difficulty of clicking it (and accidentally clicking delete or rename). Comments are welcome, Gordon.

2 1