- galaxy-dev - lists.galaxyproject.org

New report page - find Datasets by ID
by Assaf Gordon 17 Jan '09

17 Jan '09

Hello, I'd like to suggest a small feature: A new report page that helps finding datasets information based on ID. This report allows getting all the information about a dataset ( user email, history name, dataset name, file path on server) based on either the Dataset-ID or the HistoryDatasetAssociation-ID. I think this tool is useful for Galaxy servers with small number of users (as it is in my case). It might be also useful for 'larger' Galaxies. This tool serves two common scenarios: 1. When a tool crashes, or generates and error, or just outputs something weird - if the tool has a log, or if I look at the galaxy log - I can easily see the input dataset that was used with this tool (e.g. 'dataset_1192.dat') - Now I want to know what was this file, which user used it, and for what (As I said - in a small Galaxy I personally know all the users). So I just go to this report page, enter 1192 as the File-ID, and get all the needed information. 2. When a user complains about something not working properly, or when I want to change a dataset in-place without creating a new dataset (again, probably something that happens only with on a small Galaxy) - I just ask the user for the number of the dataset, and with this report I quickly find out which file on the local disk correspond to the user's dataset. Regards, - Gordon.

1 0

[hg] galaxy 1709: More changes to mutability tool.
by gua110＠scofield.bx.psu.edu 15 Jan '09

15 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/90d04fcbcdf4 changeset: 1709:90d04fcbcdf4 user: guru date: Thu Jan 15 13:10:04 2009 -0500 description: More changes to mutability tool. 2 file(s) affected in this change: tools/regVariation/microsats_mutability.py tools/regVariation/microsats_mutability.xml diffs (99 lines): diff -r 78c2f403cb43 -r 90d04fcbcdf4 tools/regVariation/microsats_mutability.py --- a/tools/regVariation/microsats_mutability.py Thu Jan 15 10:49:01 2009 -0500 +++ b/tools/regVariation/microsats_mutability.py Thu Jan 15 13:10:04 2009 -0500 @@ -128,11 +128,13 @@ uniq_s_elems_2 = get_binned_lists(uniq_s_elems_2,s_bin_size) for pitem1 in uniq_elems_1: - repeats1 = [] - repeats2 = [] + #repeats1 = [] + #repeats2 = [] thresholds = [] if s_group_cols[0] != -1: #Sub-group by feature is not None for sitem1 in uniq_s_elems_1: + repeats1 = [] + repeats2 = [] if type(sitem1) == type(''): sitem1 = sitem1.strip() for bline in blk_lines: @@ -223,11 +225,13 @@ count1[str(pitem1)]=sum(repeats2) for pitem2 in uniq_elems_2: - repeats1 = [] - repeats2 = [] + #repeats1 = [] + #repeats2 = [] thresholds = [] if s_group_cols[0] != -1: #Sub-group by feature is not None for sitem2 in uniq_s_elems_2: + repeats1 = [] + repeats2 = [] if type(sitem2)==type(''): sitem2 = sitem2.strip() for bline in blk_lines: @@ -349,11 +353,10 @@ count = count1[key] mut = "%.2e" %(mut/num_generations) if region == 'align': - print >>fout, str(blk) + '\t'+seq1 + '\t' + start1+ '\t'+end1+ '\t'+seq2 + '\t'+start2+ '\t'+end2+ '\t'+key.strip()+ '\t'+str(mut) + '\t'+ str(count) + print >>fout, str(blk) + '\t'+seq1 + '\t' + seq2 + '\t' +key.strip()+ '\t'+str(mut) + '\t'+ str(count) elif region == 'win': fout.write("%s\t%s\t%s\t%s\n" %(blk,key.strip(),mut,count)) fout.flush() - #print >>fout, blk + '\t'+key.strip()+ '\t'+str(mut)+ '\t'+ str(count) #catch any remaining repeats, for instance if the orthologous position contained different repeat units for remaining_key in mut2.keys(): @@ -361,7 +364,7 @@ mut = "%.2e" %(mut/num_generations) count = count2[remaining_key] if region == 'align': - print >>fout, str(blk) + '\t'+seq1 + '\t' + start1+ '\t'+end1+ '\t'+seq2 + '\t'+start2+ '\t'+end2+ '\t'+remaining_key.strip()+ '\t'+str(mut)+ '\t'+ str(count) + print >>fout, str(blk) + '\t'+seq1 + '\t'+seq2 + '\t'+remaining_key.strip()+ '\t'+str(mut)+ '\t'+ str(count) elif region == 'win': fout.write("%s\t%s\t%s\t%s\n" %(blk,remaining_key.strip(),mut,count)) fout.flush() @@ -420,17 +423,9 @@ fix_strand = True) msatTree = quicksect.IntervalTree() for item in msats: - #print >>sys.stderr, item if type( item ) is GenomicInterval: msatTree.insert( item, msats.linenum, item.fields ) - """ - result = [] - msatTree.traverse(lambda node: result.append( node )) - for n in result: - print >>sys.stderr,n.other - print >>sys.stderr,msatTree.chroms - #sys.exit() - """ + for iline in fint: try: iline = iline.rstrip('\r\n') @@ -460,7 +455,7 @@ print "Skipped %d intervals as invalid." %(skipped) elif region == 'align': if s_group_cols[0] != -1: - print >>fout, "#Window\tSpecies_1\tWindow_Start\tWindow_End\tSpecies_2\tGroupby_Feature\tSubGroupby_Feature\tMutability\tCount" + print >>fout, "#Window\tSpecies_1\tSpecies_2\tGroupby_Feature\tSubGroupby_Feature\tMutability\tCount" else: print >>fout, "#Window\tSpecies_1\tWindow_Start\tWindow_End\tSpecies_2\tGroupby_Feature\tMutability\tCount" prev_bnum = -1 diff -r 78c2f403cb43 -r 90d04fcbcdf4 tools/regVariation/microsats_mutability.xml --- a/tools/regVariation/microsats_mutability.xml Thu Jan 15 10:49:01 2009 -0500 +++ b/tools/regVariation/microsats_mutability.xml Thu Jan 15 13:10:04 2009 -0500 @@ -106,6 +106,10 @@ This tool computes microsatellite mutability for the orthologous microsatellites fetched from 'Extract Orthologous Microsatellites from pair-wise alignments' tool. +Mutability is computed according to the method described in the following paper: + +*Webster et al., Microsatellite evolution inferred from human-chimpanzee genomic sequence alignments, Proc Natl Acad Sci 2002 June 25; 99(13): 8748-8753* + ----- .. class:: warningmark

1 0

Adding the 'name' content of the selection parameter
by Assaf Gordon 15 Jan '09

15 Jan '09

Hello, Continuing the discussion from Dec. 19th, http://mail.bx.psu.edu/pipermail/galaxy-user/2008-December/000422.html The following hack adds an additional key/value which contains the 'name' of the selected option in a <select> input parameter. With this hack, the following tool configuration is possible (note the output/label option): ========================== <tool> .. <inputs> <input format="fasta" type="data" name="input"> <input type="select" name="database" label="Database"> <option value="/home/gordon/long/path/hg18.fa">Human (hg18)</option> <option value="/home/gordon/long/path/dm3.fa">Fly (dm3)</option> <option value="/home/gordon/long/path/mm9.fa">Mouse (mm9)</option> </input> </inputs> <outputs>  <data format="txt" name="output" label="Blat $input1 on $database" />  <data format="txt" name="output" label="Blat $input1 on $database_name" /> </outputs> ========================== Currently, using "$database" in the output/label part will put the *value* of the selected option (e.g. "/home/gordon/long/path/hg18.fa") in the dataset's name. This doesn't really help the user understand what the dataset contains. With this hack, for each <select> input, a new variable is created (with the "_name" suffix) which will contain the *name* of the selected option (e.g. "Human (hg18)"). For example, if the user selected the second option, Then "$database" will contain "/home/gordon/long/path/dm3.fa", and "$database_name" will contain "Fly (dm3)". I'm well aware this is an ugly hack. A better way would be to have each parameter (in the incoming dictionary) be an object (e.g. ToolParameter) and not a string. Then we could write something like "$database.name" or "$database.value". The hack contains two patches. The first add support for 'value_to_display_text' with dynamic options, in "Parameters/Basic.py": ============================================================== --- galaxy_prod/lib/galaxy/tools/parameters/basic.py +++ galaxy_devel/lib/galaxy/tools/parameters/basic.py @@ -576,7 +576,7 @@ elif len( value ) == 1: value = value[0] return value - def value_to_display_text( self, value, app ): + def value_to_display_text( self, value, app, other_values={} ): if isinstance( value, UnvalidatedValue ): suffix = "\n(value not yet validated)" value = value.value @@ -584,10 +584,11 @@ suffix = "" if not isinstance( value, list ): value = [ value ] - # FIXME: Currently only translating values back to labels if they - # are not dynamic if self.is_dynamic: - rval = map( str, value ) + rval = [ ] + for name in self.options.get_options ( app , other_values ) : + if ( name[1] in value ): + rval.append ( name[0] ) else: options = list( self.static_options ) rval = [] ============================================================== The second patch iterates the tool's parameters, and for each <select> type parameter, adds the additional key/value for the selected option: ============================================================== --- galaxy_prod/lib/galaxy/tools/actions/__init__.py +++ galaxy_devel/lib/galaxy/tools/actions/__init__.py @@ -96,6 +96,18 @@ on_text = "" # Add the dbkey to the incoming parameters incoming[ "dbkey" ] = input_dbkey + + ##28dec2008, gordon + ## For every 'Select' parameter, we get only the 'value' part of the selection from the HTTP request. + ## The following code gets the 'name' part for the selected option + selection_parameter_names = {} + for param_name, param_selected_value in incoming.iteritems(): + param_obj = tool.get_param ( param_name ) + if isinstance( param_obj, basic.SelectToolParameter): + param_selected_name = param_obj.value_to_display_text( param_selected_value, trans, incoming ) + selection_parameter_names [ param_name + "_name" ] = param_selected_name.replace("\n","") + incoming.update( selection_parameter_names ) + # Keep track of parent / child relationships, we'll create all the # datasets first, then create the associations parent_to_child_pairs = [] ============================================================== I tried to make the changes as non intrusive as possible. I hope the added 'other_value' parameter doesn't affect other places that call 'value_to_display_text' - but I'm not quite sure about it (I'm still very new to python). I've tested this hack with several tools, and it seems to work (or at least not crash horribly). Comments are welcomed, Gordon.

2 1

request: add newlines to dataset's info field
by Assaf Gordon 15 Jan '09

15 Jan '09

Hello, I'd like to request/suggest a tiny change in the dataset's info field. Whenever a tool outputs some lines into STDOUT, they are displayed in the dataset's info field. Currently, if the info field contains several lines, they are displayed as a single line (because HTML ignores new lines). The following patch causes the info field to be displayed correctly. ================================================================= --- galaxy_prod/lib/galaxy/datatypes/data.py +++ galaxy_devel/lib/galaxy/datatypes/data.py @@ -132,7 +132,7 @@ def display_info(self, dataset): """Returns formated html of dataset info""" try: - return escape(dataset.info) + return escape(dataset.info).replace("\n", "<br>") except: return "info unavailable" def validate(self, dataset): ================================================================== Thanks, gordon.

2 1

[hg] galaxy 1708: Change History.copy() back to use datasets rat...
by Greg Von Kuster 15 Jan '09

15 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/78c2f403cb43 changeset: 1708:78c2f403cb43 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Jan 15 10:49:01 2009 -0500 description: Change History.copy() back to use datasets rather than active_datasets. Add checks in set_peek() to properly set peek on purged datasets. Fixes sharing a history that contains purged datasets. 9 file(s) affected in this change: lib/galaxy/datatypes/data.py lib/galaxy/datatypes/genetics.py lib/galaxy/datatypes/images.py lib/galaxy/datatypes/interval.py lib/galaxy/datatypes/qualityscore.py lib/galaxy/datatypes/sequence.py lib/galaxy/datatypes/xml.py lib/galaxy/model/__init__.py tools/data_source/echo.xml diffs (441 lines): diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/data.py --- a/lib/galaxy/datatypes/data.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/data.py Thu Jan 15 10:49:01 2009 -0500 @@ -103,8 +103,12 @@ return False def set_peek( self, dataset ): """Set the peek and blurb text""" - dataset.peek = '' - dataset.blurb = 'data' + if not dataset.dataset.purged: + dataset.peek = '' + dataset.blurb = 'data' + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): """Create HTML table, used for displaying peek""" out = ['<table cellspacing="0" cellpadding="3">'] @@ -275,19 +279,27 @@ return 'text/plain' def set_peek( self, dataset, line_count=None ): - dataset.peek = get_file_peek( dataset.file_name ) - if line_count is None: - dataset.blurb = "%s lines" % util.commaify( str( get_line_count( dataset.file_name ) ) ) + if not dataset.dataset.purged: + # The file must exist on disk for the get_file_peek() method + dataset.peek = get_file_peek( dataset.file_name ) + if line_count is None: + dataset.blurb = "%s lines" % util.commaify( str( get_line_count( dataset.file_name ) ) ) + else: + dataset.blurb = "%s lines" % util.commaify( str( line_count ) ) else: - dataset.blurb = "%s lines" % util.commaify( str( line_count ) ) + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' class Binary( Data ): """Binary data""" - def set_peek( self, dataset ): """Set the peek and blurb text""" - dataset.peek = 'binary data' - dataset.blurb = 'data' + if not dataset.dataset.purged: + dataset.peek = 'binary data' + dataset.blurb = 'data' + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def get_test_fname( fname ): """Returns test data filename""" diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/genetics.py --- a/lib/galaxy/datatypes/genetics.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/genetics.py Thu Jan 15 10:49:01 2009 -0500 @@ -43,11 +43,14 @@ def set_peek( self, dataset ): """Set the peek and blurb text""" - dataset.peek = data.get_file_peek( dataset.file_name ) - ## dataset.peek = self.make_html_table( dataset.peek ) - dataset.blurb = util.commaify( str( data.get_line_count( dataset.file_name ) ) ) + " rows" - #i don't think set_meta should not be called here, it should be called separately - self.set_meta( dataset ) + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + dataset.blurb = util.commaify( str( data.get_line_count( dataset.file_name ) ) ) + " rows" + #i don't think set_meta should not be called here, it should be called separately + self.set_meta( dataset ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def get_estimated_display_viewport( self, dataset ): """Return a chrom, start, stop tuple for viewing a file.""" @@ -130,8 +133,12 @@ file_ext="snpmatrix" def set_peek( self, dataset ): - dataset.peek = "Binary RGenetics file" - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = "Binary RGenetics file" + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def sniff( self, filename ): """ """ diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/images.py --- a/lib/galaxy/datatypes/images.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/images.py Thu Jan 15 10:49:01 2009 -0500 @@ -14,9 +14,13 @@ """Class describing an ab1 binary sequence file""" file_ext = "ab1" def set_peek( self, dataset ): - export_url = "/history_add_to?"+urlencode({'history_id':dataset.history_id,'ext':'ab1','name':'ab1 sequence','info':'Sequence file','dbkey':dataset.dbkey}) - dataset.peek = "Binary ab1 sequence file" - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + export_url = "/history_add_to?" + urlencode({'history_id':dataset.history_id,'ext':'ab1','name':'ab1 sequence','info':'Sequence file','dbkey':dataset.dbkey}) + dataset.peek = "Binary ab1 sequence file" + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: return dataset.peek @@ -27,9 +31,13 @@ """Class describing an scf binary sequence file""" file_ext = "scf" def set_peek( self, dataset ): - export_url = "/history_add_to?"+urlencode({'history_id':dataset.history_id,'ext':'scf','name':'scf sequence','info':'Sequence file','dbkey':dataset.dbkey}) - dataset.peek = "Binary scf sequence file" - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + export_url = "/history_add_to?" + urlencode({'history_id':dataset.history_id,'ext':'scf','name':'scf sequence','info':'Sequence file','dbkey':dataset.dbkey}) + dataset.peek = "Binary scf sequence file" + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: return dataset.peek @@ -40,10 +48,14 @@ """Class describing a zip archive of binary sequence files""" file_ext = "binseq.zip" def set_peek( self, dataset ): - zip_file = zipfile.ZipFile( dataset.file_name, "r" ) - num_files = len( zip_file.namelist() ) - dataset.peek = "Archive of %s binary sequence files" % ( str( num_files ) ) - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + zip_file = zipfile.ZipFile( dataset.file_name, "r" ) + num_files = len( zip_file.namelist() ) + dataset.peek = "Archive of %s binary sequence files" % ( str( num_files ) ) + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: return dataset.peek @@ -57,10 +69,14 @@ """Class describing a zip archive of text sequence files""" file_ext = "txtseq.zip" def set_peek( self, dataset ): - zip_file = zipfile.ZipFile( dataset.file_name, "r" ) - num_files = len( zip_file.namelist() ) - dataset.peek = "Archive of %s text sequence files" % ( str( num_files ) ) - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + zip_file = zipfile.ZipFile( dataset.file_name, "r" ) + num_files = len( zip_file.namelist() ) + dataset.peek = "Archive of %s text sequence files" % ( str( num_files ) ) + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: return dataset.peek @@ -73,8 +89,12 @@ class Image( data.Data ): """Class describing an image""" def set_peek( self, dataset ): - dataset.peek = 'Image in %s format' % dataset.extension - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = 'Image in %s format' % dataset.extension + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def create_applet_tag_peek( class_name, archive, params ): text = """ @@ -104,18 +124,22 @@ """Class describing a GMAJ Applet""" file_ext = "gmaj.zip" def set_peek( self, dataset ): - params = { - "bundle":"display?id=%s&tofile=yes&toext=.zip" % dataset.id, - "buttonlabel": "Launch GMAJ", - "nobutton": "false", - "urlpause" :"100", - "debug": "false", - "posturl": "history_add_to?%s" % urlencode( { 'history_id': dataset.history_id, 'ext': 'maf', 'name': 'GMAJ Output on data %s' % dataset.hid, 'info': 'Added by GMAJ', 'dbkey': dataset.dbkey } ) - } - class_name = "edu.psu.bx.gmaj.MajApplet.class" - archive = "/static/gmaj/gmaj.jar" - dataset.peek = create_applet_tag_peek( class_name, archive, params ) - dataset.blurb = 'GMAJ Multiple Alignment Viewer' + if not dataset.dataset.purged: + params = { + "bundle":"display?id=%s&tofile=yes&toext=.zip" % dataset.id, + "buttonlabel": "Launch GMAJ", + "nobutton": "false", + "urlpause" :"100", + "debug": "false", + "posturl": "history_add_to?%s" % urlencode( { 'history_id': dataset.history_id, 'ext': 'maf', 'name': 'GMAJ Output on data %s' % dataset.hid, 'info': 'Added by GMAJ', 'dbkey': dataset.dbkey } ) + } + class_name = "edu.psu.bx.gmaj.MajApplet.class" + archive = "/static/gmaj/gmaj.jar" + dataset.peek = create_applet_tag_peek( class_name, archive, params ) + dataset.blurb = 'GMAJ Multiple Alignment Viewer' + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: return dataset.peek @@ -147,8 +171,12 @@ """Class describing an html file""" file_ext = "html" def set_peek( self, dataset ): - dataset.peek = "HTML file" - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = "HTML file" + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def get_mime(self): """Returns the mime type of the datatype""" return 'text/html' @@ -176,17 +204,21 @@ """Class describing a LAJ Applet""" file_ext = "laj" def set_peek( self, dataset ): - params = { - "alignfile1": "display?id=%s" % dataset.id, - "buttonlabel": "Launch LAJ", - "title": "LAJ in Galaxy", - "posturl": "history_add_to?%s" % urlencode( { 'history_id': dataset.history_id, 'ext': 'lav', 'name': 'LAJ Output', 'info': 'Added by LAJ', 'dbkey': dataset.dbkey } ), - "noseq": "true" - } - class_name = "edu.psu.cse.bio.laj.LajApplet.class" - archive = "/static/laj/laj.jar" - dataset.peek = create_applet_tag_peek( class_name, archive, params ) - dataset.blurb = 'LAJ Multiple Alignment Viewer' + if not dataset.dataset.purged: + params = { + "alignfile1": "display?id=%s" % dataset.id, + "buttonlabel": "Launch LAJ", + "title": "LAJ in Galaxy", + "posturl": "history_add_to?%s" % urlencode( { 'history_id': dataset.history_id, 'ext': 'lav', 'name': 'LAJ Output', 'info': 'Added by LAJ', 'dbkey': dataset.dbkey } ), + "noseq": "true" + } + class_name = "edu.psu.cse.bio.laj.LajApplet.class" + archive = "/static/laj/laj.jar" + dataset.peek = create_applet_tag_peek( class_name, archive, params ) + dataset.blurb = 'LAJ Multiple Alignment Viewer' + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: return dataset.peek diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/interval.py --- a/lib/galaxy/datatypes/interval.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/interval.py Thu Jan 15 10:49:01 2009 -0500 @@ -59,11 +59,15 @@ def set_peek( self, dataset, line_count=None ): """Set the peek and blurb text""" - dataset.peek = data.get_file_peek( dataset.file_name ) - if line_count is None: - dataset.blurb = "%s regions" % util.commaify( str( data.get_line_count( dataset.file_name ) ) ) + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + if line_count is None: + dataset.blurb = "%s regions" % util.commaify( str( data.get_line_count( dataset.file_name ) ) ) + else: + dataset.blurb = "%s regions" % util.commaify( str( line_count ) ) else: - dataset.blurb = "%s regions" % util.commaify( str( line_count ) ) + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def set_meta( self, dataset, overwrite = True, first_line_is_header = False, **kwd ): Tabular.set_meta( self, dataset, overwrite = overwrite, skip = 0 ) diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/qualityscore.py --- a/lib/galaxy/datatypes/qualityscore.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/qualityscore.py Thu Jan 15 10:49:01 2009 -0500 @@ -16,11 +16,15 @@ file_ext = "qual" def set_peek( self, dataset, line_count=None ): - dataset.peek = data.get_file_peek( dataset.file_name ) - if line_count is None: - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + if line_count is None: + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.blurb = "%s lines, Quality score file" % util.commaify( str( line_count ) ) else: - dataset.blurb = "%s lines, Quality score file" % util.commaify( str( line_count ) ) + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def display_peek(self, dataset): try: diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/sequence.py --- a/lib/galaxy/datatypes/sequence.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/sequence.py Thu Jan 15 10:49:01 2009 -0500 @@ -31,8 +31,12 @@ file_ext = "fasta" def set_peek( self, dataset ): - dataset.peek = data.get_file_peek( dataset.file_name ) - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def sniff( self, filename ): """ @@ -86,8 +90,12 @@ file_ext = "csfasta" def set_peek( self, dataset ): - dataset.peek = data.get_file_peek( dataset.file_name ) - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def sniff( self, filename ): """ @@ -108,8 +116,12 @@ file_ext = "fastqsolexa" def set_peek( self, dataset ): - dataset.peek = data.get_file_peek( dataset.file_name ) - dataset.blurb = data.nice_size( dataset.get_size() ) + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + dataset.blurb = data.nice_size( dataset.get_size() ) + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def sniff( self, filename ): """ diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/datatypes/xml.py --- a/lib/galaxy/datatypes/xml.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/datatypes/xml.py Thu Jan 15 10:49:01 2009 -0500 @@ -12,8 +12,12 @@ file_ext = "blastxml" def set_peek( self, dataset ): """Set the peek and blurb text""" - dataset.peek = data.get_file_peek( dataset.file_name ) - dataset.blurb = 'NCBI Blast XML data' + if not dataset.dataset.purged: + dataset.peek = data.get_file_peek( dataset.file_name ) + dataset.blurb = 'NCBI Blast XML data' + else: + dataset.peek = 'file does not exist' + dataset.blurb = 'file purged from disk' def sniff( self, filename ): """ Determines whether the file is blastxml diff -r 4cf0aa26070f -r 78c2f403cb43 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py Thu Jan 15 10:42:22 2009 -0500 +++ b/lib/galaxy/model/__init__.py Thu Jan 15 10:49:01 2009 -0500 @@ -287,14 +287,26 @@ return self.datatype.find_conversion_destination( self, accepted_formats, datatypes_registry, **kwd ) def copy( self, copy_children = False, parent_id = None ): - des = HistoryDatasetAssociation( hid=self.hid, name=self.name, info=self.info, blurb=self.blurb, peek=self.peek, extension=self.extension, dbkey=self.dbkey, dataset = self.dataset, visible=self.visible, deleted=self.deleted, parent_id=parent_id, copied_from_history_dataset_association = self ) + des = HistoryDatasetAssociation( hid=self.hid, + name=self.name, + info=self.info, + blurb=self.blurb, + peek=self.peek, + extension=self.extension, + dbkey=self.dbkey, + dataset=self.dataset, + visible=self.visible, + deleted=self.deleted, + parent_id=parent_id, + copied_from_history_dataset_association=self ) des.flush() des.set_size() des.metadata = self.metadata #need to set after flushed, as MetadataFiles require dataset.id if copy_children: for child in self.children: child_copy = child.copy( copy_children = copy_children, parent_id = des.id ) - des.set_peek() #in some instances peek relies on dataset_id, i.e. gmaj.zip for viewing MAFs + # In some instances peek relies on dataset_id ( e.g., gmaj.zip for viewing MAFs ) + des.set_peek() des.flush() return des @@ -376,7 +388,7 @@ des.flush() des.name = self.name des.user_id = self.user_id - for data in self.active_datasets: + for data in self.datasets: new_data = data.copy( copy_children = True ) des.add_dataset( new_data ) new_data.flush() diff -r 4cf0aa26070f -r 78c2f403cb43 tools/data_source/echo.xml --- a/tools/data_source/echo.xml Thu Jan 15 10:42:22 2009 -0500 +++ b/tools/data_source/echo.xml Thu Jan 15 10:49:01 2009 -0500 @@ -6,14 +6,18 @@ echoes parameters </description> - <command interpreter="python">echo.py $input $output </command> + <command interpreter="python">echo.py $input $database $output </command> <inputs> <param format="tabular" name="input" type="data" label="Input stuff"/> + <param type="select" name="database" label="Database"> + <option value="alignseq.loc">Human (hg18)</option> + <option value="faseq.loc">Fly (dm3)</option> + </param> </inputs> <outputs> - <data format="input" name="output" /> + <data format="input" name="output" label="Blat on ${database.value_label}" /> </outputs> </tool>

1 0

[hg] galaxy 1706: Add a SelectToolParameterWrapper object and a ...
by Greg Von Kuster 15 Jan '09

15 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/cfe79f4c6673 changeset: 1706:cfe79f4c6673 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Jan 14 17:20:22 2009 -0500 description: Add a SelectToolParameterWrapper object and a wrap method to Tool.execute() that allows for things like the following in tool configs: <outputs> <data format="input" name="output" label="Blat on {<input_param>.value_label}" /> </outputs> Also replace EOL chars with <br> in a tool's output data.info so that they are displayed in HTML. 3 file(s) affected in this change: lib/galaxy/datatypes/data.py lib/galaxy/tools/__init__.py lib/galaxy/tools/actions/__init__.py diffs (108 lines): diff -r 4a067f42c725 -r cfe79f4c6673 lib/galaxy/datatypes/data.py --- a/lib/galaxy/datatypes/data.py Wed Jan 14 16:15:51 2009 -0500 +++ b/lib/galaxy/datatypes/data.py Wed Jan 14 17:20:22 2009 -0500 @@ -132,7 +132,8 @@ def display_info(self, dataset): """Returns formated html of dataset info""" try: - return escape(dataset.info) + # Change new line chars to html + return escape( dataset.info ).replace( "\r", "\n" ).replace( "\n", "<br>" ) except: return "info unavailable" def validate(self, dataset): diff -r 4a067f42c725 -r cfe79f4c6673 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py Wed Jan 14 16:15:51 2009 -0500 +++ b/lib/galaxy/tools/__init__.py Wed Jan 14 17:20:22 2009 -0500 @@ -1072,10 +1072,11 @@ datatypes_registry = self.app.datatypes_registry, tool = self, name = input.name ) + elif isinstance( input, SelectToolParameter ): + input_values[ input.name ] = SelectToolParameterWrapper( input, input_values[ input.name ], self.app ) else: - input_values[ input.name ] = \ - InputValueWrapper( input, input_values[ input.name ], param_dict ) - # HACK: only wrap if check_values is false, this deals with external + input_values[ input.name ] = InputValueWrapper( input, input_values[ input.name ], param_dict ) + # HACK: only wrap if check_values is not false, this deals with external # tools where the inputs don't even get passed through. These # tools (e.g. UCSC) should really be handled in a special way. if self.check_values: @@ -1416,7 +1417,21 @@ return self.input.to_param_dict_string( self.value, self._other_values ) def __getattr__( self, key ): return getattr( self.value, key ) - + +class SelectToolParameterWrapper( object ): + """ + Wraps a SelectTooParameter so that __str__ returns the selected value, but all other + attributes are accessible. + """ + def __init__( self, input, value, app ): + self.input = input + self.value = value + self.input.value_label = input.value_to_display_text( value, app ) + def __str__( self ): + return self.input.to_param_dict_string( self.value ) + def __getattr__( self, key ): + return getattr( self.input, key ) + class DatasetFilenameWrapper( object ): """ Wraps a dataset so that __str__ returns the filename, but all other diff -r 4a067f42c725 -r cfe79f4c6673 lib/galaxy/tools/actions/__init__.py --- a/lib/galaxy/tools/actions/__init__.py Wed Jan 14 16:15:51 2009 -0500 +++ b/lib/galaxy/tools/actions/__init__.py Wed Jan 14 17:20:22 2009 -0500 @@ -1,9 +1,11 @@ from galaxy.util.bunch import Bunch from galaxy.tools.parameters import * +from galaxy.tools.parameters.grouping import * from galaxy.util.template import fill_template from galaxy.util.none_like import NoneDataset from galaxy.web import url_for from galaxy.jobs import JOB_OK +import galaxy.tools import logging log = logging.getLogger( __name__ ) @@ -65,6 +67,26 @@ return input_datasets def execute(self, tool, trans, incoming={}, set_output_hid=True ): + def wrap_values( inputs, input_values ): + # Wrap tool inputs as necessary + for input in inputs.itervalues(): + if isinstance( input, Repeat ): + for d in input_values[ input.name ]: + wrap_values( input.inputs, d ) + elif isinstance( input, Conditional ): + values = input_values[ input.name ] + current = values["__current_case__"] + wrap_values( input.cases[current].inputs, values ) + elif isinstance( input, DataToolParameter ): + input_values[ input.name ] = \ + galaxy.tools.DatasetFilenameWrapper( input_values[ input.name ], + datatypes_registry = trans.app.datatypes_registry, + tool = tool, + name = input.name ) + elif isinstance( input, SelectToolParameter ): + input_values[ input.name ] = galaxy.tools.SelectToolParameterWrapper( input, input_values[ input.name ], tool.app ) + else: + input_values[ input.name ] = galaxy.tools.InputValueWrapper( input, input_values[ input.name ], incoming ) out_data = {} # Collect any input datasets from the incoming parameters inp_data = self.collect_input_datasets( tool, incoming, trans ) @@ -151,6 +173,11 @@ # Set output label if output.label: params = dict( incoming ) + # wrapping the params allows the tool config to contain things like + # <outputs> + # <data format="input" name="output" label="Blat on ${<input_param>.name}" /> + # </outputs> + wrap_values( tool.inputs, params ) params['tool'] = tool params['on_string'] = on_text data.name = fill_template( output.label, context=params )

1 0

[hg] galaxy 1707: Fixed a bug in microsatellite mutability tool.
by Greg Von Kuster 15 Jan '09

15 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/4cf0aa26070f changeset: 1707:4cf0aa26070f user: guru date: Thu Jan 15 10:42:22 2009 -0500 description: Fixed a bug in microsatellite mutability tool. 1 file(s) affected in this change: tools/regVariation/microsats_mutability.py diffs (23 lines): diff -r cfe79f4c6673 -r 4cf0aa26070f tools/regVariation/microsats_mutability.py --- a/tools/regVariation/microsats_mutability.py Wed Jan 14 17:20:22 2009 -0500 +++ b/tools/regVariation/microsats_mutability.py Thu Jan 15 10:42:22 2009 -0500 @@ -409,19 +409,9 @@ blk=0 win=0 linestr="" - ff=open("junkmabc","w") if region == 'win': - """ - sorted_infile = tempfile.NamedTemporaryFile() - if winspecies == 1: - cmdline = "sort -n -k 3"+" -o "+sorted_infile.name+" "+infile - elif winspecies == 2: - cmdline = "sort -n -k 10"+" -o "+sorted_infile.name+" "+infile - os.system(cmdline) - print >>ff, "Finished sorting" - """ msats = NiceReaderWrapper( fileinput.FileInput( infile ), chrom_col = speciesind, start_col = speciesind+1,

1 0

[hg] galaxy 1705: When copying a history, only copy active_datas...
by Greg Von Kuster 14 Jan '09

14 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/4a067f42c725 changeset: 1705:4a067f42c725 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Jan 14 16:15:51 2009 -0500 description: When copying a history, only copy active_datasets - fixes sharing a history containing purged datasets. 1 file(s) affected in this change: lib/galaxy/model/__init__.py diffs (12 lines): diff -r 941266327bc3 -r 4a067f42c725 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py Wed Jan 14 10:49:14 2009 -0500 +++ b/lib/galaxy/model/__init__.py Wed Jan 14 16:15:51 2009 -0500 @@ -376,7 +376,7 @@ des.flush() des.name = self.name des.user_id = self.user_id - for data in self.datasets: + for data in self.active_datasets: new_data = data.copy( copy_children = True ) des.add_dataset( new_data ) new_data.flush()

1 0

[hg] galaxy 1704: Improvements to job recovery and clarification...
by Nate Coraor 14 Jan '09

14 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/941266327bc3 changeset: 1704:941266327bc3 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 14 10:49:14 2009 -0500 description: Improvements to job recovery and clarification of the false path's job id 2 file(s) affected in this change: lib/galaxy/jobs/__init__.py lib/galaxy/tools/__init__.py diffs (79 lines): diff -r c259ce9cb50d -r 941266327bc3 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Wed Jan 14 08:50:31 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Wed Jan 14 10:49:14 2009 -0500 @@ -111,19 +111,19 @@ def __check_jobs_at_startup( self ): """ - Checks all jobs that are in the 'running' or 'queued' state in the - database and requeues or cleans up as necessary. Only run as the + Checks all jobs that are in the 'new', 'queued' or 'running' state in + the database and requeues or cleans up as necessary. Only run as the job manager starts. """ model = self.app.model - # Jobs in the NEW state won't be requeued unless we're tracking in the database - if not self.track_jobs_in_database: - for job in model.Job.filter( model.Job.c.state==model.Job.states.NEW ).all(): - log.debug( "no runner: %s is still in new state, adding to the jobs queue" %job.id ) + for job in model.Job.filter( model.Job.c.state==model.Job.states.NEW ).all(): + log.debug( "no runner: %s is still in new state, adding to the jobs queue" %job.id ) + self.queue.put( ( job.id, job.tool_id ) ) + for job in model.Job.filter( (model.Job.c.state == model.Job.states.RUNNING) | (model.Job.c.state == model.Job.states.QUEUED) ).all(): + if job.job_runner_name is None: + log.debug( "no runner: %s is still in queued state, adding to the jobs queue" %job.id ) self.queue.put( ( job.id, job.tool_id ) ) - for job in model.Job.filter( (model.Job.c.state == model.Job.states.RUNNING) | (model.Job.c.state == model.Job.states.QUEUED) ).all(): - if job.job_runner_name is not None: - # why are we passing the queue to the wrapper? + else: job_wrapper = JobWrapper( job, self.app.toolbox.tools_by_id[ job.tool_id ], self ) self.dispatcher.recover( job, job_wrapper ) @@ -383,12 +383,12 @@ if not job.state == model.Job.states.DELETED: for dataset_assoc in job.output_datasets: if self.app.config.outputs_to_working_directory: - false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.id ) ) - if os.path.exists( false_path ): + false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.dataset.id ) ) + try: shutil.move( false_path, dataset_assoc.dataset.file_name ) log.debug( "fail(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) - else: - log.warning( "fail(): Missing output file in working directory: %s" % false_path ) + except ( IOError, OSError ), e: + log.error( "fail(): Missing output file in working directory: %s" % e ) dataset = dataset_assoc.dataset dataset.refresh() dataset.state = dataset.states.ERROR @@ -453,12 +453,13 @@ job.state = 'ok' for dataset_assoc in job.output_datasets: if self.app.config.outputs_to_working_directory: - false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.id ) ) - if os.path.exists( false_path ): + false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.dataset.id ) ) + try: shutil.move( false_path, dataset_assoc.dataset.file_name ) log.debug( "finish(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) - else: - log.warning( "finish(): Missing output file in working directory: %s" % false_path ) + except ( IOError, OSError ): + self.fail( "The job's output dataset(s) could not be read" ) + return for dataset in dataset_assoc.dataset.dataset.history_associations: #need to update all associated output hdas, i.e. history was shared with job running dataset.blurb = 'done' dataset.peek = 'no peek' diff -r c259ce9cb50d -r 941266327bc3 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py Wed Jan 14 08:50:31 2009 -0500 +++ b/lib/galaxy/tools/__init__.py Wed Jan 14 10:49:14 2009 -0500 @@ -1098,7 +1098,7 @@ for name, data in output_datasets.items(): # Write outputs to the working directory (for security purposes) if desired. if self.app.config.outputs_to_working_directory and working_directory is not None: - false_path = os.path.abspath( os.path.join( working_directory, "galaxy_dataset_%d.dat" % data.id ) ) + false_path = os.path.abspath( os.path.join( working_directory, "galaxy_dataset_%d.dat" % data.dataset.id ) ) param_dict[name] = DatasetFilenameWrapper( data, false_path = false_path ) open( false_path, 'w' ).close() else:

1 0

[hg] galaxy 1702: Set the working directory in the wrapper when ...
by Nate Coraor 14 Jan '09

14 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/b868c8d902d2 changeset: 1702:b868c8d902d2 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Jan 14 08:43:00 2009 -0500 description: Set the working directory in the wrapper when recovering jobs at startup. 1 file(s) affected in this change: lib/galaxy/jobs/__init__.py diffs (12 lines): diff -r c3a20d503bd3 -r b868c8d902d2 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Tue Jan 13 12:18:08 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Wed Jan 14 08:43:00 2009 -0500 @@ -125,6 +125,8 @@ if job.job_runner_name is not None: # why are we passing the queue to the wrapper? job_wrapper = JobWrapper( job, self.app.toolbox.tools_by_id[ job.tool_id ], self ) + job_wrapper.working_directory = \ + os.path.join( self.app.config.job_working_directory, str( job.id ) ) self.dispatcher.recover( job, job_wrapper ) def __monitor( self ):

1 0