- galaxy-dev - lists.galaxyproject.org

[hg] galaxy 1717: Update welcome page with Anton's copy from sec...
by Nate Coraor 22 Jan '09

22 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/4004ab767cf6 changeset: 1717:4004ab767cf6 user: Nate Coraor <nate(a)bx.psu.edu> date: Thu Jan 22 12:29:44 2009 -0500 description: Update welcome page with Anton's copy from security 1 file(s) affected in this change: static/welcome.html diffs (116 lines): diff -r 19d18731ce19 -r 4004ab767cf6 static/welcome.html --- a/static/welcome.html Thu Jan 22 11:22:39 2009 -0500 +++ b/static/welcome.html Thu Jan 22 12:29:44 2009 -0500 @@ -6,48 +6,80 @@ <meta name="generator" content="Docutils 0.3.9: http://docutils.sourceforge.net/" /> <title></title> <link rel="stylesheet" href="style/base.css" type="text/css" /> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta name="generator" content="Docutils 0.3.9: http://docutils.sourceforge.net/" /> + <title></title> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> + <meta name="generator" content="Docutils 0.6: http://docutils.sourceforge.net/"/> + <script src="http://www.apple.com/library/quicktime/scripts/ac_quicktime.js" language="JavaScript" type="text/javascript"></script> + <script src="http://www.apple.com/library/quicktime/scripts/qtp_library.js" language="JavaScript" type="text/javascript"></script> + <link href="http://www.apple.com/library/quicktime/stylesheets/qtp_library.css" rel="StyleSheet" type="text/css" /> + <link rel="stylesheet" href="style/base.css" type="text/css" /> + </head> + </head> <body> <div class="document"> -<div class="warningmessage"> - <strong>Galaxy and 2008 Meeting Season</strong> - <hr> - <ul> - <li><a target="_blank" href="http://www.beyondgenome.com/08Personalized_Medicine_Day1.asp">Beyond Genome</a> | San Francisco | June 9 - 11</li> - <li>ISMB 2008 | Toronto | July 19 - 23</li> - <li>Genome Informatics | Hinxton, UK | September 10 - 14</li> - <li><a target="_blank" href="http://www.ashg.org/2008meeting/pages/workshops.shtml#4">ASHG</a> | Philadelphia | November 11 - 15</li> - </ul> - <hr> - Download <strong>new</strong> Galaxy brochure <a href="/static/images/brochure.pdf">here</a>. +<div class="infomessage"> + <strong>We are hiring!</strong> + <hr> + Thanks to your support and an unprecedented level of usage, we are looking for an experienced software developer to join our team. For more information about this position, please, click <a target="_blank" href="https://www.bx.psu.edu/cgi-bin/trac.cgi/galaxy/wiki/PythonDeveloper">here</a>. </div> <hr> -<div class="section" align="center"> - <strong>Unsequenced Genomes of the World</strong> | June 2008 - <br> - <br> - <img src="images/welcomePhoto.jpg" border="0"> - <br> - Przewalski's Horse (<i>Equus przewalskii</i>) | Escondido, California - <br> - <br> -</div> +<div class="welcomeBlue" id="screencasts" align="center"> +<strong>Introducing Galactic Quickies</strong> + <hr> + Galactic quickies are <i>super-short</i> screencasts that are <i>always</i> under 5 minutes. We thought it may be a good way to spread the word about Galaxy's functionality while keeping the "annoyance factor" to the minimum. The quickies will be updated weekly. + <hr> + <table width="100%" border="0" height="100%"> + <tr> + <td valign="middle" align="center"> + + <br> + <script type="text/javascript"> + + </script> + <noscript> + <object width="640" height="480" classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab"> + <param name="src" value="http://screencast.g2.bx.psu.edu/galaxy/quickie1_TabSeq/quickie1_TabSeq.jpg" /> + <param name="href" value="http://screencast.g2.bx.psu.edu/galaxy/quickie1_TabSeq/quickie1_TabSeq.mov" /> + <param name="target" value="myself" /> + <param name="controller" value="false" /> + <param name="autoplay" value="false" /> + <param name="scale" value="aspect" /> + <embed width="640" height="480" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" + src="http://screencast.g2.bx.psu.edu/galaxy/quickie1_TabSeq/quickie1_TabSeq.jpg" + href="http://screencast.g2.bx.psu.edu/galaxy/quickie1_TabSeq/quickie1_TabSeq.mov" + target="myself" + controller="false" + autoplay="false" + scale="aspect"> + </embed> + </object> + </noscript> + + </td> + </tr> + </table> + <br> + For a high resolution version <a target="_blank" class="reference" href="http://screencast.g2.bx.psu.edu/galaxy/quickie1_TabSeq/">click here</a>. + <br> + <hr> + </div> <hr> -<div class="welcomeBlue"> - <strong>Galaxy is for Biologists</strong> - <br> - Use this site to access popular sources of data like the UCSC Table Browser. Run analyses right on the spot using a variety of integrated tools. Your results are always available and can be easily shared with others. Just <a target="_blank" href="http://g2.trac.bx.psu.edu/wiki/ScreenCasts">watch</a> how. -</div> -<br> -<div class="welcomeRed"> - <strong>Galaxy is for Developers</strong> - <br> - Galaxy is an easy-to-use, open-source, scalable framework for tool and data integration. Stop wasting time writing interfaces and get your tools used by biologists! Galaxy includes everything you need to get started, so <a target="_blank" href="http://g2.trac.bx.psu.edu/wiki/HowToInstall">download</a> and start <a target="_blank" href="http://g2.trac.bx.psu.edu/wiki/HowToInstall">integrating</a>! -</div> + <p><a target="_blank" class="reference" href="http://g2.trac.bx.psu.edu/wiki/GalaxyTeam">Galaxy team</a> is a part of <a target="_blank" class="reference" href="http://www.bx.psu.edu">BX</a> at <a target="_blank" class="reference" href="http://www.psu.edu">Penn State</a>.</p> <hr class="docutils" /> This project is supported in part by <a target="_blank" class="reference" href="http://www.nsf.gov">NSF</a> and <a target="_blank" class="reference" href="http://www.huck.psu.edu">the Huck Institutes of the Life Sciences</a>. <p><small>Galaxy build: <b>$Rev$</b></small></p>

1 0

[hg] galaxy 1716: Fixes in the upload tool for removing temporar...
by Nate Coraor 22 Jan '09

22 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/19d18731ce19 changeset: 1716:19d18731ce19 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Jan 22 11:22:39 2009 -0500 description: Fixes in the upload tool for removing temporary files. 1 file(s) affected in this change: lib/galaxy/tools/actions/upload.py diffs (81 lines): diff -r 5c293dbf0726 -r 19d18731ce19 lib/galaxy/tools/actions/upload.py --- a/lib/galaxy/tools/actions/upload.py Thu Jan 22 08:50:52 2009 -0500 +++ b/lib/galaxy/tools/actions/upload.py Thu Jan 22 11:22:39 2009 -0500 @@ -13,7 +13,11 @@ def __init__( self ): self.empty = False self.line_count = None - + def remove_tempfile( self, filename ): + try: + os.unlink( filename ) + except: + log.exception( 'failure removing temporary file: %s' % filename ) def execute( self, tool, trans, incoming={}, set_output_hid = True ): data_file = incoming['file_data'] file_type = incoming['file_type'] @@ -48,6 +52,7 @@ data_list.append( self.add_file( trans, data_file.local_filename, file_name, file_type, dbkey, space_to_tab=space_to_tab ) ) except Exception, e: log.exception( 'exception in add_file using datafile.local_filename %s: %s' % ( data_file.local_filename, str( e ) ) ) + self.remove_tempfile( data_file.local_filename ) return self.upload_empty( trans, job, "Error:", str( e ) ) elif 'filename' in dir( data_file ): file_name = data_file.filename @@ -57,16 +62,13 @@ temp_name = sniff.stream_to_file( data_file.file, prefix='upload' ) except Exception, e: log.exception( 'exception in sniff.stream_to_file using file %s: %s' % ( data_file.filename, str( e ) ) ) - try: - # Attempt to remove temporary file - os.unlink( temp_name ) - except: - log.exception( 'failure removing temporary file: %s' % temp_name ) + self.remove_tempfile( temp_name ) return self.upload_empty( trans, job, "Error:", str( e ) ) try: data_list.append( self.add_file( trans, temp_name, file_name, file_type, dbkey, space_to_tab=space_to_tab ) ) except Exception, e: log.exception( 'exception in add_file using file temp_name %s: %s' % ( str( temp_name ), str( e ) ) ) + self.remove_tempfile( temp_name ) return self.upload_empty( trans, job, "Error:", str( e ) ) if url_paste not in [ None, "" ]: if url_paste.lower().find( 'http://' ) >= 0 or url_paste.lower().find( 'ftp://' ) >= 0: @@ -78,16 +80,13 @@ temp_name = sniff.stream_to_file( urllib.urlopen( line ), prefix='url_paste' ) except Exception, e: log.exception( 'exception in sniff.stream_to_file using url_paste %s: %s' % ( url_paste, str( e ) ) ) - try: - # Attempt to remove temporary file - os.unlink( temp_name ) - except: - log.exception( 'failure removing temporary file: %s' % temp_name ) + self.remove_tempfile( temp_name ) return self.upload_empty( trans, job, "Error:", str( e ) ) try: data_list.append( self.add_file( trans, temp_name, line, file_type, dbkey, info="uploaded url", space_to_tab=space_to_tab ) ) except Exception, e: log.exception( 'exception in add_file using url_paste temp_name %s: %s' % ( str( temp_name ), str( e ) ) ) + self.remove_tempfile( temp_name ) return self.upload_empty( trans, job, "Error:", str( e ) ) else: is_valid = False @@ -101,16 +100,13 @@ temp_name = sniff.stream_to_file( StringIO.StringIO( url_paste ), prefix='strio_url_paste' ) except Exception, e: log.exception( 'exception in sniff.stream_to_file using StringIO.StringIO( url_paste ) %s: %s' % ( url_paste, str( e ) ) ) - try: - # Attempt to remove temporary file - os.unlink( temp_name ) - except: - log.exception( 'failure removing temporary file: %s' % temp_name ) + self.remove_tempfile( temp_name ) return self.upload_empty( trans, job, "Error:", str( e ) ) try: data_list.append( self.add_file( trans, temp_name, 'Pasted Entry', file_type, dbkey, info="pasted entry", space_to_tab=space_to_tab ) ) except Exception, e: log.excception( 'exception in add_file using StringIO.StringIO( url_paste ) temp_name %s: %s' % ( str( temp_name ), str( e ) ) ) + self.remove_tempfile( temp_name ) return self.upload_empty( trans, job, "Error:", str( e ) ) else: return self.upload_empty( trans, job, "No data error:", "you pasted no data." )

1 0

[hg] galaxy 1713: remove the usage of hast tables in the fasta-t...
by Greg Von Kuster 22 Jan '09

22 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/6d849785ff86 changeset: 1713:6d849785ff86 user: wychung date: Mon Jan 19 15:47:02 2009 -0500 description: remove the usage of hast tables in the fasta-tools 8 file(s) affected in this change: test-data/fasta_to_tabular_out1.tabular test-data/fasta_to_tabular_out3.tabular test-data/fasta_tool_compute_length_1.out test-data/fasta_tool_compute_length_3.out test-data/fasta_tool_filter_length_1.out tools/fasta_tools/fasta_compute_length.py tools/fasta_tools/fasta_filter_by_length.py tools/fasta_tools/fasta_to_tabular.py diffs (305 lines): diff -r e76a153769d4 -r 6d849785ff86 test-data/fasta_to_tabular_out1.tabular --- a/test-data/fasta_to_tabular_out1.tabular Mon Jan 19 11:47:29 2009 -0500 +++ b/test-data/fasta_to_tabular_out1.tabular Mon Jan 19 15:47:02 2009 -0500 @@ -14,5 +14,5 @@ EYKX4VC01BB4QL length=57 xy=0431_0363 region=1 run=R_2007_11_07_16_15_57_ GGGGAGGAGCTAATAATATGCTCTTGGGGAGGAGCTAATTATATGCTCTTGGGGAGG EYKX4VC01BJ37M length=64 xy=0522_0192 region=1 run=R_2007_11_07_16_15_57_ TCGAGTATGTATCAAGGACTACATACAAATTTGCCAAAAGAGATTATGCACTATCCCGACTTCC EYKX4VC01BV9R8 length=54 xy=0660_2038 region=1 run=R_2007_11_07_16_15_57_ AAAACTCGGAGAAACTATTCAGCAGCACTGCGTTTCGCTGAATTTTAGACCGTT +EYKX4VC01CEPP8 length=60 xy=0870_2350 region=1 run=R_2007_11_07_16_15_57_ CTGGGTGGGTGCACTACAGGAACGTCATTTGTTCAATCCTCACGTTGTTGTTAGTGTCAG EYKX4VC01BTLME length=78 xy=0630_0292 region=1 run=R_2007_11_07_16_15_57_ TTATCCACACGCTGTCCGGATCCAGCGCCAGGCGCCGACGCTGGACTTCCGCCGCCTGCGCCCAGTTGCCCTGACTTC -EYKX4VC01CEPP8 length=60 xy=0870_2350 region=1 run=R_2007_11_07_16_15_57_ CTGGGTGGGTGCACTACAGGAACGTCATTTGTTCAATCCTCACGTTGTTGTTAGTGTCAG diff -r e76a153769d4 -r 6d849785ff86 test-data/fasta_to_tabular_out3.tabular --- a/test-data/fasta_to_tabular_out3.tabular Mon Jan 19 11:47:29 2009 -0500 +++ b/test-data/fasta_to_tabular_out3.tabular Mon Jan 19 15:47:02 2009 -0500 @@ -14,5 +14,5 @@ EYKX4VC01BB4QL GGGGAGGAGCTAATAATATGCTCTTGGGGAGGAGCTAATTATATGCTCTTGGGGAGG EYKX4VC01BJ37M TCGAGTATGTATCAAGGACTACATACAAATTTGCCAAAAGAGATTATGCACTATCCCGACTTCC EYKX4VC01BV9R8 AAAACTCGGAGAAACTATTCAGCAGCACTGCGTTTCGCTGAATTTTAGACCGTT +EYKX4VC01CEPP8 CTGGGTGGGTGCACTACAGGAACGTCATTTGTTCAATCCTCACGTTGTTGTTAGTGTCAG EYKX4VC01BTLME TTATCCACACGCTGTCCGGATCCAGCGCCAGGCGCCGACGCTGGACTTCCGCCGCCTGCGCCCAGTTGCCCTGACTTC -EYKX4VC01CEPP8 CTGGGTGGGTGCACTACAGGAACGTCATTTGTTCAATCCTCACGTTGTTGTTAGTGTCAG diff -r e76a153769d4 -r 6d849785ff86 test-data/fasta_tool_compute_length_1.out --- a/test-data/fasta_tool_compute_length_1.out Mon Jan 19 11:47:29 2009 -0500 +++ b/test-data/fasta_tool_compute_length_1.out Mon Jan 19 15:47:02 2009 -0500 @@ -14,5 +14,5 @@ EYKX4VC01BB4QL length=57 xy=0431_0363 region=1 run=R_2007_11_07_16_15_57_ 57 EYKX4VC01BJ37M length=64 xy=0522_0192 region=1 run=R_2007_11_07_16_15_57_ 64 EYKX4VC01BV9R8 length=54 xy=0660_2038 region=1 run=R_2007_11_07_16_15_57_ 54 +EYKX4VC01CEPP8 length=60 xy=0870_2350 region=1 run=R_2007_11_07_16_15_57_ 60 EYKX4VC01BTLME length=78 xy=0630_0292 region=1 run=R_2007_11_07_16_15_57_ 78 -EYKX4VC01CEPP8 length=60 xy=0870_2350 region=1 run=R_2007_11_07_16_15_57_ 60 diff -r e76a153769d4 -r 6d849785ff86 test-data/fasta_tool_compute_length_3.out --- a/test-data/fasta_tool_compute_length_3.out Mon Jan 19 11:47:29 2009 -0500 +++ b/test-data/fasta_tool_compute_length_3.out Mon Jan 19 15:47:02 2009 -0500 @@ -14,5 +14,5 @@ EYKX4VC01BB4QL 57 EYKX4VC01BJ37M 64 EYKX4VC01BV9R8 54 +EYKX4VC01CEPP8 60 EYKX4VC01BTLME 78 -EYKX4VC01CEPP8 60 diff -r e76a153769d4 -r 6d849785ff86 test-data/fasta_tool_filter_length_1.out --- a/test-data/fasta_tool_filter_length_1.out Mon Jan 19 11:47:29 2009 -0500 +++ b/test-data/fasta_tool_filter_length_1.out Mon Jan 19 15:47:02 2009 -0500 @@ -53,9 +53,9 @@ >EYKX4VC01BV9R8 length=54 xy=0660_2038 region=1 run=R_2007_11_07_16_15_57_ AAAACTCGGAGAAACTATTCAGCAGCACTGCGTTTCGCTGAATTTTAGAC CGTT +>EYKX4VC01CEPP8 length=60 xy=0870_2350 region=1 run=R_2007_11_07_16_15_57_ +CTGGGTGGGTGCACTACAGGAACGTCATTTGTTCAATCCTCACGTTGTTG +TTAGTGTCAG >EYKX4VC01BTLME length=78 xy=0630_0292 region=1 run=R_2007_11_07_16_15_57_ TTATCCACACGCTGTCCGGATCCAGCGCCAGGCGCCGACGCTGGACTTCC GCCGCCTGCGCCCAGTTGCCCTGACTTC ->EYKX4VC01CEPP8 length=60 xy=0870_2350 region=1 run=R_2007_11_07_16_15_57_ -CTGGGTGGGTGCACTACAGGAACGTCATTTGTTCAATCCTCACGTTGTTG -TTAGTGTCAG diff -r e76a153769d4 -r 6d849785ff86 tools/fasta_tools/fasta_compute_length.py --- a/tools/fasta_tools/fasta_compute_length.py Mon Jan 19 11:47:29 2009 -0500 +++ b/tools/fasta_tools/fasta_compute_length.py Mon Jan 19 15:47:02 2009 -0500 @@ -1,8 +1,8 @@ #! /usr/bin/python """ -Input: fasta, minimal length, maximal length -Output: fasta -Return sequences whose lengths are within the range. +Input: fasta, int +Output: tabular +Return titles with lengths of corresponding seq """ import sys, os @@ -10,41 +10,37 @@ assert sys.version_info[:2] >= ( 2, 4 ) def __main__(): - input_filename = sys.argv[1] - output_filename = sys.argv[2] + + infile = sys.argv[1] + outfile = sys.argv[2] keep_first = int( sys.argv[3] ) - tmp_title = tmp_seq = '' - tmp_seq_count = 0 - seq_hash = {} + + fasta_title = fasta_seq = '' + # number of char to keep in the title if keep_first == 0: keep_first = None else: keep_first += 1 - for i, line in enumerate( file( input_filename ) ): + out = open(outfile, 'w') + + for i, line in enumerate( file( infile ) ): line = line.rstrip( '\r\n' ) if not line or line.startswith( '#' ): continue if line[0] == '>': - if len( tmp_seq ) > 0: - tmp_seq_count += 1 - seq_hash[ ( tmp_seq_count, tmp_title ) ] = tmp_seq - tmp_title = line - tmp_seq = '' + if len( fasta_seq ) > 0 : + out.write( "%s\t%d\n" % ( fasta_title[ 1:keep_first ], len( fasta_seq ) ) ) + fasta_title = line + fasta_seq = '' else: - tmp_seq = "%s%s" % ( tmp_seq, line ) - if line.split() and line.split()[0].isdigit(): - tmp_seq = "%s " % tmp_seq - if len( tmp_seq ) > 0: - seq_hash[ ( tmp_seq_count, tmp_title ) ] = tmp_seq - - title_keys = seq_hash.keys() - title_keys.sort() - output_handle = open( output_filename, 'w' ) - for i, fasta_title in title_keys: - tmp_seq = seq_hash[ ( i, fasta_title ) ] - output_handle.write( "%s\t%d\n" % ( fasta_title[ 1:keep_first ], len( tmp_seq ) ) ) - output_handle.close() + fasta_seq = "%s%s" % ( fasta_seq, line ) + + # check the last sequence + if len( fasta_seq ) > 0: + out.write( "%s\t%d\n" % ( fasta_title[ 1:keep_first ], len( fasta_seq ) ) ) + + out.close() if __name__ == "__main__" : __main__() \ No newline at end of file diff -r e76a153769d4 -r 6d849785ff86 tools/fasta_tools/fasta_filter_by_length.py --- a/tools/fasta_tools/fasta_filter_by_length.py Mon Jan 19 11:47:29 2009 -0500 +++ b/tools/fasta_tools/fasta_filter_by_length.py Mon Jan 19 15:47:02 2009 -0500 @@ -15,7 +15,7 @@ def __main__(): - input_filename = sys.argv[1] + infile = sys.argv[1] try: min_length = int( sys.argv[2] ) except: @@ -24,49 +24,62 @@ max_length = int( sys.argv[3] ) except: stop_err( "Maximum length of the return sequence requires a numerical value." ) - output_filename = sys.argv[4] - tmp_title = tmp_seq = '' - tmp_seq_count = 0 - seq_hash = {} + outfile = sys.argv[4] + fasta_title = fasta_seq = '' + at_least_one = 0 + + out = open( outfile, 'w' ) - for i, line in enumerate( file( input_filename ) ): + for i, line in enumerate( file( infile ) ): line = line.rstrip( '\r\n' ) if not line or line.startswith( '#' ): continue + if line[0] == '>': - if len( tmp_seq ) > 0: - tmp_seq_count += 1 - seq_hash[ ( tmp_seq_count, tmp_title ) ] = tmp_seq - tmp_title = line - tmp_seq = '' + if len( fasta_seq ) > 0: + + if max_length <= 0: + compare_max_length = len( fasta_seq ) + 1 + else: + compare_max_length = max_length + + l = len( fasta_seq ) + + if l >= min_length and l <= compare_max_length: + at_least_one += 1 + out.write( "%s\n" % fasta_title ) + c = 0 + s = fasta_seq + while c < l: + b = min( c + 50, l ) + out.write( "%s\n" % s[ c:b ] ) + c = b + + fasta_title = line + fasta_seq = '' else: - tmp_seq = "%s%s" % ( tmp_seq, line ) - if line.split()[0].isdigit(): - tmp_seq = "%s " % tmp_seq - if len( tmp_seq ) > 0: - seq_hash[ ( tmp_seq_count, tmp_title ) ] = tmp_seq + fasta_seq = "%s%s" % ( fasta_seq, line ) - title_keys = seq_hash.keys() - title_keys.sort() - output_handle = open( output_filename, 'w' ) - at_least_one = 0 - for i, fasta_title in title_keys: - tmp_seq = seq_hash[ ( i, fasta_title ) ] + if len( fasta_seq ) > 0: + if max_length <= 0: - compare_max_length = len( tmp_seq ) + 1 + compare_max_length = len( fasta_seq ) + 1 else: compare_max_length = max_length - l = len( tmp_seq ) + + l = len( fasta_seq ) + if l >= min_length and l <= compare_max_length: at_least_one += 1 - output_handle.write( "%s\n" % fasta_title ) + out.write( "%s\n" % fasta_title ) c = 0 - s = tmp_seq + s = fasta_seq while c < l: b = min( c + 50, l ) - output_handle.write( "%s\n" % s[ c:b ] ) + out.write( "%s\n" % s[ c:b ] ) c = b - output_handle.close() + + out.close() if at_least_one == 0: print "There is no sequence that falls within your range." diff -r e76a153769d4 -r 6d849785ff86 tools/fasta_tools/fasta_to_tabular.py --- a/tools/fasta_tools/fasta_to_tabular.py Mon Jan 19 11:47:29 2009 -0500 +++ b/tools/fasta_tools/fasta_to_tabular.py Mon Jan 19 15:47:02 2009 -0500 @@ -1,9 +1,9 @@ #! /usr/bin/python # This code exists in 2 places: ~/datatypes/converters and ~/tools/fasta_tools """ -Input: fasta, minimal length, maximal length -Output: fasta -Return sequences whose lengths are within the range. +Input: fasta, int +Output: tabular +format convert: fasta to tabular """ import sys, os @@ -14,38 +14,31 @@ infile = sys.argv[1] outfile = sys.argv[2] keep_first = int( sys.argv[3] ) - title = '' - sequence = '' - sequence_count = 0 + fasta_title = fasta_seq = '' if keep_first == 0: keep_first = None else: keep_first += 1 + out = open( outfile, 'w' ) + for i, line in enumerate( open( infile ) ): line = line.rstrip( '\r\n' ) if not line or line.startswith( '#' ): continue if line.startswith( '>' ): - if sequence: - sequence_count += 1 - seq_hash[( sequence_count, title )] = sequence - title = line - sequence = '' + if fasta_seq: + out.write( "%s\t%s\n" %( fasta_title[ 1:keep_first ], fasta_seq ) ) + fasta_title = line + fasta_seq = '' else: - sequence = "%s%s" % ( sequence, line ) - if line.split() and line.split()[0].isdigit(): - sequence += ' ' - if sequence: - seq_hash[( sequence_count, title )] = sequence - # return only those lengths are in the range - title_keys = seq_hash.keys() - title_keys.sort() - out = open( outfile, 'w' ) - for i, fasta_title in title_keys: - sequence = seq_hash[( i, fasta_title )] - out.write( "%s\t%s\n" %( fasta_title[ 1:keep_first ], sequence ) ) + if line: + fasta_seq = "%s%s" % ( fasta_seq, line ) + + if fasta_seq: + out.write( "%s\t%s\n" %( fasta_title[ 1:keep_first ], fasta_seq ) ) + out.close() if __name__ == "__main__" : __main__() \ No newline at end of file

1 0

[hg] galaxy 1715: Fixes for formatting end of line chars in html...
by Greg Von Kuster 22 Jan '09

22 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/5c293dbf0726 changeset: 1715:5c293dbf0726 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Jan 22 08:50:52 2009 -0500 description: Fixes for formatting end of line chars in html when displaying dataset.info. Also change the biomart tool version. 3 file(s) affected in this change: lib/galaxy/datatypes/data.py tools/data_source/biomart.xml tools/data_source/biomart_test.xml diffs (62 lines): diff -r dbb3210e4549 -r 5c293dbf0726 lib/galaxy/datatypes/data.py --- a/lib/galaxy/datatypes/data.py Wed Jan 21 17:03:39 2009 -0500 +++ b/lib/galaxy/datatypes/data.py Thu Jan 22 08:50:52 2009 -0500 @@ -128,16 +128,22 @@ out = "Can't create peek %s" % str( exc ) return out def display_name(self, dataset): - """Returns formated html of dataset name""" + """Returns formatted html of dataset name""" try: return escape(dataset.name) except: return "name unavailable" def display_info(self, dataset): - """Returns formated html of dataset info""" + """Returns formatted html of dataset info""" try: # Change new line chars to html - return escape( dataset.info ).replace( "\r", "\n" ).replace( "\n", "<br>" ) + if dataset.info.find( '\r\n' ) >= 0: + dataset.info = dataset.info.replace( '\r\n', '<br/>' ) + if dataset.info.find( '\r' ) >= 0: + dataset.info = dataset.info.replace( '\r', '<br/>' ) + if dataset.info.find( '\n' ) >= 0: + dataset.info = dataset.info.replace( '\n', '<br/>' ) + return escape( dataset.info ) except: return "info unavailable" def validate(self, dataset): @@ -154,7 +160,7 @@ Adds a display app to the datatype. app_id is a unique id label is the primary display label, ie display at 'UCSC' - file_function is a string containing the name of the function that returns a properly formated display + file_function is a string containing the name of the function that returns a properly formatted display links_function is a string containing the name of the function that returns a list of (link_name,link) """ self.supported_display_apps = self.supported_display_apps.copy() diff -r dbb3210e4549 -r 5c293dbf0726 tools/data_source/biomart.xml --- a/tools/data_source/biomart.xml Wed Jan 21 17:03:39 2009 -0500 +++ b/tools/data_source/biomart.xml Thu Jan 22 08:50:52 2009 -0500 @@ -7,7 +7,7 @@ TODO: Hack to get biomart to work - the 'add_to_URL' param can be eliminated when the Biomart team encodes URL prior to sending, meanwhile everything including and beyond the first '&' is truncated from URL. They said they'll let us know when this is fixed at their end. --> -<tool name="BioMart" id="biomart" tool_type="data_source" URL_method="get"> +<tool name="BioMart" id="biomart" tool_type="data_source" URL_method="get" version="1.0.1"> <description>Central server</description> <command interpreter="python">data_source.py $output</command> <inputs action="http://www.biomart.org/biomart/martview" check_values="false" method="get" target="_top"> diff -r dbb3210e4549 -r 5c293dbf0726 tools/data_source/biomart_test.xml --- a/tools/data_source/biomart_test.xml Wed Jan 21 17:03:39 2009 -0500 +++ b/tools/data_source/biomart_test.xml Thu Jan 22 08:50:52 2009 -0500 @@ -7,7 +7,7 @@ TODO: Hack to get biomart to work - the 'add_to_URL' param can be eliminated when the Biomart team encodes URL prior to sending, meanwhile everything including and beyond the first '&' is truncated from URL. They said they'll let us know when this is fixed at their end. --> -<tool name="BioMart" id="biomart_test" tool_type="data_source" URL_method="get"> +<tool name="BioMart" id="biomart_test" tool_type="data_source" URL_method="get" version="1.0.1"> <description>Test server</description> <command interpreter="python">data_source.py $output</command> <inputs action="http://test.biomart.org/biomart/martview" check_values="false" method="get" target="_top">

1 0

[hg] galaxy 1714: Fixes for Biomart and UCSC to better ensure th...
by Greg Von Kuster 22 Jan '09

22 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/dbb3210e4549 changeset: 1714:dbb3210e4549 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Jan 21 17:03:39 2009 -0500 description: Fixes for Biomart and UCSC to better ensure the data type is correctly set. A data type of "tabular" is now the default, and data will be sniffed if this is the data type. 6 file(s) affected in this change: lib/galaxy/tools/__init__.py tools/data_source/biomart.xml tools/data_source/biomart_test.xml tools/data_source/ucsc_tablebrowser.xml tools/data_source/ucsc_tablebrowser_archaea.xml tools/data_source/ucsc_tablebrowser_test.xml diffs (182 lines): diff -r 6d849785ff86 -r dbb3210e4549 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py Mon Jan 19 15:47:02 2009 -0500 +++ b/lib/galaxy/tools/__init__.py Wed Jan 21 17:03:39 2009 -0500 @@ -1240,8 +1240,6 @@ def exec_before_job( self, app, inp_data, out_data, param_dict={} ): if self.tool_type == 'data_source': - # List for converting UCSC to Galaxy exts, if not in following dictionary, use provided datatype - data_type_to_ext = { 'wigdata':'wig', 'tab':'interval', 'hyperlinks':'html', 'sequence':'fasta' } dbkey = param_dict.get( 'dbkey' ) organism = param_dict.get( 'organism' ) table = param_dict.get( 'table' ) @@ -1263,12 +1261,9 @@ data.name = '%s on %s' % ( data.name, gb_landmark_region ) data.info = info data.dbkey = dbkey - try: - data_type = data_type_to_ext[ data_type ] - except: - pass - if data_type not in app.datatypes_registry.datatypes_by_extension: - data_type = 'interval' + if data_type not in app.datatypes_registry.datatypes_by_extension: + # Setting data_type to tabular will force the data to be sniffed in exec_after_process() + data_type = 'tabular' data = app.datatypes_registry.change_datatype( data, data_type ) # Store external data source's request parameters temporarily in output file. # In case the config setting for "outputs_to_working_directory" is True, we must write to @@ -1284,8 +1279,6 @@ return out_data def exec_after_process( self, app, inp_data, out_data, param_dict ): - # TODO: for data_source tools at least, this code can probably be handled more optimally by adding a new - # tag set in the tool config. if self.tool_type == 'data_source': name, data = out_data.items()[0] data.set_size() @@ -1294,9 +1287,10 @@ data.info = param_dict.get( 'info', data.name ) data.dbkey = param_dict.get( 'dbkey', data.dbkey ) data.extension = param_dict.get( 'data_type', data.extension ) - if data.extension == 'txt': + if data.extension in [ 'txt', 'tabular' ]: data_type = sniff.guess_ext( data.file_name, sniff_order=app.datatypes_registry.sniff_order ) - data = app.datatypes_registry.change_datatype( data, data_type ) + if data.extension != data_type: + data = app.datatypes_registry.change_datatype( data, data_type ) elif not isinstance( data.datatype, datatypes.interval.Bed ) and isinstance( data.datatype, datatypes.interval.Interval ): data.set_meta() if data.missing_meta(): diff -r 6d849785ff86 -r dbb3210e4549 tools/data_source/biomart.xml --- a/tools/data_source/biomart.xml Mon Jan 19 15:47:02 2009 -0500 +++ b/tools/data_source/biomart.xml Wed Jan 21 17:03:39 2009 -0500 @@ -21,17 +21,21 @@ <param_from_source name="GALAXY_URL" missing="0" /> </add_to_url> </request_param> + <request_param galaxy_name="data_type" remote_name="exportView_outputformat" missing="tabular" > + <data_type_translation> + <format galaxy_format="tabular" remote_format="TSV" /> + </data_type_translation> + </request_param> <request_param galaxy_name="dbkey" remote_name="dbkey" missing="?" /> <request_param galaxy_name="organism" remote_name="organism" missing="" /> <request_param galaxy_name="table" remote_name="table" missing="" /> <request_param galaxy_name="description" remote_name="description" missing="" /> <request_param galaxy_name="name" remote_name="name" missing="Biomart query" /> <request_param galaxy_name="info" remote_name="info" missing="" /> - <request_param galaxy_name="data_type" remote_name="type" missing="txt" /> </request_param_translation> <uihints minwidth="800"/> <outputs> - <data name="output" format="txt" /> + <data name="output" format="tabular" /> </outputs> <options sanitize="False" refresh="True"/> </tool> diff -r 6d849785ff86 -r dbb3210e4549 tools/data_source/biomart_test.xml --- a/tools/data_source/biomart_test.xml Mon Jan 19 15:47:02 2009 -0500 +++ b/tools/data_source/biomart_test.xml Wed Jan 21 17:03:39 2009 -0500 @@ -21,17 +21,21 @@ <param_from_source name="GALAXY_URL" missing="0" /> </add_to_url> </request_param> + <request_param galaxy_name="data_type" remote_name="exportView_outputformat" missing="tabular" > + <data_type_translation> + <format galaxy_format="tabular" remote_format="TSV" /> + </data_type_translation> + </request_param> <request_param galaxy_name="dbkey" remote_name="dbkey" missing="?" /> <request_param galaxy_name="organism" remote_name="organism" missing="" /> <request_param galaxy_name="table" remote_name="table" missing="" /> <request_param galaxy_name="description" remote_name="description" missing="" /> <request_param galaxy_name="name" remote_name="name" missing="Biomart test query" /> <request_param galaxy_name="info" remote_name="info" missing="" /> - <request_param galaxy_name="data_type" remote_name="type" missing="txt" /> </request_param_translation> <uihints minwidth="800"/> <outputs> - <data name="output" format="txt" /> + <data name="output" format="tabular" /> </outputs> <options sanitize="False" refresh="True"/> </tool> diff -r 6d849785ff86 -r dbb3210e4549 tools/data_source/ucsc_tablebrowser.xml --- a/tools/data_source/ucsc_tablebrowser.xml Mon Jan 19 15:47:02 2009 -0500 +++ b/tools/data_source/ucsc_tablebrowser.xml Wed Jan 21 17:03:39 2009 -0500 @@ -21,16 +21,20 @@ <request_param galaxy_name="organism" remote_name="org" missing="unknown species" /> <request_param galaxy_name="table" remote_name="hgta_track" missing="unknown table" /> <request_param galaxy_name="description" remote_name="hgta_regionType" missing="no description" /> - <request_param galaxy_name="data_type" remote_name="hgta_outputType" missing="bed" > + <request_param galaxy_name="data_type" remote_name="hgta_outputType" missing="tabular" > <data_type_translation> <format galaxy_format="tabular" remote_format="primaryTable" /> <format galaxy_format="tabular" remote_format="selectedFields" /> + <format galaxy_format="wig" remote_format="wigdata" /> + <format galaxy_format="interval" remote_format="tab" /> + <format galaxy_format="html" remote_format="hyperlinks" /> + <format galaxy_format="fasta" remote_format="sequence" /> </data_type_translation> </request_param> </request_param_translation> <uihints minwidth="800"/> <outputs> - <data name="output" format="bed" /> + <data name="output" format="tabular" /> </outputs> <options sanitize="False" refresh="True"/> </tool> diff -r 6d849785ff86 -r dbb3210e4549 tools/data_source/ucsc_tablebrowser_archaea.xml --- a/tools/data_source/ucsc_tablebrowser_archaea.xml Mon Jan 19 15:47:02 2009 -0500 +++ b/tools/data_source/ucsc_tablebrowser_archaea.xml Wed Jan 21 17:03:39 2009 -0500 @@ -21,16 +21,20 @@ <request_param galaxy_name="organism" remote_name="org" missing="unknown species" /> <request_param galaxy_name="table" remote_name="hgta_track" missing="" /> <request_param galaxy_name="description" remote_name="hgta_regionType" missing="" /> - <request_param galaxy_name="data_type" remote_name="hgta_outputType" missing="bed" > + <request_param galaxy_name="data_type" remote_name="hgta_outputType" missing="tabular" > <data_type_translation> <format galaxy_format="tabular" remote_format="primaryTable" /> <format galaxy_format="tabular" remote_format="selectedFields" /> + <format galaxy_format="wig" remote_format="wigdata" /> + <format galaxy_format="interval" remote_format="tab" /> + <format galaxy_format="html" remote_format="hyperlinks" /> + <format galaxy_format="fasta" remote_format="sequence" /> </data_type_translation> </request_param> </request_param_translation> <uihints minwidth="800"/> <outputs> - <data name="output" format="bed" /> + <data name="output" format="tabular" /> </outputs> <options sanitize="False" refresh="True"/> </tool> diff -r 6d849785ff86 -r dbb3210e4549 tools/data_source/ucsc_tablebrowser_test.xml --- a/tools/data_source/ucsc_tablebrowser_test.xml Mon Jan 19 15:47:02 2009 -0500 +++ b/tools/data_source/ucsc_tablebrowser_test.xml Wed Jan 21 17:03:39 2009 -0500 @@ -21,16 +21,20 @@ <request_param galaxy_name="organism" remote_name="org" missing="unknown species" /> <request_param galaxy_name="table" remote_name="hgta_track" missing="" /> <request_param galaxy_name="description" remote_name="hgta_regionType" missing="" /> - <request_param galaxy_name="data_type" remote_name="hgta_outputType" missing="bed" > + <request_param galaxy_name="data_type" remote_name="hgta_outputType" missing="tabular" > <data_type_translation> <format galaxy_format="tabular" remote_format="primaryTable" /> <format galaxy_format="tabular" remote_format="selectedFields" /> + <format galaxy_format="wig" remote_format="wigdata" /> + <format galaxy_format="interval" remote_format="tab" /> + <format galaxy_format="html" remote_format="hyperlinks" /> + <format galaxy_format="fasta" remote_format="sequence" /> </data_type_translation> </request_param> </request_param_translation> <uihints minwidth="800"/> <outputs> - <data name="output" format="bed" /> + <data name="output" format="tabular" /> </outputs> <options sanitize="False" refresh="True"/> </tool>

1 0

[hg] galaxy 1712: Flush newly created users directly in __get_or...
by Nate Coraor 19 Jan '09

19 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e76a153769d4 changeset: 1712:e76a153769d4 user: Nate Coraor <nate(a)bx.psu.edu> date: Mon Jan 19 11:47:29 2009 -0500 description: Flush newly created users directly in __get_or_create_remote_user, solves the new user creation when using remote_user bug found by Ross Lazarus. Also, external users get a random password now (makes the secret key safer if a bad guy gets the database). 1 file(s) affected in this change: lib/galaxy/web/framework/__init__.py diffs (41 lines): diff -r 2a36ccdb2a38 -r e76a153769d4 lib/galaxy/web/framework/__init__.py --- a/lib/galaxy/web/framework/__init__.py Fri Jan 16 13:30:01 2009 -0500 +++ b/lib/galaxy/web/framework/__init__.py Mon Jan 19 11:47:29 2009 -0500 @@ -4,7 +4,7 @@ import pkg_resources -import os, sys, time +import os, sys, time, random, string pkg_resources.require( "Cheetah" ) from Cheetah.Template import Template import base @@ -229,6 +229,9 @@ user_for_new_session = self.__get_or_create_remote_user( remote_user_email ) log.warning( "User logged in as '%s' externally, but has a cookie as '%s' invalidating session", remote_user_email, prev_galaxy_session.user.email ) + else: + # No session exists, get/create user for new session + user_for_new_session = self.__get_or_create_remote_user( remote_user_email ) else: if galaxy_session is not None and galaxy_session.user and galaxy_session.user.external: # Remote user support is not enabled, but there is an existing @@ -282,15 +285,15 @@ def __get_or_create_remote_user( self, remote_user_email ): """ Return the user in $HTTP_REMOTE_USER and create if necessary - - Caller is responsible for flushing the returned user. """ # remote_user middleware ensures HTTP_REMOTE_USER exists user = self.app.model.User.filter_by( email=remote_user_email ).first() if user is None: + random.seed() user = self.app.model.User( email=remote_user_email ) - user.set_password_cleartext( 'external' ) + user.set_password_cleartext( ''.join( random.sample( string.letters + string.digits, 12 ) ) ) user.external = True + user.flush() #self.log_event( "Automatically created account '%s'", user.email ) return user def __update_session_cookie( self, name='galaxysession' ):

1 0

[hg] galaxy 1710: Eliminate the use of the "order=" for the snif...
by Nate Coraor 19 Jan '09

19 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/c565de071f7f changeset: 1710:c565de071f7f user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Jan 15 14:23:53 2009 -0500 description: Eliminate the use of the "order=" for the sniffers in datatypes.conf - since we've moved from using an ini file to an xml file, ordering is implicit. 2 file(s) affected in this change: datatypes_conf.xml.sample lib/galaxy/datatypes/registry.py diffs (74 lines): diff -r 90d04fcbcdf4 -r c565de071f7f datatypes_conf.xml.sample --- a/datatypes_conf.xml.sample Thu Jan 15 13:10:04 2009 -0500 +++ b/datatypes_conf.xml.sample Thu Jan 15 14:23:53 2009 -0500 @@ -146,20 +146,22 @@  - <sniffer order="005" type="galaxy.datatypes.xml:BlastXml"/> - <sniffer order="010" type="galaxy.datatypes.sequence:Maf"/> - <sniffer order="015" type="galaxy.datatypes.sequence:Lav"/> - <sniffer order="020" type="galaxy.datatypes.sequence:Fasta"/> - <sniffer order="025" type="galaxy.datatypes.sequence:FastqSolexa"/> - <sniffer order="030" type="galaxy.datatypes.interval:Wiggle"/> - <sniffer order="035" type="galaxy.datatypes.images:Html"/> - <sniffer order="040" type="galaxy.datatypes.sequence:Axt"/> - <sniffer order="045" type="galaxy.datatypes.interval:Bed"/> - <sniffer order="050" type="galaxy.datatypes.interval:CustomTrack"/> - <sniffer order="055" type="galaxy.datatypes.interval:Gff"/> - <sniffer order="060" type="galaxy.datatypes.interval:Gff3"/> - <sniffer order="065" type="galaxy.datatypes.interval:Interval"/> + <sniffer type="galaxy.datatypes.xml:BlastXml"/> + <sniffer type="galaxy.datatypes.sequence:Maf"/> + <sniffer type="galaxy.datatypes.sequence:Lav"/> + <sniffer type="galaxy.datatypes.sequence:Fasta"/> + <sniffer type="galaxy.datatypes.sequence:FastqSolexa"/> + <sniffer type="galaxy.datatypes.interval:Wiggle"/> + <sniffer type="galaxy.datatypes.images:Html"/> + <sniffer type="galaxy.datatypes.sequence:Axt"/> + <sniffer type="galaxy.datatypes.interval:Bed"/> + <sniffer type="galaxy.datatypes.interval:CustomTrack"/> + <sniffer type="galaxy.datatypes.interval:Gff"/> + <sniffer type="galaxy.datatypes.interval:Gff3"/> + <sniffer type="galaxy.datatypes.interval:Interval"/> </sniffers> </datatypes> diff -r 90d04fcbcdf4 -r c565de071f7f lib/galaxy/datatypes/registry.py --- a/lib/galaxy/datatypes/registry.py Thu Jan 15 13:10:04 2009 -0500 +++ b/lib/galaxy/datatypes/registry.py Thu Jan 15 14:23:53 2009 -0500 @@ -59,18 +59,15 @@ self.converters.append( ( converter_config, extension, target_datatype ) ) except Exception, e: self.log.warning( 'Error loading datatype "%s", problem: %s' % ( extension, str( e ) ) ) - # Load datatype sniffers from config + # Load datatype sniffers from the config sniff_order = [] sniffers = root.find( 'sniffers' ) for elem in sniffers.findall( 'sniffer' ): - order = elem.get( 'order', None ) type = elem.get( 'type', None ) - if order and type: - sniff_order.append( ( order, type ) ) - sniff_order.sort() - for ele in sniff_order: + if type: + sniff_order.append( type ) + for type in sniff_order: try: - type = ele[1] fields = type.split( ":" ) datatype_module = fields[0] datatype_class = fields[1] @@ -147,6 +144,7 @@ sequence.Maf(), sequence.Lav(), sequence.Fasta(), + sequence.FastqSolexa(), interval.Wiggle(), images.Html(), sequence.Axt(),

1 0

[hg] galaxy 1711: Removed an option from grouping tool, which is...
by Nate Coraor 19 Jan '09

19 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/2a36ccdb2a38 changeset: 1711:2a36ccdb2a38 user: guru date: Fri Jan 16 13:30:01 2009 -0500 description: Removed an option from grouping tool, which is now part of the LCA tool under Taxonomy. 2 file(s) affected in this change: tools/stats/grouping.py tools/stats/grouping.xml diffs (354 lines): diff -r c565de071f7f -r 2a36ccdb2a38 tools/stats/grouping.py --- a/tools/stats/grouping.py Thu Jan 15 14:23:53 2009 -0500 +++ b/tools/stats/grouping.py Fri Jan 16 13:30:01 2009 -0500 @@ -12,15 +12,13 @@ def main(): inputfile = sys.argv[2] - in_columns = int( sys.argv[5] ) - show_remaining_cols = sys.argv[4] ops = [] cols = [] rounds = [] elems = [] - for var in sys.argv[6:]: + for var in sys.argv[4:]: ops.append(var.split()[0]) cols.append(var.split()[1]) rounds.append(var.split()[2]) @@ -81,19 +79,9 @@ if error_code != 0: stop_err( "Sorting input dataset resulted in error: %s: %s" %( error_code, stdout )) - - if show_remaining_cols == 'yes': - show_cols_list = [1]*in_columns - show_cols_list[group_col] = 0 - for c in cols: - c = int(c)-1 - show_cols_list[c] = 0 - #at the end of this, only the indices of the remaining columns will be set to 1 - remaining_cols = [j for j,k in enumerate(show_cols_list) if k==1] #this is the list of remaining column indices - + prev_item = "" prev_vals = [] - remaining_vals = [] skipped_lines = 0 first_invalid_line = 0 invalid_line = '' @@ -128,30 +116,22 @@ invalid_column = col+1 if valid: prev_vals[i].append(fields[col].strip()) - #Store values from all the remaning columns - if show_remaining_cols == 'yes': - for j, index in enumerate(remaining_cols): - remaining_vals[j].append(fields[index].strip()) else: """ When a new value is encountered, write the previous value and the corresponding aggregate values into the output file. This works due to the sort on group_col we've applied to the data above. """ - out_list = ['']*in_columns - out_list[group_col] = str(prev_item) - + out_str = prev_item + for i, op in enumerate( ops ): rfunc = "r." + op if op not in ['c','length','unique','random']: for j, elem in enumerate( prev_vals[i] ): prev_vals[i][j] = float( elem ) + rout = "%g" %( eval( rfunc )( prev_vals[i] )) if rounds[i] == 'yes': - rout = "%f" %( eval( rfunc )( prev_vals[i] )) rout = int(round(float(rout))) - else: - rout = "%g" %( eval( rfunc )( prev_vals[i] )) - else: if op != 'random': rout = eval( rfunc )( prev_vals[i] ) @@ -162,21 +142,9 @@ if op == 'unique': rfunc = "r.length" rout = eval( rfunc )( rout ) - - out_list[int(cols[i])-1] = str(rout) - - if show_remaining_cols == 'yes': - for index,el in enumerate(remaining_cols): - if index == 0: - try: - random_index = random.randint(0,len(remaining_vals[index])-1) - except: - random_index = 0 - #pick a random value from each of the remaning columns - rand_out = remaining_vals[index][random_index] - out_list[el] = str(rand_out) - - print >>fout, '\t'.join([elem for elem in out_list if elem != '']) + out_str += "\t" + str(rout) + + print >>fout, out_str prev_item = item prev_vals = [] @@ -185,14 +153,6 @@ val_list = [] val_list.append(fields[col].strip()) prev_vals.append(val_list) - - if show_remaining_cols == 'yes': - remaining_vals = [] - for index in remaining_cols: - remaining_val_list = [] - remaining_val_list.append(fields[index].strip()) - remaining_vals.append(remaining_val_list) - else: # This only occurs once, right at the start of the iteration. prev_item = item @@ -201,15 +161,8 @@ val_list = [] val_list.append(fields[col].strip()) prev_vals.append(val_list) - - if show_remaining_cols == 'yes': - remaining_vals = [] - for index in remaining_cols: - remaining_val_list = [] - remaining_val_list.append(fields[index].strip()) - remaining_vals.append(remaining_val_list) - except Exception: + except Exception, exc: skipped_lines += 1 if not first_invalid_line: first_invalid_line = ii+1 @@ -219,8 +172,7 @@ first_invalid_line = ii+1 # Handle the last grouped value - out_list = ['']*in_columns - out_list[group_col] = str(prev_item) + out_str = prev_item for i, op in enumerate(ops): rfunc = "r." + op @@ -228,11 +180,9 @@ if op not in ['c','length','unique','random']: for j, elem in enumerate( prev_vals[i] ): prev_vals[i][j] = float( elem ) + rout = '%g' %( eval( rfunc )( prev_vals[i] )) if rounds[i] == 'yes': - rout = '%f' %( eval( rfunc )( prev_vals[i] )) rout = int(round(float(rout))) - else: - rout = '%g' %( eval( rfunc )( prev_vals[i] )) else: if op != 'random': rout = eval( rfunc )( prev_vals[i] ) @@ -243,22 +193,13 @@ if op == 'unique': rfunc = "r.length" rout = eval( rfunc )( rout ) - out_list[int(cols[i])-1] = str(rout) + out_str += "\t" + str( rout ) except: skipped_lines += 1 if not first_invalid_line: first_invalid_line = ii+1 - if show_remaining_cols == 'yes': - for index,el in enumerate(remaining_cols): - if index == 0: - try: - random_index = random.randint(0,len(remaining_vals[index])-1) - except: - random_index = 0 - rand_out = remaining_vals[index][random_index] - out_list[el] = str(rand_out) - print >>fout, '\t'.join([elem for elem in out_list if elem != '']) + print >>fout, out_str # Generate a useful info message. msg = "--Group by c%d: " %(group_col+1) diff -r c565de071f7f -r 2a36ccdb2a38 tools/stats/grouping.xml --- a/tools/stats/grouping.xml Thu Jan 15 14:23:53 2009 -0500 +++ b/tools/stats/grouping.xml Fri Jan 16 13:30:01 2009 -0500 @@ -1,49 +1,43 @@ -<tool id="Grouping1" name="Group" version="1.5.0"> - <description>data by a column and perform aggregate operation on other columns.</description> - <command interpreter="python"> - grouping.py - $out_file1 - $input1 +<tool id="Grouping1" name="Group" version="1.4.0"> + <description>data by a column and perform aggregate operation on other columns.</description> + <command interpreter="python"> + grouping.py + $out_file1 + $input1 $groupcol - $othercols - ${input1.metadata.columns} - #for $op in $operations - '${op.optype} - ${op.opcol} - ${op.opround}' - #end for - </command> - <inputs> - <param format="tabular" name="input1" type="data" label="Select data" help="Query missing? See TIP below."/> - <param name="groupcol" label="Group by column" type="data_column" data_ref="input1" /> - <repeat name="operations" title="Operation"> - <param name="optype" type="select" label="Type"> - <option value="mean">Mean</option> - <option value="max">Maximum</option> - <option value="min">Minimum</option> - <option value="sum">Sum</option> - <option value="length">Count</option> - <option value="unique">Count Distinct</option> + #for $op in $operations + '${op.optype} + ${op.opcol} + ${op.opround}' + #end for + </command> + <inputs> + <param format="tabular" name="input1" type="data" label="Select data" help="Query missing? See TIP below."/> + <param name="groupcol" label="Group by column" type="data_column" data_ref="input1" /> + <repeat name="operations" title="Operation"> + <param name="optype" type="select" label="Type"> + <option value="mean">Mean</option> + <option value="max">Maximum</option> + <option value="min">Minimum</option> + <option value="sum">Sum</option> + <option value="length">Count</option> + <option value="unique">Count Distinct</option> <option value="c">Concatenate</option> - <option value="random">Randomly pick</option> - </param> - <param name="opcol" label="On column" type="data_column" data_ref="input1" /> - <param name="opround" type="select" label="Round result to nearest integer?"> - <option value="no">NO</option> - <option value="yes">YES</option> - </param> - </repeat> - <param name="othercols" type="select" label="Randomly pick an entry from each of the remaining columns (besides the columns chosen for group and aggregate operations above) ?"> + <option value="random">Randomly pick</option> + </param> + <param name="opcol" label="On column" type="data_column" data_ref="input1" /> + <param name="opround" type="select" label="Round result to nearest integer?"> <option value="no">NO</option> <option value="yes">YES</option> - </param> - </inputs> - <outputs> - <data format="input" name="out_file1" metadata_source="input1" /> - </outputs> - <requirements> - <requirement type="python-module">rpy</requirement> - </requirements> + </param> + </repeat> + </inputs> + <outputs> + <data format="input" name="out_file1" metadata_source="input1" /> + </outputs> + <requirements> + <requirement type="python-module">rpy</requirement> + </requirements> <tests>  <test> @@ -52,9 +46,9 @@ <param name="optype" value="mean"/> <param name="opcol" value="2"/> <param name="opround" value="no"/> - <param name="othercols" value="no"/> <output name="out_file1" file="groupby_out1.dat"/> </test> +  <test> <param name="input1" value="1.tabular"/> @@ -62,39 +56,38 @@ <param name="optype" value="mean"/> <param name="opcol" value="2"/> <param name="opround" value="no"/> - <param name="othercols" value="no"/> <output name="out_file1" file="groupby_out2.dat"/> - </test> - </tests> - <help> - -.. class:: infomark - -**TIP:** If your data is not TAB delimited, use *Text Manipulation->Convert* - ------ - -**Syntax** - -This tool allows you to group the input dataset by a particular column and perform aggregate functions like Mean, Sum, Max, Min and Concatenate on other columns. - -- All invalid, blank and comment lines are skipped when performing the aggregate functions. The number of skipped lines is displayed in the resulting history item. - ------ - -**Example** - -- For the following input:: - - chr22 1000 NM_17 - chr22 2000 NM_18 - chr10 2200 NM_10 - chr10 1200 NM_11 - chr22 1600 NM_19 - -- running this tool with **Group by column 1**, Operations **Mean on column 2** and **Concatenate on column 3** will return:: - - chr10 1700.00 ['NM_11', 'NM_10'] - chr22 1533.33 ['NM_17', 'NM_19', 'NM_18'] - </help> -</tool> + </test> + </tests> + <help> + +.. class:: infomark + +**TIP:** If your data is not TAB delimited, use *Text Manipulation->Convert* + +----- + +**Syntax** + +This tool allows you to group the input dataset by a particular column and perform aggregate functions like Mean, Sum, Max, Min and Concatenate on other columns. + +- All invalid, blank and comment lines are skipped when performing the aggregate functions. The number of skipped lines is displayed in the resulting history item. + +----- + +**Example** + +- For the following input:: + + chr22 1000 NM_17 + chr22 2000 NM_18 + chr10 2200 NM_10 + chr10 1200 NM_11 + chr22 1600 NM_19 + +- running this tool with **Group by column 1**, Operations **Mean on column 2** and **Concatenate on column 3** will return:: + + chr10 1700.00 ['NM_11', 'NM_10'] + chr22 1533.33 ['NM_17', 'NM_19', 'NM_18'] + </help> +</tool>

1 0

New report page - disk usage over time
by Assaf Gordon 18 Jan '09

18 Jan '09

Last report (for today), Shows how much space is consumed each month. At this going rate, my new 1.0TB disk wouldn't last more than six months... :-) Regards, - Gordon.

1 0

New report page - disk usage by file-size
by Assaf Gordon 17 Jan '09

17 Jan '09

Hello, I'd like to suggest another small feature: A report page that shows the largest datasets, with the relevant user information. With this report I can see which users use the most space. The current report (of files over 4GB) is not enough for me - so far I don't have files larger than 4GB, but have many files over 1GB (and less than 1TB of storage space...). As with my previous email - this feature works fine for small Galaxies, where one knows the users personally (and can set them straight, so to speak). With larger installation, this report might not be necessary. Regards, Gordon.

1 0