- galaxy-commits - lists.galaxyproject.org

commit/galaxy-central: kanwei: Restore error icon for color-blindness; needs further testing
by Bitbucket 06 Jun '11

06 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/e4ef73d003f6/ changeset: e4ef73d003f6 branches: user: kanwei date: 2011-06-06 17:12:04 summary: Restore error icon for color-blindness; needs further testing affected #: 1 file (41 bytes) --- a/templates/root/history_common.mako Mon Jun 06 09:37:06 2011 -0400 +++ b/templates/root/history_common.mako Mon Jun 06 11:12:04 2011 -0400 @@ -69,6 +69,7 @@ <a class="icon-button delete tooltip" title="Delete" href="${h.url_for( action='delete', id=data.id, show_deleted_on_refresh=show_deleted_on_refresh )}" id="historyItemDeleter-${data.id}"></a> %endif </div> + <span class="state-icon"></span><span class="historyItemTitle">${hid}: ${data.display_name()}</span></div> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Disable update check in cuffdiff wrapper.
by Bitbucket 06 Jun '11

06 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/1ab3dfb5d929/ changeset: 1ab3dfb5d929 branches: user: jgoecks date: 2011-06-06 15:37:06 summary: Disable update check in cuffdiff wrapper. affected #: 1 file (36 bytes) --- a/tools/ngs_rna/cuffdiff_wrapper.py Mon Jun 06 09:08:59 2011 -0400 +++ b/tools/ngs_rna/cuffdiff_wrapper.py Mon Jun 06 09:37:06 2011 -0400 @@ -101,7 +101,7 @@ try: tmp = tempfile.NamedTemporaryFile().name tmp_stdout = open( tmp, 'wb' ) - proc = subprocess.Popen( args='cuffdiff 2>&1', shell=True, stdout=tmp_stdout ) + proc = subprocess.Popen( args='cuffdiff --no-update-check 2>&1', shell=True, stdout=tmp_stdout ) tmp_stdout.close() returncode = proc.wait() stdout = None @@ -135,7 +135,7 @@ # Build command. # Base; always use quiet mode to avoid problems with storing log output. - cmd = "cuffdiff -q" + cmd = "cuffdiff --no-update-check -q" # Add options. if options.inner_dist_std_dev: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Removed unspecified build validators from Cufflinks/compare/diff tools as these cause trouble when running tools on custom genomes.
by Bitbucket 06 Jun '11

06 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/058a5d7a4f84/ changeset: 058a5d7a4f84 branches: user: jgoecks date: 2011-06-06 15:08:59 summary: Removed unspecified build validators from Cufflinks/compare/diff tools as these cause trouble when running tools on custom genomes. affected #: 3 files (205 bytes) --- a/tools/ngs_rna/cuffcompare_wrapper.xml Fri Jun 03 17:00:20 2011 -0400 +++ b/tools/ngs_rna/cuffcompare_wrapper.xml Mon Jun 06 09:08:59 2011 -0400 @@ -33,9 +33,7 @@ #end for </command><inputs> - <param format="gtf" name="first_input" type="data" label="GTF file produced by Cufflinks" help=""> - <validator type="unspecified_build" /> - </param> + <param format="gtf" name="first_input" type="data" label="GTF file produced by Cufflinks" help=""/><repeat name="input_files" title="Additional GTF Input Files"><param format="gtf" name="additional_input" type="data" label="GTF file produced by Cufflinks" help=""><validator type="unspecified_build" /> --- a/tools/ngs_rna/cuffdiff_wrapper.xml Fri Jun 03 17:00:20 2011 -0400 +++ b/tools/ngs_rna/cuffdiff_wrapper.xml Mon Jun 06 09:08:59 2011 -0400 @@ -67,9 +67,7 @@ </command><inputs> - <param format="gtf" name="gtf_input" type="data" label="Transcripts" help="A transcript GTF file produced by cufflinks, cuffcompare, or other source."> - <validator type="unspecified_build" /> - </param> + <param format="gtf" name="gtf_input" type="data" label="Transcripts" help="A transcript GTF file produced by cufflinks, cuffcompare, or other source."/><conditional name="group_analysis"><param name="do_groups" type="select" label="Perform replicate analysis" help="Perform cuffdiff with replicates in each group."><option value="No">No</option> --- a/tools/ngs_rna/cufflinks_wrapper.xml Fri Jun 03 17:00:20 2011 -0400 +++ b/tools/ngs_rna/cufflinks_wrapper.xml Mon Jun 06 09:08:59 2011 -0400 @@ -41,9 +41,7 @@ #end if </command><inputs> - <param format="sam,bam" name="input" type="data" label="SAM or BAM file of aligned RNA-Seq reads" help=""> - <validator type="unspecified_build" /> - </param> + <param format="sam,bam" name="input" type="data" label="SAM or BAM file of aligned RNA-Seq reads" help=""/><param name="max_intron_len" type="integer" value="300000" min="1" max="600000" label="Max Intron Length" help=""/><param name="min_isoform_fraction" type="float" value="0.05" min="0" max="1" label="Min Isoform Fraction" help=""/><param name="pre_mrna_fraction" type="float" value="0.05" min="0" max="1" label="Pre MRNA Fraction" help=""/> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: kanwei: Remove error icon in front of red datasets, since red clearly denotes fail already
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/aa55fc70f31e/ changeset: aa55fc70f31e branches: user: kanwei date: 2011-06-03 23:00:20 summary: Remove error icon in front of red datasets, since red clearly denotes fail already affected #: 1 file (36 bytes) --- a/templates/root/history_common.mako Fri Jun 03 16:46:14 2011 -0400 +++ b/templates/root/history_common.mako Fri Jun 03 17:00:20 2011 -0400 @@ -29,7 +29,6 @@ </div> %endif - ## Header row for history items (name, state, action buttons) <div style="overflow: hidden;" class="historyItemTitleBar"><div class="historyItemButtons"> @@ -70,7 +69,6 @@ <a class="icon-button delete tooltip" title="Delete" href="${h.url_for( action='delete', id=data.id, show_deleted_on_refresh=show_deleted_on_refresh )}" id="historyItemDeleter-${data.id}"></a> %endif </div> - <span class="state-icon"></span><span class="historyItemTitle">${hid}: ${data.display_name()}</span></div> @@ -192,7 +190,7 @@ ## icons. <div style="clear: both"></div> %endif - <div style="clear: both"></div> + <div style="clear: both"></div> %for display_app in data.datatype.get_display_types(): <% target_frame, display_links = data.datatype.get_display_links( data, display_app, app, request.base ) %> %if len( display_links ) > 0: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Clean up displaying content in the tool shed.
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/7c5dcf903d84/ changeset: 7c5dcf903d84 branches: user: greg date: 2011-06-03 22:46:14 summary: Clean up displaying content in the tool shed. affected #: 3 files (625 bytes) --- a/lib/galaxy/datatypes/checkers.py Fri Jun 03 16:22:02 2011 -0400 +++ b/lib/galaxy/datatypes/checkers.py Fri Jun 03 16:46:14 2011 -0400 @@ -25,9 +25,13 @@ temp.close() return False -def check_binary( file_path ): +def check_binary( name, file_path=True ): + # Handles files if file_path is True or text if file_path is False is_binary = False - temp = open( file_path, "U" ) + if file_path: + temp = open( name, "U" ) + else: + temp = name chars_read = 0 for chars in temp: for char in chars: @@ -39,7 +43,8 @@ break if chars_read > 100: break - temp.close() + if file_path: + temp.close() return is_binary def check_gzip( file_path ): @@ -94,6 +99,10 @@ return True return False +def is_bz2( file_path ): + is_bz2, is_valid = check_bz2( file_path ) + return is_bz2 + def is_gzip( file_path ): is_gzipped, is_valid = check_gzip( file_path ) - return is_gzipped and is_valid + return is_gzipped --- a/lib/galaxy/webapps/community/controllers/repository.py Fri Jun 03 16:22:02 2011 -0400 +++ b/lib/galaxy/webapps/community/controllers/repository.py Fri Jun 03 16:46:14 2011 -0400 @@ -3,6 +3,7 @@ from datetime import * from galaxy import util +from galaxy.datatypes.checkers import * from galaxy.web.base.controller import * from galaxy.webapps.community import model from galaxy.webapps.community.model import directory_hash_id @@ -13,12 +14,12 @@ log = logging.getLogger( __name__ ) -VALID_REPOSITORYNAME_RE = re.compile( "^[a-z0-9\_]+$" ) -# Characters that are valid -VALID_CHARS = set( string.letters + string.digits ) # Characters that must be html escaped MAPPED_CHARS = { '>' :'>', '<' :'<' } +MAX_CONTENT_SIZE = 32768 +VALID_CHARS = set( string.letters + string.digits + "'\"-=_.()/+*^,:?!#[]%\\$@;" ) +VALID_REPOSITORYNAME_RE = re.compile( "^[a-z0-9\_]+$" ) class CategoryListGrid( grids.Grid ): # TODO rename this class to be categoryListGrid when we eliminate all the tools stuff. @@ -173,17 +174,6 @@ repository_list_grid = RepositoryListGrid() category_list_grid = CategoryListGrid() - # TODO: These translations are costly, so figure out a better way... - def from_html_escaped( self, text ): - """Restores sanitized text""" - for key, value in MAPPED_CHARS.items(): - text = text.replace( value, key ) - return text - def to_html_escape_code( self, text ): - """Restricts the characters that are allowed in a text""" - for key, value in MAPPED_CHARS.items(): - text = text.replace( key, value ) - return text @web.expose def index( self, trans, **kwd ): params = util.Params( kwd ) @@ -567,23 +557,9 @@ ctx_parent = ctx.parents()[0] modified, added, removed, deleted, unknown, ignored, clean = repo.status( node1=ctx_parent.node(), node2=ctx.node() ) anchors = modified + added + removed + deleted + unknown + ignored + clean - def is_binary( chars ): - is_binary = False - chars_read = 0 - for char in chars: - chars_read += 1 - if ord( char ) > 128: - is_binary = True - break - return is_binary diffs = [] for diff in patch.diff( repo, node1=ctx_parent.node(), node2=ctx.node() ): - if not util.is_multi_byte( diff ) and not is_binary( diff ): - # TODO: is there a better way? - diffs.append( self.to_html_escape_code( diff ) ) - else: - fixed_diff = diff.split( '\n' )[0] + '\nFile contains non-ascii characters that cannot be displayed\n' - diffs.append( fixed_diff ) + diffs.append( self.to_html_escaped( diff ) ) return trans.fill_template( '/webapps/community/repository/view_changeset.mako', repository=repository, ctx=ctx, @@ -681,28 +657,60 @@ message=message ) ) @web.json def get_file_contents( self, trans, file_path ): - def print_ticks( d ): - # pexpect timeout method - pass # Avoid caching trans.response.headers['Pragma'] = 'no-cache' trans.response.headers['Expires'] = '0' - if os.stat( file_path ).st_size > 32768: - return 'File size larger than maximum viewing size of 32 kb' - cmd = "cat %s" % file_path - # Handle the authentication message if ssh keys are not set - the message is - # something like: "Are you sure you want to continue connecting (yes/no)." - output = pexpect.run( cmd, - events={ pexpect.TIMEOUT : print_ticks }, - timeout=10 ) - output = self.to_html_escape_code( output ) - return unicode( output.replace( '\r\n', '<br/>' ).replace( ' ', ' ' ) ) + if is_gzip( file_path ): + to_html = self.to_html_str( '\ngzip compressed file\n' ) + elif is_bz2( file_path ): + to_html = self.to_html_str( '\nbz2 compressed file\n' ) + elif check_zip( file_path ): + to_html = self.to_html_str( '\nzip compressed file\n' ) + elif check_binary( file_path ): + to_html = self.to_html_str( '\nBinary file\n' ) + else: + to_html = '' + for i, line in enumerate( open( file_path ) ): + to_html = '%s%s' % ( to_html, self.to_html_str( line ) ) + if len( to_html ) > MAX_CONTENT_SIZE: + large_str = '\nFile contents truncated because file size is larger than maximum viewing size of %d\n' % MAX_CONTENT_SIZE + to_html = '%s%s' % ( to_html, self.to_html_str( large_str ) ) + break + return to_html @web.expose def help( self, trans, **kwd ): params = util.Params( kwd ) message = util.restore_text( params.get( 'message', '' ) ) status = params.get( 'status', 'done' ) return trans.fill_template( '/webapps/community/repository/help.mako', message=message, status=status, **kwd ) + def to_html_escaped( self, text ): + """Translates the characters in text to html values""" + translated = [] + for c in text: + if c in [ '\r\n', '\n', ' ', '\t' ] or c in VALID_CHARS: + translated.append( c ) + elif c in MAPPED_CHARS: + translated.append( MAPPED_CHARS[ c ] ) + else: + translated.append( 'X' ) + return ''.join( translated ) + def to_html_str( self, text ): + """Translates the characters in text to sn html string""" + translated = [] + for c in text: + if c in VALID_CHARS: + translated.append( c ) + elif c in MAPPED_CHARS: + translated.append( MAPPED_CHARS[ c ] ) + elif c == ' ': + translated.append( ' ' ) + elif c == '\t': + translated.append( '    ' ) + elif c in [ '\r\n', '\n' ]: + translated.append( '<br/>' ) + else: + translated.append( 'X' ) + return ''.join( translated ) def __build_allow_push_select_field( self, trans, current_push_list, selected_value='none' ): options = [] for user in trans.sa_session.query( trans.model.User ): --- a/templates/webapps/community/repository/common.mako Fri Jun 03 16:22:02 2011 -0400 +++ b/templates/webapps/community/repository/common.mako Fri Jun 03 16:46:14 2011 -0400 @@ -42,7 +42,6 @@ document.upload_form.upload_point.value = selKeys[0]; }, onActivate: function(dtnode) { - // TODO: certainly not ideal - find a better way to display the file contents. var cell = $("#file_contents"); var selected_value; if (dtnode.data.key == 'root') { Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: kanwei: tab -> spaces
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/ca59bd157bba/ changeset: ca59bd157bba branches: user: kanwei date: 2011-06-03 22:22:02 summary: tab -> spaces affected #: 1 file (70 bytes) --- a/tools/ngs_rna/tophat_wrapper.xml Fri Jun 03 16:09:01 2011 -0400 +++ b/tools/ngs_rna/tophat_wrapper.xml Fri Jun 03 16:22:02 2011 -0400 @@ -513,7 +513,7 @@ <param name="sPaired" value="single"/><param name="input1" ftype="fastqsanger" value="tophat_in2.fastqsanger"/><param name="sSettingsType" value="full"/> - <param name="library_type" value="FR Unstranded"/> + <param name="library_type" value="FR Unstranded"/><param name="anchor_length" value="8"/><param name="splice_mismatches" value="0"/><param name="min_intron_length" value="70"/> @@ -525,8 +525,8 @@ <param name="seg_mismatches" value="2"/><param name="seg_length" value="25"/><param name="allow_indel_search" value="Yes"/> - <param name="max_insertion_length" value="3"/> - <param name="max_deletion_length" value="3"/> + <param name="max_insertion_length" value="3"/> + <param name="max_deletion_length" value="3"/><param name="use_junctions" value="Yes" /><param name="use_annotations" value="No" /><param name="use_juncs" value="No" /> @@ -557,7 +557,7 @@ <param name="input2" ftype="fastqsanger" value="tophat_in3.fastqsanger"/><param name="mate_inner_distance" value="20"/><param name="pSettingsType" value="full"/> - <param name="library_type" value="FR Unstranded"/> + <param name="library_type" value="FR Unstranded"/><param name="mate_std_dev" value="20"/><param name="anchor_length" value="8"/><param name="splice_mismatches" value="0"/> @@ -637,13 +637,13 @@ -r This is the expected (mean) inner distance between mate pairs. For, example, for paired end runs with fragments selected at 300bp, where each end is 50bp, you should set -r to be 200. There is no default, and this parameter is required for paired end runs. - --mate-std-dev INT The standard deviation for the distribution on inner distances between mate pairs. The default is 20bp. - -a/--min-anchor-length INT The "anchor length". TopHat will report junctions spanned by reads with at least this many bases on each side of the junction. Note that individual spliced + --mate-std-dev INT The standard deviation for the distribution on inner distances between mate pairs. The default is 20bp. + -a/--min-anchor-length INT The "anchor length". TopHat will report junctions spanned by reads with at least this many bases on each side of the junction. Note that individual spliced alignments may span a junction with fewer than this many bases on one side. However, every junction involved in spliced alignments is supported by at least one - read with this many bases on each side. This must be at least 3 and the default is 8. - -m/--splice-mismatches INT The maximum number of mismatches that may appear in the "anchor" region of a spliced alignment. The default is 0. - -i/--min-intron-length INT The minimum intron length. TopHat will ignore donor/acceptor pairs closer than this many bases apart. The default is 70. - -I/--max-intron-length INT The maximum intron length. When searching for junctions ab initio, TopHat will ignore donor/acceptor pairs farther than this many bases apart, except when such a pair is supported by a split segment alignment of a long read. The default is 500000. + read with this many bases on each side. This must be at least 3 and the default is 8. + -m/--splice-mismatches INT The maximum number of mismatches that may appear in the "anchor" region of a spliced alignment. The default is 0. + -i/--min-intron-length INT The minimum intron length. TopHat will ignore donor/acceptor pairs closer than this many bases apart. The default is 70. + -I/--max-intron-length INT The maximum intron length. When searching for junctions ab initio, TopHat will ignore donor/acceptor pairs farther than this many bases apart, except when such a pair is supported by a split segment alignment of a long read. The default is 500000. -F/--min-isoform-fraction 0.0-1.0 TopHat filters out junctions supported by too few alignments. Suppose a junction spanning two exons, is supported by S reads. Let the average depth of coverage of exon A be D, and assume that it is higher than B. If S / D is less than the minimum isoform fraction, the junction is not reported. A value of zero disables the filter. The default is 0.15. @@ -652,20 +652,20 @@ -G/--GTF [GTF 2.2 file] Supply TopHat with a list of gene model annotations. TopHat will use the exon records in this file to build a set of known splice junctions for each gene, and will attempt to align reads to these junctions even if they would not normally be covered by the initial mapping. -j/--raw-juncs [juncs file] Supply TopHat with a list of raw junctions. Junctions are specified one per line, in a tab-delimited format. Records look like: [chrom] [left] [right] [+/-], left and right are zero-based coordinates, and specify the last character of the left sequenced to be spliced to the first character of the right sequence, inclusive. -no-novel-juncs Only look for junctions indicated in the supplied GFF file. (ignored without -G) - --no-closure-search Disables the mate pair closure-based search for junctions. Currently, has no effect - closure search is off by default. - --closure-search Enables the mate pair closure-based search for junctions. Closure-based search should only be used when the expected inner distance between mates is small (about or less than 50bp) - --no-coverage-search Disables the coverage based search for junctions. - --coverage-search Enables the coverage based search for junctions. Use when coverage search is disabled by default (such as for reads 75bp or longer), for maximum sensitivity. - --microexon-search With this option, the pipeline will attempt to find alignments incident to microexons. Works only for reads 50bp or longer. - --butterfly-search TopHat will use a slower but potentially more sensitive algorithm to find junctions in addition to its standard search. Consider using this if you expect that your experiment produced a lot of reads from pre-mRNA, that fall within the introns of your transcripts. - --segment-mismatches Read segments are mapped independently, allowing up to this many mismatches in each segment alignment. The default is 2. - --segment-length Each read is cut up into segments, each at least this long. These segments are mapped independently. The default is 25. - --min-closure-exon During closure search for paired end reads, exonic hops in the potential splice graph must be at least this long. The default is 50. - --min-closure-intron The minimum intron length that may be found during closure search. The default is 50. - --max-closure-intron The maximum intron length that may be found during closure search. The default is 5000. - --min-coverage-intron The minimum intron length that may be found during coverage search. The default is 50. - --max-coverage-intron The maximum intron length that may be found during coverage search. The default is 20000. - --min-segment-intron The minimum intron length that may be found during split-segment search. The default is 50. - --max-segment-intron The maximum intron length that may be found during split-segment search. The default is 500000. + --no-closure-search Disables the mate pair closure-based search for junctions. Currently, has no effect - closure search is off by default. + --closure-search Enables the mate pair closure-based search for junctions. Closure-based search should only be used when the expected inner distance between mates is small (about or less than 50bp) + --no-coverage-search Disables the coverage based search for junctions. + --coverage-search Enables the coverage based search for junctions. Use when coverage search is disabled by default (such as for reads 75bp or longer), for maximum sensitivity. + --microexon-search With this option, the pipeline will attempt to find alignments incident to microexons. Works only for reads 50bp or longer. + --butterfly-search TopHat will use a slower but potentially more sensitive algorithm to find junctions in addition to its standard search. Consider using this if you expect that your experiment produced a lot of reads from pre-mRNA, that fall within the introns of your transcripts. + --segment-mismatches Read segments are mapped independently, allowing up to this many mismatches in each segment alignment. The default is 2. + --segment-length Each read is cut up into segments, each at least this long. These segments are mapped independently. The default is 25. + --min-closure-exon During closure search for paired end reads, exonic hops in the potential splice graph must be at least this long. The default is 50. + --min-closure-intron The minimum intron length that may be found during closure search. The default is 50. + --max-closure-intron The maximum intron length that may be found during closure search. The default is 5000. + --min-coverage-intron The minimum intron length that may be found during coverage search. The default is 50. + --max-coverage-intron The maximum intron length that may be found during coverage search. The default is 20000. + --min-segment-intron The minimum intron length that may be found during split-segment search. The default is 50. + --max-segment-intron The maximum intron length that may be found during split-segment search. The default is 500000. </help></tool> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: kanwei: Add dataset used for blastxml unit test
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/734e145779e4/ changeset: 734e145779e4 branches: user: kanwei date: 2011-06-03 22:09:01 summary: Add dataset used for blastxml unit test affected #: 1 file (0 bytes) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: kanwei: Grouping tool:
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/7ee052a1f4f2/ changeset: 7ee052a1f4f2 branches: user: kanwei date: 2011-06-03 22:08:26 summary: Grouping tool: - Converted to use numpy, which is included with Galaxy, instead of rpy, which is not. - Don't simply skip lines if a numeric parameter encounters non-numeric data, return descriptive error with line contents instead - General refactoring - Added new comprehensive test, disabled until test framework supports multiple repeat inputs affected #: 3 files (3.3 KB) --- a/tools/stats/grouping.py Fri Jun 03 15:36:59 2011 -0400 +++ b/tools/stats/grouping.py Fri Jun 03 16:08:26 2011 -0400 @@ -1,73 +1,58 @@ #!/usr/bin/env python # Guruprasad Ananda # Refactored 2011, Kanwei Li +# Refactored to use numpy instead of rpy """ This tool provides the SQL "group by" functionality. """ -import sys, string, re, commands, tempfile, random -from rpy import * +import sys, commands, tempfile, random +import pkg_resources +pkg_resources.require( "numpy" ) +import numpy + from itertools import groupby def stop_err(msg): sys.stderr.write(msg) sys.exit() +def mode(data): + counts = {} + for x in data: + counts[x] = counts.get(x,0) + 1 + maxcount = max(counts.values()) + modelist = [] + for x in counts: + if counts[x] == maxcount: + modelist.append( str(x) ) + return ','.join(modelist) + def main(): inputfile = sys.argv[2] ignorecase = int(sys.argv[4]) ops = [] cols = [] - rounds = [] - elems = [] + round_val = [] + data_ary = [] for var in sys.argv[5:]: - ops.append(var.split()[0]) - cols.append(var.split()[1]) - rounds.append(var.split()[2]) - - if 'Mode' in ops: - try: - r.library('prettyR') - except: - stop_err('R package prettyR could not be loaded. Please make sure it is installed.') - + op, col, do_round = var.split() + ops.append(op) + cols.append(col) + round_val.append(do_round) """ At this point, ops, cols and rounds will look something like this: ops: ['mean', 'min', 'c'] cols: ['1', '3', '4'] - rounds: ['no', 'yes' 'no'] + round_val: ['no', 'yes' 'no'] """ - - for i, line in enumerate( file ( inputfile )): - line = line.rstrip('\r\n') - if len( line )>0 and not line.startswith( '#' ): - elems = line.split( '\t' ) - break - if i == 30: - break # Hopefully we'll never get here... - - if len( elems )<1: - stop_err( "The data in your input dataset is either missing or not formatted properly." ) - + try: group_col = int( sys.argv[3] )-1 except: stop_err( "Group column not specified." ) str_ops = ['c', 'length', 'unique', 'random', 'cuniq', 'Mode'] #ops that can handle string/non-numeric inputs - for k, col in enumerate(cols): - col = int(col)-1 - if ops[k] not in str_ops: - # We'll get here only if the user didn't choose 'Concatenate' or 'Count' or 'Count Distinct' or 'pick randomly', which are the - # only aggregation functions that can be used on columns containing strings. - try: - float( elems[col] ) - except: - try: - msg = "Operation '%s' cannot be performed on non-numeric column %d containing value '%s'." %( ops[k], col+1, elems[col] ) - except: - msg = "Operation '%s' cannot be performed on non-numeric data." %ops[k] - stop_err( msg ) tmpfile = tempfile.NamedTemporaryFile() @@ -83,7 +68,7 @@ case = '' if ignorecase == 1: case = '-f' - command_line = "sort -t ' ' " + case + " -k" + str(group_col+1) +"," + str(group_col+1) + " -o " + tmpfile.name + " " + inputfile + command_line = "sort -t ' ' %s -k%s,%s -o %s %s" % (case, group_col+1, group_col+1, tmpfile.name, inputfile) except Exception, exc: stop_err( 'Initialization error -> %s' %str(exc) ) @@ -92,21 +77,16 @@ if error_code != 0: stop_err( "Sorting input dataset resulted in error: %s: %s" %( error_code, stdout )) - prev_item = None - skipped_lines = 0 - first_invalid_line = None - invalid_value = '' - invalid_column = 0 fout = open(sys.argv[1], "w") def is_new_item(line): item = line.strip().split("\t")[group_col] if ignorecase == 1: - item = item.lower() + return item.lower() return item for key, line_list in groupby(tmpfile, key=is_new_item): - op_vals = [ [] for op in cols ] + op_vals = [ [] for op in ops ] out_str = key multiple_modes = False mode_index = None @@ -115,86 +95,65 @@ fields = line.strip().split("\t") for i, col in enumerate(cols): col = int(col)-1 # cXX from galaxy is 1-based - val = fields[col].strip() - # Before appending the current value, make sure it is numeric if the - # operation for the column requires it. - if ops[i] not in str_ops: - try: - float(val) - except ValueError: - skipped_lines += 1 - if first_invalid_line is None: - first_invalid_line = i+1 - invalid_value = fields[col] - invalid_column = col+1 - break + try: + val = fields[col].strip() + op_vals[i].append(val) + except IndexError: + sys.stderr.write( 'Could not access the value for column %s on line: "%s". Make sure file is tab-delimited.\n' % (col+1, line) ) + sys.exit( 1 ) - op_vals[i].append(val) + # Generate string for each op for this group + for i, op in enumerate( ops ): + data = op_vals[i] + rval = "" + if op == "mode": + rval = mode( data ) + elif op == "length": + rval = len( data ) + elif op == "random": + rval = random.choice(data) + elif op in ['cat', 'cat_uniq']: + if op == 'cat_uniq': + data = numpy.unique(data) + rval = ','.join(data) + elif op == "unique": + rval = len( numpy.unique(data) ) + else: + # some kind of numpy fn + try: + data = map(float, data) + except ValueError: + sys.stderr.write( "Operation %s expected number values but got %s instead.\n" % (op, data) ) + sys.exit( 1 ) + rval = getattr(numpy, op)( data ) + if round_val[i] == 'yes': + rval = round(rval) + else: + rval = '%g' % rval + + out_str += "\t%s" % rval - for i, op in enumerate( ops ): - if op == 'cuniq': - rfunc = "r.c" - else: - rfunc = "r." + op - if op not in str_ops: - for j, elem in enumerate( op_vals[i] ): - op_vals[i][j] = float( elem ) - rout = eval( rfunc )( op_vals[i] ) - if rounds[i] == 'yes': - rout = round(float(rout)) - else: - rout = '%g' %(float(rout)) - else: - if op != 'random': - rout = eval( rfunc )( op_vals[i] ) - else: - try: - rand_index = random.randint(0,len(op_vals[i])-1) #if the two inputs to randint are equal, it seems to throw a ValueError. This can't be reproduced with the python interpreter in its interactive mode. - except: - rand_index = 0 - rout = op_vals[i][rand_index] - - if op == 'Mode' and rout == '>1 mode': - multiple_modes = True - mode_index = i - if op == 'unique': - rfunc = "r.length" - rout = eval( rfunc )( rout ) - if op in ['c', 'cuniq']: - if isinstance(rout, list): - if op == 'cuniq': - rout = set(rout) - out_str += "\t" + ','.join(rout) - else: - out_str += "\t" + str(rout) - else: - out_str += "\t" + str(rout) - if multiple_modes and mode_index != None: - out_str_list = out_str.split('\t') - for val in op_vals[mode_index]: - out_str = '\t'.join(out_str_list[:mode_index+1]) + '\t' + str(val) + '\t' + '\t'.join(out_str_list[mode_index+2:]) - fout.write(out_str.rstrip('\t') + "\n") - else: - fout.write(out_str + "\n") + fout.write(out_str + "\n") # Generate a useful info message. msg = "--Group by c%d: " %(group_col+1) for i, op in enumerate(ops): - if op == 'c': + if op == 'cat': op = 'concat' + elif op == 'cat_uniq': + op = 'concat_distinct' elif op == 'length': op = 'count' elif op == 'unique': op = 'count_distinct' elif op == 'random': op = 'randomly_pick' - elif op == 'cuniq': - op = 'concat_distinct' + msg += op + "[c" + cols[i] + "] " - if skipped_lines > 0: - msg+= "--skipped %d invalid lines starting with line %d. Value '%s' in column %d is not numeric." % ( skipped_lines, first_invalid_line, invalid_value, invalid_column ) print msg + fout.close() + tmpfile.close() if __name__ == "__main__": main() --- a/tools/stats/grouping.xml Fri Jun 03 15:36:59 2011 -0400 +++ b/tools/stats/grouping.xml Fri Jun 03 16:08:26 2011 -0400 @@ -1,4 +1,4 @@ -<tool id="Grouping1" name="Group" version="1.9.4"> +<tool id="Grouping1" name="Group" version="2.0.0"><description>data by a column and perform aggregate operation on other columns.</description><command interpreter="python"> grouping.py @@ -22,16 +22,16 @@ <param name="optype" type="select" label="Type"><option value="mean">Mean</option><option value="median">Median</option> - <option value="Mode">Mode</option> + <option value="mode">Mode</option><option value="max">Maximum</option><option value="min">Minimum</option><option value="sum">Sum</option><option value="length">Count</option><option value="unique">Count Distinct</option> - <option value="c">Concatenate</option> - <option value="cuniq">Concatenate Distinct</option> + <option value="cat">Concatenate</option> + <option value="cat_uniq">Concatenate Distinct</option><option value="random">Randomly pick</option> - <option value="sd">Standard deviation</option> + <option value="std">Standard deviation</option></param><param name="opcol" label="On column" type="data_column" data_ref="input1" /><param name="opround" type="select" label="Round result to nearest integer?"> @@ -44,7 +44,7 @@ <data format="tabular" name="out_file1" /></outputs><requirements> - <requirement type="python-module">rpy</requirement> + <requirement type="python-module">numpy</requirement></requirements><tests> @@ -57,8 +57,16 @@ <param name="opround" value="no"/><output name="out_file1" file="groupby_out1.dat"/></test> - -  +  + </tests><help> @@ -79,9 +88,7 @@ **Syntax** -This tool allows you to group the input dataset by a particular column and perform aggregate functions like Mean, Median, Mode, Sum, Max, Min, Count, Random draw and Concatenate on other columns. - -- All invalid, blank and comment lines are skipped when performing the aggregate functions. The number of skipped lines is displayed in the resulting history item. +This tool allows you to group the input dataset by a particular column and perform aggregate functions: Mean, Median, Mode, Sum, Max, Min, Count, Randomly pick, and Concatenate on any column. - If multiple modes are present, all are reported. Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Move duplicate data type checker methods from sniff and upload into a new ~/datatypes/checkers.py.
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/c2bc73b017cb/ changeset: c2bc73b017cb branches: user: greg date: 2011-06-03 21:36:59 summary: Move duplicate data type checker methods from sniff and upload into a new ~/datatypes/checkers.py. affected #: 3 files (4.4 KB) --- a/lib/galaxy/datatypes/sniff.py Fri Jun 03 15:18:31 2011 -0400 +++ b/lib/galaxy/datatypes/sniff.py Fri Jun 03 15:36:59 2011 -0400 @@ -4,6 +4,7 @@ import logging, sys, os, csv, tempfile, shutil, re, zipfile, gzip import registry from galaxy import util +from galaxy.datatypes.checkers import * from galaxy.datatypes.binary import unsniffable_binary_formats log = logging.getLogger(__name__) @@ -319,59 +320,6 @@ return 'tabular' #default tabular data type file extension return 'txt' #default text data type file extension - -#Methods Used below can be used to upload new datasets into Galaxy. Currently used by the data_source.py script/tools. -#These should be further abstracted and merged with upload.py script/tool functionality. -def is_gzip( filename ): - temp = open( filename, "U" ) - magic_check = temp.read( 2 ) - temp.close() - if magic_check != util.gzip_magic: - return False - return True - - -def is_binary( filename ): - is_binary = False - temp = open( filename, "U" ) - chars_read = 0 - for chars in temp: - for char in chars: - chars_read += 1 - if ord( char ) > 128: - is_binary = True - break - if chars_read > 100: - break - if chars_read > 100: - break - temp.close() - return is_binary - -def is_html( temp_name, chunk=None ): - if chunk is None: - temp = open(temp_name, "U") - else: - temp = chunk - regexp1 = re.compile( "<A\s+[^>]*HREF[^>]+>", re.I ) - regexp2 = re.compile( "<IFRAME[^>]*>", re.I ) - regexp3 = re.compile( "<FRAMESET[^>]*>", re.I ) - regexp4 = re.compile( "<META[^>]*>", re.I ) - regexp5 = re.compile( "<SCRIPT[^>]*>", re.I ) - lineno = 0 - for line in temp: - lineno += 1 - matches = regexp1.search( line ) or regexp2.search( line ) or regexp3.search( line ) or regexp4.search( line ) or regexp5.search( line ) - if matches: - if chunk is None: - temp.close() - return True - if lineno > 100: - break - if chunk is None: - temp.close() - return False - def handle_compressed_file( filename, datatypes_registry, ext = 'auto' ): CHUNK_SIZE = 2**20 # 1Mb is_compressed = False @@ -429,10 +377,10 @@ if ext in AUTO_DETECT_EXTENSIONS: ext = guess_ext( filename, sniff_order = datatypes_registry.sniff_order, is_multi_byte=is_multi_byte ) - if is_binary( filename ): + if check_binary( filename ): if ext not in unsniffable_binary_formats and not datatypes_registry.get_datatype_by_extension( ext ).sniff( filename ): raise InappropriateDatasetContentError, 'The binary uploaded file contains inappropriate content.' - elif is_html( filename ): + elif check_html( filename ): raise InappropriateDatasetContentError, 'The uploaded file contains inappropriate HTML content.' return ext @@ -449,4 +397,3 @@ if __name__ == '__main__': import doctest, sys doctest.testmod(sys.modules[__name__]) - --- a/tools/data_source/upload.py Fri Jun 03 15:18:31 2011 -0400 +++ b/tools/data_source/upload.py Fri Jun 03 15:36:59 2011 -0400 @@ -8,6 +8,7 @@ from galaxy import eggs # need to import model before sniff to resolve a circular import dependency import galaxy.model +from galaxy.datatypes.checkers import * from galaxy.datatypes import sniff from galaxy.datatypes.binary import * from galaxy.datatypes.images import Pdf @@ -48,104 +49,20 @@ return [safe_dict(x) for x in d] else: return d -def check_html( temp_name, chunk=None ): - if chunk is None: - temp = open(temp_name, "U") - else: - temp = chunk - regexp1 = re.compile( "<A\s+[^>]*HREF[^>]+>", re.I ) - regexp2 = re.compile( "<IFRAME[^>]*>", re.I ) - regexp3 = re.compile( "<FRAMESET[^>]*>", re.I ) - regexp4 = re.compile( "<META[^>]*>", re.I ) - regexp5 = re.compile( "<SCRIPT[^>]*>", re.I ) - lineno = 0 - for line in temp: - lineno += 1 - matches = regexp1.search( line ) or regexp2.search( line ) or regexp3.search( line ) or regexp4.search( line ) or regexp5.search( line ) - if matches: - if chunk is None: - temp.close() - return True - if lineno > 100: - break - if chunk is None: - temp.close() - return False -def check_binary( temp_name ): - is_binary = False - temp = open( temp_name, "U" ) - chars_read = 0 - for chars in temp: - for char in chars: - chars_read += 1 - if ord( char ) > 128: - is_binary = True - break - if chars_read > 100: - break - if chars_read > 100: - break - temp.close() - return is_binary -def check_bam( temp_name ): - return Bam().sniff( temp_name ) -def check_sff( temp_name ): - return Sff().sniff( temp_name ) -def check_pdf( temp_name ): - return Pdf().sniff( temp_name ) -def check_bigwig( temp_name ): - return BigWig().sniff( temp_name ) -def check_bigbed( temp_name ): - return BigBed().sniff( temp_name ) -def check_gzip( temp_name ): - # This method returns a tuple of booleans representing ( is_gzipped, is_valid ) - # Make sure we have a gzipped file - try: - temp = open( temp_name, "U" ) - magic_check = temp.read( 2 ) - temp.close() - if magic_check != util.gzip_magic: - return ( False, False ) - except: - return ( False, False ) - # We support some binary data types, so check if the compressed binary file is valid - # If the file is Bam, it should already have been detected as such, so we'll just check - # for sff format. - try: - header = gzip.open( temp_name ).read(4) - if binascii.b2a_hex( header ) == binascii.hexlify( '.sff' ): - return ( True, True ) - except: - return( False, False ) - CHUNK_SIZE = 2**15 # 32Kb - gzipped_file = gzip.GzipFile( temp_name, mode='rb' ) - chunk = gzipped_file.read( CHUNK_SIZE ) - gzipped_file.close() - # See if we have a compressed HTML file - if check_html( temp_name, chunk=chunk ): - return ( True, False ) - return ( True, True ) -def check_bz2( temp_name ): - try: - temp = open( temp_name, "U" ) - magic_check = temp.read( 3 ) - temp.close() - if magic_check != util.bz2_magic: - return ( False, False ) - except: - return( False, False ) - CHUNK_SIZE = 2**15 # reKb - bzipped_file = bz2.BZ2File( temp_name, mode='rb' ) - chunk = bzipped_file.read( CHUNK_SIZE ) - bzipped_file.close() - # See if we have a compressed HTML file - if check_html( temp_name, chunk=chunk ): - return ( True, False ) - return ( True, True ) -def check_zip( temp_name ): - if zipfile.is_zipfile( temp_name ): - return True - return False +def check_bam( file_path ): + return Bam().sniff( file_path ) + +def check_sff( file_path ): + return Sff().sniff( file_path ) + +def check_pdf( file_path ): + return Pdf().sniff( file_path ) + +def check_bigwig( file_path ): + return BigWig().sniff( file_path ) + +def check_bigbed( file_path ): + return BigBed().sniff( file_path ) def parse_outputs( args ): rval = {} for arg in args: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: natefoo: Prevent upload job creation from clobbering the input if linking data into Galaxy.
by Bitbucket 03 Jun '11

03 Jun '11

1 new changeset in galaxy-central: http://bitbucket.org/galaxy/galaxy-central/changeset/38fe5eeb1d6f/ changeset: 38fe5eeb1d6f branches: user: natefoo date: 2011-06-03 21:18:31 summary: Prevent upload job creation from clobbering the input if linking data into Galaxy. affected #: 2 files (338 bytes) --- a/lib/galaxy/jobs/__init__.py Fri Jun 03 11:25:02 2011 -0400 +++ b/lib/galaxy/jobs/__init__.py Fri Jun 03 15:18:31 2011 -0400 @@ -752,7 +752,10 @@ sizes = [] output_paths = self.get_output_fnames() for outfile in [ str( o ) for o in output_paths ]: - sizes.append( ( outfile, os.stat( outfile ).st_size ) ) + if os.path.exists( outfile ): + sizes.append( ( outfile, os.stat( outfile ).st_size ) ) + else: + sizes.append( ( outfile, 0 ) ) return sizes def setup_external_metadata( self, exec_dir = None, tmp_dir = None, dataset_files_path = None, config_root = None, datatypes_config = None, set_extension = True, **kwds ): @@ -999,7 +1002,10 @@ sizes = [] output_paths = self.get_output_fnames() for outfile in [ str( o ) for o in output_paths ]: - sizes.append( ( outfile, os.stat( outfile ).st_size ) ) + if os.path.exists( outfile ): + sizes.append( ( outfile, os.stat( outfile ).st_size ) ) + else: + sizes.append( ( outfile, 0 ) ) return sizes def setup_external_metadata( self, exec_dir = None, tmp_dir = None, dataset_files_path = None, config_root = None, datatypes_config = None, set_extension = True, **kwds ): --- a/lib/galaxy/tools/actions/upload_common.py Fri Jun 03 11:25:02 2011 -0400 +++ b/lib/galaxy/tools/actions/upload_common.py Fri Jun 03 15:18:31 2011 -0400 @@ -323,12 +323,14 @@ for i, dataset in enumerate( data_list ): job.add_output_library_dataset( 'output%i' % i, dataset ) # Create an empty file immediately - open( dataset.file_name, "w" ).close() + if not dataset.dataset.external_filename: + open( dataset.file_name, "w" ).close() else: for i, dataset in enumerate( data_list ): job.add_output_dataset( 'output%i' % i, dataset ) # Create an empty file immediately - open( dataset.file_name, "w" ).close() + if not dataset.dataset.external_filename: + open( dataset.file_name, "w" ).close() job.state = job.states.NEW trans.sa_session.add( job ) trans.sa_session.flush() Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0