- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Fwd: [galaxy-user] starting workflows with a pool of input files
by Matthias Dodt 12 Oct '09

12 Oct '09

Okay Thanks;-)! greetings mat Dominic Tolle schrieb: > fyI > > Begin forwarded message: > >> *From: *Greg Von Kuster <ghv2(a)psu.edu <mailto:ghv2@psu.edu>> >> *Date: *October 8, 2009 3:34:35 PM GMT+02:00 >> *To: *Matthias Dodt <matthias.dodt(a)mdc-berlin.de >> <mailto:matthias.dodt@mdc-berlin.de>> >> *Cc: *galaxy <galaxy-user(a)bx.psu.edu <mailto:galaxy-user@bx.psu.edu>> >> *Subject: **Re: [galaxy-user] starting workflows with a pool of input >> files* >> >> Hello Matthias, >> >> I've opened the following ticket for this question - you can "follow" >> the ticket in bitbucket if you want. >> >> http://bitbucket.org/galaxy/galaxy-central/issue/197/starting-workflows-wit… >> >> Greg Von Kuster >> Galaxy Development Team >> >> >> Matthias Dodt wrote: >>> Hi there! >>> >>> I would like to start a workflow several times (with a pool of input >>> files). >>> >>> The reason is: >>> Lets say we process huge FASTA files. To do so, i incorporated a >>> splitting tool into our galaxy, which creates several smaller >>> split-files, which can be more easily processed on a cluster in >>> parallel. Now i would like to start a workflow with these splitfiles as >>> input (without starting it manually by clicking lets say 100 times - >>> for >>> each split-file - the same procedure). >>> >>> Is there an easy way to do that? >>> >>> thanks! >>> >>> greetings >>> >>> mat >>> _______________________________________________ >>> galaxy-user mailing list >>> galaxy-user(a)bx.psu.edu >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> >>> >> _______________________________________________ >> galaxy-user mailing list >> galaxy-user(a)bx.psu.edu >> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >

1 0

Starting Workflow n-times
by Matthias Dodt 12 Oct '09

12 Oct '09

Hi there! I already posted this question to the user list but didnt get any answer - would be nice;-)! I would like to start a workflow several times (with a pool of input files). The reason is: Lets say we process huge FASTA files. To do so, i incorporated a splitting tool into our galaxy, which creates several smaller split-files, which can be more easily processed on a cluster in parallel. Now i would like to start a workflow with these splitfiles as input (without starting it manually by clicking 100 times - for each split-file - the same procedure). How could that be accomplished? thanks! greetings mat

1 0

NGS impressions (long)
by Gordon, Assaf 10 Oct '09

10 Oct '09

Hello, I'd like to share couple of first impressions after testing the updated NGS version. =============== Biggest issue for me is the support for Solexa formats. (Obiovusly, I'm biased, and think that Solexa machines are much more common/useful than other types). Uploading a FASTQ file automatically detects the type (fastqsolexa or fastqsanger). That's a good start, but there's a catch: If I'm not aware that there are multiple types of FASTQ files - I'm stuck. Bowtie & BWA tools don't display my FASTQ file as possible input dataset. A novice user will need to notice two things: 1. the small warning on the tools' pages: "BWA accepts files in Sanger FASTQ format." "Must have Sanger-scaled quality values with ASCII offset 33 " 2. the fact the the file type is "fastqsolexa" (on the expanded green dataset rectangle). Even if he does notice, there's no way for him to know what to do... The easier (and wrong) solution is to click the pencil icon and change the type to "fastqsanger". It's wrong because the quality scores are not re-scaled, but ironically it kind of works and bowtie even returns pretty good results (just to show you how much those quality scores are worth). Another way (which is also not perfect) is to use the "Quality Format Converter" twice: first from ASCII(64) to Numeric, and then from numeric to ASCII(33). This still doesn't re-scale the quality scores, but at least they are now with offset 33. As the guy who actually wrote the "Quality Format convert" tool, let me tell you that the tool's interface is not understandable at all (that's my fault, of course). It is not clear if the "FASTQ ASCII OFFSET" input parameter refers to the input format or output format. What another user tried to do (seemed very logical from his POV), is to run the tool just ONCE: Select a 'fastqsolexa' file (with ASCII 64) as input dataset, and select ASCII and 33 as the tool's parameters (hoping to convert ASCII-64 to ASCII-33). This obviously doesn't work (the tool returns an error). Basically, if you've uploaded a solexa FASTQ file, you're stuck - but it's worse then just being stuck - you don't know why you're stuck or how to fix it. At least for bowtie, there's a command line argument to accept different scales - so you could build an XML file that adds those command line arguments based on the file type, e.g.: ================= <command> bwa_wrapper.py [ gzillion other options ] #if $input.ext == "fastqsolexa" --solexa-qual #else --pthread33-quals #endif </command> ================ This will not solve the 33/64 issue, or the 1.3 vs 1.4 quality scores, but might still be better. Since I don't trust the quality scores at all, and most of the time prefer to discard them, The thing I'd most like to do is be able to map FASTA (not FASTQ) files. bowtie supports this (with "-f"), and with another "if" in the XML this can be done (I think). =============== Another issue is selecting the organism. Galaxy supports so many of them, that selecting the one you want becomes a bit annoying. The "Upload File" starts with "--Additional Species are Below --" selected (half-way down the list). I need to scroll up if my organism is one of common ones, or down if it's an "additional". Another rant about forgotten usability in the Web2.0 era: on every normal operating system, typing the first couple of letters will scroll the list down to a line starting with those letters. But since we start half-way down the list, typing "hu" (for human) gets me to "Human Herpesvirus 1" instead of "human (hg18)". Right now, the easiest way I found to select 'human' is: 1. move focus to the list box (with TAB, not by clicking the mouse, of course) 2. click the "home" key on the keyboard (if you use a Mac, I don't know what key you need for that...) 3. click "h", then "u" - I get to the "human GRCh37 (hg19)" 4. click "down arrow" once to get to the 'right' human (hg18). And believe me, all you mice-loving people - my keyboard way is faster ;) And after spending so much effort on selecting the right organism at upload time, and with most of the other Galaxy tools respecting (and even requiring) the dbkey to be set - Having a stand-alone list-box in the Bowtie/BWA tools to select a reference organism database is downright insulting. Also, the default for BWA is "hg18" and the default for Bowtie is "hg19" - a bit confusing. Note that even the UCSC Genome Browser still has "hg18" as default (not "hg19"). My greedy suggestion: An edit-box with auto completion. And when you auto-complete, complete both with the short names (e.g. "hg" -> "hg19/hg18/hg17") and with full names (e.g "hu" -> "human") and with mid-string names (e.g "viri" => "Dro. virilis") AND expand the genus name too (because I want to be able to write "dro" and get to the Drosophilas, I don't want to write "D. "). Better yet, also add couple of quick selections (one button per organism) for the most common ones. ================= Another issue is the index database file used for bowtie and bwa. I don't know if you've downloaded the pre-built one or built it yourself, but the index file you have contains useless (IMHO) identifiers. Here's a sample of bowtie mapping result: gi|51511727|ref|NC_000011.8|NC_000011 125559911 25 In short: WTF? Yes, I learned (after a quick search) that "NC_000011.8" is simply "chr11". But please, in the name of all that is holy, rebuild the indexes with the commonly known chromosome names. Same goes for BWA, which is only slightly better: chr17_random.nib:1-2617613 492542 0 Hurrah, I my sequence mapped to "chr17_random.nib:1-2617613" .... This is not just me being spoiled - when I use the "SAM to Interval" tool to get normal mapping results[*], I get: ============ chr17_random.nib:1-2617613 510910 510946 + chr17_random.nib:1-2617613 512402 512438 + chr13.nib:1-114142980 45846690 45846726 - chr17_random.nib:1-2617613 492541 492577 - =========== see http://main.g2.bx.psu.edu/datasets/841134/display/index This will never work with the UCSC Genome browser (or any other sane browser, for that matter), so the interval file is useless. =========================== Another issue is the accuracy of BWA/Bowtie results. Take a look at the following history: http://main.g2.bx.psu.edu/history/imp?id=72116af6496fcec3 It contains just one sequences in fastqsolexa format (dataset 2). It's a real sequences from a real FASTQ file (except for the machine name and flowcell number, which I've changed). This is the sequence: ATCGCTTCTCGGCCTTTTGGCNAAGATCAAGTGTAG Dataset 3 is conversion to numeric values. Dataset 8 is conversion to ASCII-33 values. Dataset 9 is BWA mapping, just one result (chrom 17) with one mismatch: FOOBAR_27_FC42AGAAXX:1:1:2:435 0 chr17_random.nib:1-2617613 510911 0 36M Dataset 10 is Bowtie mapping, just one result (chrom 9) with one mismatch: FOOBAR_27_FC42AGAAXX:1:1:2:435 16 gi|89161216|ref|NC_000009.10|NC_000009 18530316 0 As a sanity check, run it through BLAT on the UCSC Genome browser (with hg18): You'll get 25 (!) results with 1 mismatch, and many more with more mismatches. The blat results include the chr9 (from BWA) and chr17 (from Bowtie). There's no way that the quality scores in the input FASTQ affected the mapping so much that BWA and Bowtie can decide to ignore all coordinates except one (and not even the same one). I've written it before - running bowtie and BWA with the default parameters and without understanding the complexities of those programs is down right misleading (IMHO). A novice user who just wants to get results fast - will be partial results at best (especially when looking for "unique mappers"). The "commonly used" parameters for BWA and Bowtie are not right for everyone (not right for most, even). Offering them as the default without a big flashing red warning is a disservice. ======= Thanks for reading so far. Regardless, galaxy is still the best thing since sliced bread, many people here use it daily and it makes everyone's life a lot easier - so don't take the above too hard. -gordon [*] not sure if it's obvious, but I despise the "SAM" format.

3 3

[hg] galaxy 2861: Fixed bug that prevented copying datasets to a...
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/c7dd346c20d5 changeset: 2861:c7dd346c20d5 user: jeremy goecks <jeremy.goecks at emory.edu> date: Thu Oct 08 22:01:36 2009 -0400 description: Fixed bug that prevented copying datasets to a new history. 1 file(s) affected in this change: lib/galaxy/web/controllers/dataset.py diffs (12 lines): diff -r 682cde08c3d5 -r c7dd346c20d5 lib/galaxy/web/controllers/dataset.py --- a/lib/galaxy/web/controllers/dataset.py Thu Oct 08 21:02:36 2009 -0400 +++ b/lib/galaxy/web/controllers/dataset.py Thu Oct 08 22:01:36 2009 -0400 @@ -363,7 +363,7 @@ new_history = trans.app.model.History() if new_history_name: new_history.name = new_history_name - new_history_name = user + new_history.user = user new_history.flush() target_history_ids.append( new_history.id ) if user:

1 0

[hg] galaxy 2860: peeking large datasets only displays a portion...
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/682cde08c3d5 changeset: 2860:682cde08c3d5 user: Kanwei Li <kanwei(a)gmail.com> date: Thu Oct 08 21:02:36 2009 -0400 description: peeking large datasets only displays a portion, but allows one to override or save 2 file(s) affected in this change: lib/galaxy/web/controllers/dataset.py templates/dataset/large_file.mako diffs (64 lines): diff -r e02620ad3f41 -r 682cde08c3d5 lib/galaxy/web/controllers/dataset.py --- a/lib/galaxy/web/controllers/dataset.py Thu Oct 08 18:07:02 2009 -0400 +++ b/lib/galaxy/web/controllers/dataset.py Thu Oct 08 21:02:36 2009 -0400 @@ -199,7 +199,7 @@ return 'This link may not be followed from within Galaxy.' @web.expose - def display(self, trans, dataset_id=None, filename=None, **kwd): + def display(self, trans, dataset_id=None, filename=None, show_all=False, **kwd): """Catches the dataset id and displays file contents as directed""" data = trans.app.model.HistoryDatasetAssociation.get( dataset_id ) if not data: @@ -209,23 +209,29 @@ if data.state == trans.model.Dataset.states.UPLOAD: return trans.show_error_message( "Please wait until this dataset finishes uploading before attempting to view it." ) if filename is None or filename.lower() == "index": + file_path = data.file_name mime = trans.app.datatypes_registry.get_mimetype_by_extension( data.extension.lower() ) trans.response.set_content_type(mime) trans.log_event( "Display dataset id: %s" % str( dataset_id ) ) - try: - return open( data.file_name ) - except: - raise paste.httpexceptions.HTTPNotFound( "File Not Found (%s)." % ( filename ) ) + else: file_path = os.path.join( data.extra_files_path, filename ) mime, encoding = mimetypes.guess_type( file_path ) if mime is None: mime = trans.app.datatypes_registry.get_mimetype_by_extension( ".".split( file_path )[-1] ) trans.response.set_content_type( mime ) - try: + + if os.path.exists( file_path ): + max_peek_size = 1000000 # 1 MB + if show_all or os.stat( file_path ).st_size < max_peek_size: return open( file_path ) - except: - raise paste.httpexceptions.HTTPNotFound( "File Not Found (%s)." % ( filename ) ) + else: + trans.response.set_content_type( "text/html" ) + return trans.fill_template( "/dataset/large_file.mako", + truncated_data = open( file_path ).read(max_peek_size), + data = data ) + else: + raise paste.httpexceptions.HTTPNotFound( "File Not Found (%s)." % ( filename ) ) else: return trans.show_error_message( "You are not allowed to access this dataset" ) diff -r e02620ad3f41 -r 682cde08c3d5 templates/dataset/large_file.mako --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/templates/dataset/large_file.mako Thu Oct 08 21:02:36 2009 -0400 @@ -0,0 +1,11 @@ +<%inherit file="/base.mako"/> + +<div class="warningmessagelarge"> + This dataset is large and only the first megabyte is shown below.<br /> + <a href="${h.url_for( controller='dataset', action='display', dataset_id=data.id, show_all=True )}">Show all</a> | + <a href="${h.url_for( controller='root', action='display', id=data.id, tofile='yes', toext=data.ext )}">Save</a> +</div> + +<pre> +${ truncated_data } +</pre>

1 0

[hg] galaxy 2859: Modified Bowtie wrapper to directly output SAM...
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e02620ad3f41 changeset: 2859:e02620ad3f41 user: Kelly Vincent <kpvincent(a)bx.psu.edu> date: Thu Oct 08 18:07:02 2009 -0400 description: Modified Bowtie wrapper to directly output SAM format, removing header as necessary 5 file(s) affected in this change: test-data/bowtie_out1.sam test-data/bowtie_out2.sam test-data/bowtie_out3.sam tools/sr_mapping/bowtie_wrapper.py tools/sr_mapping/bowtie_wrapper.xml diffs (317 lines): diff -r 50e2d1a49815 -r e02620ad3f41 test-data/bowtie_out1.sam --- a/test-data/bowtie_out1.sam Thu Oct 08 15:53:11 2009 -0400 +++ b/test-data/bowtie_out1.sam Thu Oct 08 18:07:02 2009 -0400 @@ -1,1 +1,1 @@ -HWI-EAS91_1_30788AAXX:1:1:1513:715 16 chrM 9563 25 36M * 0 0 CTGACTACCACAACTAAACATCTATGCNNAAAAAAC I+-II?IDIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1513:715/1 16 chrM 9563 255 36M * 0 0 CTGACTACCACAACTAAACATCTATGCNNAAAAAAC I+-II?IDIIIIIIIIIIIIIIIIIII""IIIIIII XA:i:2 MD:Z:27A0G7 NM:i:2 diff -r 50e2d1a49815 -r e02620ad3f41 test-data/bowtie_out2.sam --- a/test-data/bowtie_out2.sam Thu Oct 08 15:53:11 2009 -0400 +++ b/test-data/bowtie_out2.sam Thu Oct 08 18:07:02 2009 -0400 @@ -1,2 +1,2 @@ -HWI-EAS91_1_30788AAXX:1:2:618:346 0 chrM 441 25 36M * 0 0 TAGACTACGAAAGTGACTTTAATACCTCTGACTACA IIIIIIIIIIIIIIIIIIIIIIIIIIIII%4II;I3 NM:i:0 X0:i:1 MD:Z:36 -HWI-EAS91_1_30788AAXX:1:2:618:346 16 chrM 652 25 36M * 0 0 CGATAAACCCCACCATCCATTGCTAATTCAGCCTAT )2I*IIIIIIFD6II5II.I9IIIIIIIIIIIIIII NM:i:1 X1:i:1 MD:Z:17A18 +HWI-EAS91_1_30788AAXX:1:2:618:346/1 99 chrM 441 255 36M = 652 247 TAGACTACGAAAGTGACTTTAATACCTCTGACTACA IIIIIIIIIIIIIIIIIIIIIIIIIIIII%4II;I3 XA:i:0 MD:Z:36 NM:i:0 +HWI-EAS91_1_30788AAXX:1:2:618:346/2 147 chrM 652 255 36M = 441 -247 CGATAAACCCCACCATCCATTGCTAATTCAGCCTAT )2I*IIIIIIFD6II5II.I9IIIIIIIIIIIIIII XA:i:1 MD:Z:18C17 NM:i:1 diff -r 50e2d1a49815 -r e02620ad3f41 test-data/bowtie_out3.sam --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test-data/bowtie_out3.sam Thu Oct 08 18:07:02 2009 -0400 @@ -0,0 +1,2 @@ +HWI-EAS91_1_30788AAXX:1:2:618:346/1 99 chrM 441 255 36M = 652 247 TAGACTACGAAAGTGACTTTAATACCTCTGACTACA IIIIIIIIIIIIIIIIIIIIIIIIIIIII%4II;I3 XA:i:0 MD:Z:36 NM:i:0 +HWI-EAS91_1_30788AAXX:1:2:618:346/2 147 chrM 652 255 36M = 441 -247 CGATAAACCCCACCATCCATTGCTAATTCAGCCTAT )2I*IIIIIIFD6II5II.I9IIIIIIIIIIIIIII XA:i:0 MD:Z:18C17 NM:i:1 diff -r 50e2d1a49815 -r e02620ad3f41 tools/sr_mapping/bowtie_wrapper.py --- a/tools/sr_mapping/bowtie_wrapper.py Thu Oct 08 15:53:11 2009 -0400 +++ b/tools/sr_mapping/bowtie_wrapper.py Thu Oct 08 18:07:02 2009 -0400 @@ -33,7 +33,6 @@ parser.add_option('', '--valAlign', dest='valAlign', help='Report up to n valid arguments per read') parser.add_option('', '--allValAligns', dest='allValAligns', help='Whether or not to report all valid alignments per read') parser.add_option('', '--suppressAlign', dest='suppressAlign', help='Suppress all alignments for a read if more than n reportable alignments exist') - parser.add_option('', '--offbase', dest='offbase', help='Number the first base of a reference sequence as n when outputting alignments') parser.add_option('', '--best', dest='best', help="Whether or not to make Bowtie guarantee that reported singleton alignments are 'best' in terms of stratum and in terms of the quality values at the mismatched positions") parser.add_option('', '--maxBacktracks', dest='maxBacktracks', help='Maximum number of backtracks permitted when aligning a read') parser.add_option('', '--strata', dest='strata', help='Whether or not to report only those alignments that fall in the best stratum if many valid alignments exist and are reportable') @@ -43,7 +42,6 @@ parser.add_option('', '--maxAlignAttempt', dest='maxAlignAttempt', help='Maximum number of attempts Bowtie will make to match an alignment for one mate with an alignment for the opposite mate') parser.add_option('', '--forwardAlign', dest='forwardAlign', help='Whether or not to attempt to align the forward reference strand') parser.add_option('', '--reverseAlign', dest='reverseAlign', help='Whether or not to attempt to align the reverse-complement reference strand') - parser.add_option('', '--phased', dest='phased', help='Whether or not it should alternate between using the forward and mirror indexes in a series of phases so that only half of the index is resident in memory at one time') parser.add_option('', '--offrate', dest='offrate', help='Override the offrate of the index to n') parser.add_option('', '--seed', dest='seed', help='Seed for pseudo-random number generator') parser.add_option('', '--dbkey', dest='dbkey', help='') @@ -61,8 +59,8 @@ parser.add_option('', '--iendian', dest='iendian', help='Endianness to use when serializing integers to the index file') parser.add_option('', '--iseed', dest='iseed', help='Seed for the pseudorandom number generator') parser.add_option('', '--icutoff', dest='icutoff', help='Number of first bases of the reference sequence to index') - parser.add_option('', '--ioldpmap', dest='ioldpmap', help='Use the scheme for mapping joined reference locations to original reference locations used in versions of Bowtie prior to 0.9.8') parser.add_option('', '--indexSettings', dest='index_settings', help='Whether or not indexing options are to be set') + parser.add_option('', '--suppressHeader', dest='suppressHeader', help='Suppress header') (options, args) = parser.parse_args() # index if necessary @@ -72,7 +70,7 @@ indexing_cmds = '' else: try: - indexing_cmds = '%s %s %s %s %s %s %s --offrate %s %s %s %s %s %s %s' % \ + indexing_cmds = '%s %s %s %s %s %s %s --offrate %s %s %s %s %s %s' % \ (('','--noauto')[options.iauto_b=='set'], ('','--packed')[options.ipacked=='packed'], ('','--bmax %s'%options.ibmax)[options.ibmax!='None' and options.ibmax>=1], @@ -84,8 +82,7 @@ ('','--ntoa')[options.intoa=='yes'], ('--little','--big')[options.iendian=='big'], ('','--seed %s'%options.iseed)[int(options.iseed)>0], - ('','--cutoff %s'%options.icutoff)[int(options.icutoff)>0], - ('','--oldpmap')[options.ioldpmap=='yes']) + ('','--cutoff %s'%options.icutoff)[int(options.icutoff)>0]) except ValueError: indexing_cmds = '' @@ -105,11 +102,11 @@ # set up aligning and generate aligning command options # automatically set threads to 8 in both cases if options.params == 'pre_set': - aligning_cmds = '-p %s' % options.threads + aligning_cmds = '-p %s -S' % options.threads else: try: aligning_cmds = '%s %s %s %s %s %s %s %s %s %s %s %s %s %s ' \ - '%s %s %s %s %s %s %s %s %s %s %s %s -p %s' % \ + '%s %s %s %s %s %s %s %s %s %s -p %s -S' % \ (('','-s %s'%options.skip)[options.skip!='None'], ('','-u %s'%options.alignLimit)[int(options.alignLimit)>0], ('','-5 %s'%options.trimH)[int(options.trimH)>=0], @@ -132,41 +129,47 @@ ('','-m %s'%options.suppressAlign)[int(options.suppressAlign)>=0], ('','--best')[options.best=='doBest'], ('','--strata')[options.strata=='doStrata'], - ('','-B %s'%options.offbase)[int(options.offbase)>=0], - ('','-z %s'%options.phased)[options.phased!='None'], ('','-o %s'%options.offrate)[int(options.offrate)>=0], ('','--seed %s'%options.seed)[int(options.seed)>=0], options.threads) except ValueError: aligning_cmds = '-p %s' % options.threads - tmp_out = tempfile.NamedTemporaryFile() - # prepare actual aligning commands if options.paired == 'paired': - cmd2 = 'bowtie %s %s -1 %s -2 %s > %s 2> /dev/null' % (aligning_cmds, options.ref, options.input1, options.input2, tmp_out.name) + cmd2 = 'bowtie %s %s -1 %s -2 %s > %s 2> /dev/null' % (aligning_cmds, options.ref, options.input1, options.input2, options.output) else: - cmd2 = 'bowtie %s %s %s > %s 2> /dev/null' % (aligning_cmds, options.ref, options.input1, tmp_out.name) - # prepare command to convert bowtie output to sam and alternative - cmd3 = 'bowtie2sam.pl %s > %s' % (tmp_out.name, options.output) - cmd4 = 'cp %s %s' % (tmp_out.name, options.output) + cmd2 = 'bowtie %s %s %s > %s 2> /dev/null' % (aligning_cmds, options.ref, options.input1, options.output) # align try: os.system(cmd2) except Exception, erf: stop_err("Error aligning sequence\n" + str(erf)) - if len(file(tmp_out.name,'r').read()) > 0: - #convert + + # remove header if necessary + if options.suppressHeader == 'true': + tmp_out = tempfile.NamedTemporaryFile() + cmd3 = 'cp %s %s' % (options.output, tmp_out.name) try: os.system(cmd3) except Exception, erf: - stop_err('Error converting output to sam format\n' + str(erf)) - else: - try: - os.system(cmd4) - sys.stdout.write('Alignment file contained no data') - except Exception, erf: - stop_err('Error producing alignment file. File contained no data.\n' + str(erf)) + stop_err("Error copying output file before removing headers\n" + str(erf)) + output = file(tmp_out.name, 'r') + fout = file(options.output, 'w') + header = True + line = output.readline() + while line.strip() != '': + if header: + if line.startswith('@HD') or line.startswith('@SQ') or line.startswith('@RG') or line.startswith('@PG') or line.startswith('@CO'): + pass + else: + header = False + fout.write(line) + else: + fout.write(line) + line = output.readline() + fout.close() + tmp_out.close() if __name__=="__main__": __main__() diff -r 50e2d1a49815 -r e02620ad3f41 tools/sr_mapping/bowtie_wrapper.xml --- a/tools/sr_mapping/bowtie_wrapper.xml Thu Oct 08 15:53:11 2009 -0400 +++ b/tools/sr_mapping/bowtie_wrapper.xml Thu Oct 08 18:07:02 2009 -0400 @@ -32,22 +32,15 @@ --valAlign=$singlePaired.params.valAlign --allValAligns=$singlePaired.params.allValAligns --suppressAlign=$singlePaired.params.suppressAlign - --offbase=$singlePaired.params.offbase --offrate=$singlePaired.params.offrate --seed=$singlePaired.params.seed --best=$singlePaired.params.bestOption.best #if $singlePaired.params.bestOption.best == "doBest": --maxBacktracks=$singlePaired.params.bestOption.maxBacktracks --strata=$singlePaired.params.bestOption.strata - --phased="None" #else: --maxBacktracks="None" --strata="None" - #if $singlePaired.sPaired =="single": - --phased=$singlePaired.params.bestOption.phased - #else: - --phased="None" - #end if #end if #if $singlePaired.sPaired == "single": --minInsert="None" @@ -78,7 +71,6 @@ --valAlign="None" --allValAligns="None" --suppressAlign="None" - --offbase="None" --best="None" --maxBacktracks="None" --strata="None" @@ -88,7 +80,6 @@ --maxAlignAttempt="None" --forwardAlign="None" --reverseAlign="None" - --phased="None" --offrate="None" --seed="None" #end if @@ -123,7 +114,6 @@ --iendian=$refGenomeSource.indexParams.endian --iseed=$refGenomeSource.indexParams.seed --icutoff=$refGenomeSource.indexParams.cutoff - --ioldpmap=$refGenomeSource.indexParams.oldpmap #else: --iauto_b="None" --ipacked="None" @@ -138,8 +128,8 @@ --iendian="None" --iseed="None" --icutoff="None" - --ioldpmap="None" #end if + --suppressHeader=$suppressHeader </command> <inputs> <conditional name="refGenomeSource"> @@ -200,10 +190,6 @@ </param> <param name="seed" type="integer" value="-1" label="Seed for the pseudorandom number generator (--seed)" help="Use -1 to use default" /> <param name="cutoff" type="integer" value="-1" label="Number of first bases of the reference sequence to index (--cutoff)" help="Use -1 to use default" /> - <param name="oldpmap" type="select" label="Use the scheme for mapping joined reference locations to original reference locations used in versions of Bowtie prior to 0.9.8 (--oldpmap)" help="The old scheme uses padding and the new one doesn't"> - <option value="no">Use the new scheme</option> - <option value="yes">Use the old scheme</option> - </param> </when>  </conditional> </when> @@ -244,7 +230,6 @@ <option value="doAllValAligns">Report all valid alignments</option> </param> <param name="suppressAlign" type="integer" value="-1" label="Suppress all alignments for a read if more than n reportable alignments exist (-m)" help="-1 for no limit" /> - <param name="offbase" type="integer" value="0" label="Number the first base of a reference sequence as n when outputting alignments (-B)" /> <conditional name="bestOption"> <param name="best" type="select" label="Whether or not to make Bowtie guarantee that reported singleton alignments are 'best' in terms of stratum and in terms of the quality values at the mismatched positions (--best)" help="Removes all strand bias. Only affects which alignments are reported by Bowtie. Runs slower with best option"> <option value="noBest">Do not use best</option> @@ -252,10 +237,6 @@ </param> <when value="noBest"> <param name="maxBacktracks" type="integer" value="125" label="Maximum number of backtracks permitted when aligning a read (--maxbts)" /> - <param name="phased" type="select" label="Whether or not it should alternate between using the forward and mirror indexes in a series of phases so that only half of the index is resident in memory at one time (-z)"> - <option value="noPhased">Don't alternate</option> - <option value="doPhased">Do alternate</option> - </param> </when> <when value="doBest"> <param name="maxBacktracks" type="integer" value="800" label="Maximum number of backtracks permitted when aligning a read (--maxbts)" /> @@ -318,7 +299,6 @@ <option value="doAllValAligns">Report all valid alignments</option> </param> <param name="suppressAlign" type="integer" value="-1" label="Suppress all alignments for a pair if more than n reportable alignments exist (-m)" help="-1 for no limit" /> - <param name="offbase" type="integer" value="0" label="Number the first base of a reference sequence as n when outputting alignments (-B)" /> <conditional name="bestOption"> <param name="best" type="select" label="Whether or not to make Bowtie guarantee that reported singleton alignments are 'best' in terms of stratum and in terms of the quality values at the mismatched positions (--best)" help="Removes all strand bias. Only affects which alignments are reported by Bowtie. Runs slower with best option"> <option value="noBest">Do not use best</option> @@ -341,6 +321,7 @@ </conditional>  </when>  </conditional>  + <param name="suppressHeader" type="boolean" truevalue="true" falsevalue="false" checked="true" label="Suppress the header in the output SAM file" help="Bowtie produces SAM with several lines of header information" /> </inputs> <outputs> <data format="sam" name="output" /> @@ -352,6 +333,7 @@ <param name="sPaired" value="single" /> <param name="input1" ftype="fastqsanger" value="bowtie_in1.fastq" /> <param name="settings_type" value="pre_set" /> + <param name="suppressHeader" value="true" /> <output name="output" ftype="sam" file="bowtie_out1.sam" /> </test> <test> @@ -362,6 +344,7 @@ <param name="input1" ftype="fastqsanger" value="bowtie_in2.fastq" /> <param name="input2" ftype="fastqsanger" value="bowtie_in3.fastq" /> <param name="settings_type" value="pre_set" /> + <param name="suppressHeader" value="true" /> <output name="output" ftype="sam" file="bowtie_out2.sam" /> </test> <test> @@ -381,10 +364,10 @@ <param name="endian" value="little" /> <param name="seed" value="-1" /> <param name="cutoff" value="-1" /> - <param name="oldpmap" value="no" /> <param name="sPaired" value="single" /> <param name="input1" ftype="fastqsanger" value="bowtie_in1.fastq" /> <param name="settings_type" value="pre_set" /> + <param name="suppressHeader" value="true" /> <output name="output" ftype="sam" file="bowtie_out1.sam" /> </test> <test> @@ -413,13 +396,13 @@ <param name="valAlign" value="1" /> <param name="allValAligns" value="noAllValAligns" /> <param name="suppressAlign" value="-1" /> - <param name="offbase" value="0" /> <param name="best" value="doBest" /> <param name="maxBacktracks" value="800" /> <param name="strata" value="noStrata" /> <param name="offrate" value="-1" /> <param name="seed" value="403" /> - <output name="output" ftype="sam" file="bowtie_out2.sam" /> + <param name="suppressHeader" value="true" /> + <output name="output" ftype="sam" file="bowtie_out3.sam" /> </test> </tests> <help> @@ -514,7 +497,6 @@ --little Endianness. [--little] --seed <int> Random seed. Use <int> as the seed for the pseudo-random number generator. [off] --cutoff <int> Cutoff. Index only the first <int> bases of the reference sequences (cumulative across sequences) and ignore the rest. [off] - --oldpmap Use old mapping scheme. Use the padding-based scheme from Bowtie versions before 0.9.8 instead of the current scheme. [off] For aligning (bowtie):: -s <int> Skip. Do not align the first <int> reads or pairs in the input. [off] @@ -542,10 +524,7 @@ -m <int> Suppress alignments. Suppress all alignments for a particular read or pair if more than <int> reportable alignments exist for it. [no limit] --best Best mode. Make Bowtie guarantee that reported singleton alignments are "best" in terms of stratum (the number of mismatches) and quality values at mismatched position. [off] --strata Best strata. When running in best mode, report alignments that fall into the best stratum if there are ones falling into more than one. [off] - -B <int> First base number. When outputting alignments, number the first base of a reference sequence as <int>. [0] - -z <int> Phased. Alternate between using the forward and mirror indexes in a series of phases such that only one half of the index is resident in memory at one time. Cannot be used with paired-end alignment. [off] -o <int> Offrate override. Override the offrate of the index with <int>. Some row markings are discarded when index read into memory. <int> must be greater than the value used to build the index (default: 5). [off] - --mm I/O for index loading. Choosing this option means that memory-mapped I/O will be used to load the index instead of the normal POSIX/C file I/O. Allows memory-efficient parallelization where using -p is not desirable. [off] --seed <int> Random seed. Use <int> as the seed for the pseudo-random number generator. [off] </help>

1 0

[hg] galaxy 2858: fixed trackster memory leak
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/50e2d1a49815 changeset: 2858:50e2d1a49815 user: Kanwei Li <kanwei(a)gmail.com> date: Thu Oct 08 15:53:11 2009 -0400 description: fixed trackster memory leak 1 file(s) affected in this change: lib/galaxy/visualization/tracks/data/array_tree.py diffs (45 lines): diff -r e1382afd8e41 -r 50e2d1a49815 lib/galaxy/visualization/tracks/data/array_tree.py --- a/lib/galaxy/visualization/tracks/data/array_tree.py Thu Oct 08 14:57:36 2009 -0400 +++ b/lib/galaxy/visualization/tracks/data/array_tree.py Thu Oct 08 15:53:11 2009 -0400 @@ -20,13 +20,15 @@ self.dataset = dataset def get_stats( self, chrom ): - d = FileArrayTreeDict( open( self.dataset.file_name ) ) + f = open( self.dataset.file_name ) + d = FileArrayTreeDict( f ) try: chrom_array_tree = d[chrom] except KeyError: return "no data" root_summary = chrom_array_tree.get_summary( 0, chrom_array_tree.levels ) + f.close() return { 'max': float( max(root_summary.maxs) ), 'min': float( min(root_summary.mins) ) } def get_data( self, chrom, start, end ): @@ -35,7 +37,8 @@ level = int( ceil( log( end - start, BLOCK_SIZE ) ) ) - 1 # Open the file - d = FileArrayTreeDict( open( self.dataset.file_name ) ) + f = open( self.dataset.file_name ) + d = FileArrayTreeDict( f ) # Get the right chromosome try: chrom_array_tree = d[chrom] @@ -51,12 +54,14 @@ # Return either data point or a summary depending on the level if level > 0: s = chrom_array_tree.get_summary( start, level ) + f.close() if s is not None: return zip( indexes, map( float, s.sums / s.counts ) ) else: return None else: v = chrom_array_tree.get_leaf( start ) + f.close() if v is not None: return zip( indexes, map( float, v ) ) else:

1 0

[hg] galaxy 2857: Modified datatypes_conf so fastqsanger type di...
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e1382afd8e41 changeset: 2857:e1382afd8e41 user: Kelly Vincent <kpvincent(a)bx.psu.edu> date: Thu Oct 08 14:57:36 2009 -0400 description: Modified datatypes_conf so fastqsanger type displays in upload list. 1 file(s) affected in this change: datatypes_conf.xml.sample diffs (12 lines): diff -r 4d5b24d03226 -r e1382afd8e41 datatypes_conf.xml.sample --- a/datatypes_conf.xml.sample Thu Oct 08 14:50:28 2009 -0400 +++ b/datatypes_conf.xml.sample Thu Oct 08 14:57:36 2009 -0400 @@ -23,7 +23,7 @@ <converter file="fasta_to_tabular_converter.xml" target_datatype="tabular"/> </datatype> <datatype extension="fastq" type="galaxy.datatypes.sequence:Fastq" display_in_upload="true"/> - <datatype extension="fastqsanger" type="galaxy.datatypes.sequence:FastqSanger"/> + <datatype extension="fastqsanger" type="galaxy.datatypes.sequence:FastqSanger" display_in_upload="true"/> <datatype extension="genetrack" type="galaxy.datatypes.tracks:GeneTrack"/> <datatype extension="gff" type="galaxy.datatypes.interval:Gff" display_in_upload="true"> <converter file="gff_to_bed_converter.xml" target_datatype="bed"/>

1 0

[hg] galaxy 2856: Modified test data and tool conf for fastq stats
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/4d5b24d03226 changeset: 2856:4d5b24d03226 user: Anton Nekrutenko <anton(a)bx.psu.edu> date: Thu Oct 08 14:50:28 2009 -0400 description: Modified test data and tool conf for fastq stats 3 file(s) affected in this change: test-data/fastq_stats1.fastq tools/fastx_toolkit/fastx_quality_statistics.xml tools/metag_tools/split_paired_reads.xml diffs (85 lines): diff -r d5a77c9e869e -r 4d5b24d03226 test-data/fastq_stats1.fastq --- a/test-data/fastq_stats1.fastq Thu Oct 08 14:09:52 2009 -0400 +++ b/test-data/fastq_stats1.fastq Thu Oct 08 14:50:28 2009 -0400 @@ -1,36 +1,36 @@ @CSHL_3_FC042AGLLWW:1:2:7:203 GTACGCATGACCGAACCCCCCNCCCCCCAATTGGTT +CSHL_3_FC042AGLLWW:1:2:7:203 -aab^V^aU]`aa^aZaaabbXEZabaaaaaaaa]]` +BBC?7?B6>ABB?B;BBBCC9&;BCBBBBBBBB>>A @CSHL_3_FC042AGLLWW:1:2:7:33 CAATGCCTCCAATTGGTTAATCCCCCTATATATACT +CSHL_3_FC042AGLLWW:1:2:7:33 -Waaa^aZaaW^U_XaWaa\WMEP^KEZXRPEEEGaa +8BBB?B;BB8?6(a)9B8BB=8.&1?,&;931&&&(BB @CSHL_3_FC042AGLLWW:1:2:7:169 GCAGCAGGCGCGTCAGAGAGCCCCCCCCCCCCCCCC +CSHL_3_FC042AGLLWW:1:2:7:169 -a_M^a\Uaaa_M_aaaaaaaaaaaaaaaV\ZUGUUR +B@.?B=6BBB@.@BBBBBBBBBBBBBBB7=;6(663 @CSHL_3_FC042AGLLWW:1:2:7:1436 AATTATTTATTAAATTTTAATAATATGGGAGACACT +CSHL_3_FC042AGLLWW:1:2:7:1436 -a^aaaaaaaaaaaaaaa_U`aaaaa_S_aaaaaVV[ +B?BBBBBBBBBBBBBBB@6ABBBBB@4@BBBBB77< @CSHL_3_FC042AGLLWW:1:2:7:292 GGAGAAATACACACAATTGGTTAATCCCCCTATATA +CSHL_3_FC042AGLLWW:1:2:7:292 -babaaaaaaaUMaaaaaaaaaaa\XEUUEP_]UERE +CBCBBBBBBB6.BBBBBBBBBBB=9&66&1@>6&3& @CSHL_3_FC042AGLLWW:1:2:7:1819 AATTCAAACCACCCCAACCCACACACAGAGATACAA +CSHL_3_FC042AGLLWW:1:2:7:1819 -a\\QVVVLaaLOEXUWUUEKUULEMUEUUKULIQMU +B==2777-BB-0&96866&,66-&.6&66,6-*2.6 @CSHL_3_FC042AGLLWW:1:2:7:1875 GCAAAAGAGTAGTGTACCCCCCCCCCCCCCCCCCCC +CSHL_3_FC042AGLLWW:1:2:7:1875 -aaaaaaaaaXUXXEXaaaaa`_ZaaaaaaaaaXEXU +BBBBBBBBB9699&9BBBBBA@;BBBBBBBBB9&96 @CSHL_3_FC042AGLLWW:1:2:8:624 ACTGCAATTGGTTAATCCCCCTATATAGCGCTGTGG +CSHL_3_FC042AGLLWW:1:2:8:624 -aa[S^`X`aa_]]OOXMU^_[MU_aaaaaaaaaaaa +BB<4?A9ABB@>>009.6?@<.6@BBBBBBBBBBBB @CSHL_3_FC042AGLLWW:1:2:8:250 TGCCGCGCACACTGATGCAATTGGTTAATCCCCCTA +CSHL_3_FC042AGLLWW:1:2:8:250 -aaaaaaaa^aaaaaabbb[KXPEU[RXZ^JUKRKXE +BBBBBBBB?BBBBBBCCC<,91&6<39;?+6,3,9& diff -r d5a77c9e869e -r 4d5b24d03226 tools/fastx_toolkit/fastx_quality_statistics.xml --- a/tools/fastx_toolkit/fastx_quality_statistics.xml Thu Oct 08 14:09:52 2009 -0400 +++ b/tools/fastx_toolkit/fastx_quality_statistics.xml Thu Oct 08 14:50:28 2009 -0400 @@ -1,16 +1,14 @@ <tool id="cshl_fastx_quality_statistics" name="Compute quality statistics"> <description></description> - <command>zcat -f $input | fastx_quality_stats -o $output -Q $offset</command> + <command>zcat -f $input | fastx_quality_stats -o $output -Q 33</command> <inputs> <param format="fastqsanger" name="input" type="data" label="Library to analyse" /> - <param name="offset" type="hidden" value="33"/> </inputs> <tests> <test> - <param name="input" value="fastq_stats1.fastq" ftype="fastq"/> - <param name="offset" value="64" /> + <param name="input" value="fastq_stats1.fastq" ftype="fastqsanger"/> <output name="output" file="fastq_stats1.out" /> </test> </tests> diff -r d5a77c9e869e -r 4d5b24d03226 tools/metag_tools/split_paired_reads.xml --- a/tools/metag_tools/split_paired_reads.xml Thu Oct 08 14:09:52 2009 -0400 +++ b/tools/metag_tools/split_paired_reads.xml Thu Oct 08 14:50:28 2009 -0400 @@ -7,8 +7,8 @@ <param name="input" type="data" format="fastqsanger" label="Your paired-end file" /> </inputs> <outputs> - <data name="output1" format="input"/> - <data name="output2" format="input"/> + <data name="output1" format="fastqsanger"/> + <data name="output2" format="fastqsanger"/> </outputs> <tests> <test>

1 0

[hg] galaxy 2855: Add missing tag to template for browing library.
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/d5a77c9e869e changeset: 2855:d5a77c9e869e user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Oct 08 14:09:52 2009 -0400 description: Add missing tag to template for browing library. 1 file(s) affected in this change: templates/library/browse_library.mako diffs (27 lines): diff -r b9873946cada -r d5a77c9e869e templates/library/browse_library.mako --- a/templates/library/browse_library.mako Thu Oct 08 14:08:46 2009 -0400 +++ b/templates/library/browse_library.mako Thu Oct 08 14:09:52 2009 -0400 @@ -290,12 +290,8 @@ <ul class="manage-table-actions"> %if can_add: - <li> - <a class="action-button" href="${h.url_for( controller='library', action='upload_library_dataset', library_id=library.id, folder_id=library.root_folder.id )}"><span>Add datasets</span></a> - </li> - <li> - <a class="action-button" href="${h.url_for( controller='library', action='folder', new=True, obj_id=library.root_folder.id, library_id=library.id )}">Add folder</a> - </li> + <li><a class="action-button" href="${h.url_for( controller='library', action='upload_library_dataset', library_id=library.id, folder_id=library.root_folder.id )}"><span>Add datasets</span></a></li> + <li><a class="action-button" href="${h.url_for( controller='library', action='folder', new=True, obj_id=library.root_folder.id, library_id=library.id )}">Add folder</a></li> %endif %if can_modify: <li><a class="action-button" href="${h.url_for( controller='library', action='library', information=True, obj_id=library.id )}">Edit information</a></li> @@ -303,7 +299,7 @@ <li><a class="action-button" href="${h.url_for( controller='library', action='library', information=True, obj_id=library.id )}">View information</a></li> %endif %if can_add and not info_association: - <a class="action-button" href="${h.url_for( controller='library_common', action='info_template', cntrller='library', library_id=library.id, response_action='browse_library' )}">Add template</a> + <li><a class="action-button" href="${h.url_for( controller='library_common', action='info_template', cntrller='library', library_id=library.id, response_action='browse_library' )}">Add template</a></li> %endif %if can_manage: <li><a class="action-button" href="${h.url_for( controller='library', action='library', permissions=True, obj_id=library.id )}">Edit permissions</a></li>

1 0