December 2009 - galaxy-dev - lists.galaxyproject.org

Local Galaxy - very slow
by Assaf Gordon 07 Dec '09

07 Dec '09

Hello, I'm experiencing strange slowness on my local Galaxy server. When re-started, everything works fine. After some time (or some jobs, or some other events which I haven't pinpointed yet), Galaxy itself becomes very slow: When clicking on a tool, it takes 4-7 seconds for galaxy to return the HTML page. Switching Histories, viewing meta-data page - all take very long time. On the reports website, jobs are listed as 'running' even 30 minutes after the local process is terminated, And some jobs marked as 'new' are never started. I'm using a recent version (rev 3137:ddd9d5ac0457 from december 1st), and turned on heartbeat, debug, events, actions logs. Restarting apache and postgresql does not solve the problem. I'm still investigating this issue, but any suggestions or pointers would be greatly appreciated. Thanks, gordon

1 0

funtional_test -list shows weird names
by Assaf Gordon 05 Dec '09

05 Dec '09

Hi, I'm trying to add some test to fastx-toolkit, I pulled a fresh copy, tries "sh run_functional_tests.sh -list" and got the following output: ... ... =========================================================================================================================================== id name section::FASTA_manipulation-fasta_manipulation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - fasta_compute_length Compute sequence length fasta_filter_by_length Filter sequences by length fasta_concatenate0 Concatenate fasta2tab FASTA-to-Tabular tab2fasta Tabular-to-FASTA > FASTA Width _name= RNA/DNA cshl_fastx_collapser Collapse =========================================================================================================================================== id name section::NGS:_QC_and_manipulation-cshl_library_information - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - fastq_gen_conv FASTQ Groomer cshl_fastq_quality_converter Quality format converter cshl_fastx_quality_statistics Compute quality statistics cshl_fastq_quality_boxplot Draw quality score boxplot > Draw nucleotides distribution chart split_paired_reads Split paired end reads quality_score_distribution Build base quality distribution trim_reads Select high quality segments _name= SOLiD-to-FASTQ _name= Compute quality statistics _name= Draw quality score boxplot =========================================================================================================================================== ... ... Some of the tools (e.g. "FASTA width", or even "SOLiD-to-FASTQ" which isn't my tool) shows invalid 'id'. Is this normal? and how do I run test for these tools? thanks, -gordon

2 1

[hg] galaxy 3147: Missed correcting a BAM data file in my last c...
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/81475cecea2c changeset: 3147:81475cecea2c user: Greg Von Kuster <greg(a)bx.psu.edu> date: Fri Dec 04 20:24:09 2009 -0500 description: Missed correcting a BAM data file in my last commit. diffstat: lib/galaxy/datatypes/test/1.bam | 1 files changed, 0 insertions(+), 0 deletions(-) diffs (2 lines): diff -r 495dda4c0693 -r 81475cecea2c lib/galaxy/datatypes/test/1.bam Binary file lib/galaxy/datatypes/test/1.bam has changed

1 0

[hg] galaxy 3146: Fixes for Bam data type's set_meta() and sam_t...
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/495dda4c0693 changeset: 3146:495dda4c0693 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Fri Dec 04 19:51:46 2009 -0500 description: Fixes for Bam data type's set_meta() and sam_to_bam tool, indexes will now be properly created for bam datasets. Fixes also for uploading Bam files, they will no longer be uncompressed on upload. diffstat: lib/galaxy/datatypes/binary.py | 49 ++-- lib/galaxy/tools/util/maf_utilities.py | 11 +- test-data/1.bam | test-data/2.sam | 10 + test-data/3.sam | 10 + test-data/chrM.fa | 335 ------------------------------------- test-data/chr_m.fasta | 335 +++++++++++++++++++++++++++++++++++++ test-data/sam_to_bam_in1.sam | 10 - test-data/sam_to_bam_in2.sam | 10 - test-data/sam_to_bam_out1.bam | test-data/sam_to_bam_out2.bam | tools/data_source/data_source.py | 2 + tools/data_source/upload.py | 57 ++++-- tools/samtools/sam_to_bam.py | 134 +++++++++----- tools/samtools/sam_to_bam.xml | 43 ++-- tools/sr_mapping/bowtie_wrapper.xml | 4 +- tools/sr_mapping/lastz_wrapper.xml | 31 +- 17 files changed, 545 insertions(+), 496 deletions(-) diffs (1323 lines): diff -r 93474ea5c366 -r 495dda4c0693 lib/galaxy/datatypes/binary.py --- a/lib/galaxy/datatypes/binary.py Thu Dec 03 16:16:12 2009 -0500 +++ b/lib/galaxy/datatypes/binary.py Fri Dec 04 19:51:46 2009 -0500 @@ -7,7 +7,7 @@ from galaxy.datatypes import metadata from galaxy.datatypes.sniff import * from urllib import urlencode, quote_plus -import zipfile +import zipfile, gzip import os, subprocess, tempfile log = logging.getLogger(__name__) @@ -54,32 +54,35 @@ def init_meta( self, dataset, copy_from=None ): Binary.init_meta( self, dataset, copy_from=copy_from ) - """ - GVK 12/2/09: just noticed this - not good and doesn't work, so commenting out for now. - def set_meta( self, dataset, overwrite = True, **kwd ): - # Sets index for BAM file. - index_file = dataset.metadata.bam_index - if not index_file: - index_file = dataset.metadata.spec['bam_index'].param.new_file( dataset = dataset ) + def set_meta( self, dataset, overwrite = True, **kwd ): + """ Sets index for BAM file. """ + # These metadata values are not accessible by users, always overwrite + index_file = dataset.metadata.bam_index + if not index_file: + index_file = dataset.metadata.spec['bam_index'].param.new_file( dataset = dataset ) + try: + # Using a symlink from ~/database/files/dataset_XX.dat, create a temporary file + # to store the indexex generated from samtools, something like ~/tmp/dataset_XX.dat.bai tmp_dir = tempfile.gettempdir() - tmpf1 = tempfile.NamedTemporaryFile( dir=tmp_dir ) - tmpf1bai = '%s.bai' % tmpf1.name - try: - os.system( 'cd %s' % tmp_dir ) - os.system( 'cp %s %s' % ( dataset.file_name, tmpf1.name ) ) - os.system( 'samtools index %s' % tmpf1.name ) - os.system( 'cp %s %s' % ( tmpf1bai, index_file.file_name ) ) - except Exception, ex: - sys.stderr.write( 'There was a problem creating the index for the BAM file\n%s\n' + str( ex ) ) - tmpf1.close() - if os.path.exists( tmpf1bai ): - os.remove( tmpf1bai ) - dataset.metadata.bam_index = index_file - """ + tmp_file_path = os.path.join( tmp_dir, os.path.basename( dataset.file_name ) ) + # Here tmp_file_path looks something like /tmp/dataset_XX.dat + os.symlink( dataset.file_name, tmp_file_path ) + command = 'samtools index %s' % tmp_file_path + proc = subprocess.Popen( args=command, shell=True ) + proc.wait() + except: + err_msg = 'Error creating index file (%s) for BAM file (%s)' % ( str( tmp_file_path ), str( dataset.file_name ) ) + log.exception( err_msg ) + sys.stderr.write( err_msg ) + # Move the temporary index file ~/tmp/dataset_XX.dat.bai to be ~/database/files/_metadata_files/dataset_XX.dat + shutil.move( '%s.bai' % ( tmp_file_path ), index_file.file_name ) + os.unlink( tmp_file_path ) + dataset.metadata.bam_index = index_file def sniff( self, filename ): + # BAM is compressed in the BGZF format, and must not be uncompressed in Galaxy. # The first 4 bytes of any bam file is 'BAM\1', and the file is binary. try: - header = open( filename ).read(4) + header = gzip.open( filename ).read(4) if binascii.b2a_hex( header ) == binascii.hexlify( 'BAM\1' ): return True return False diff -r 93474ea5c366 -r 495dda4c0693 lib/galaxy/tools/util/maf_utilities.py --- a/lib/galaxy/tools/util/maf_utilities.py Thu Dec 03 16:16:12 2009 -0500 +++ b/lib/galaxy/tools/util/maf_utilities.py Fri Dec 04 19:51:46 2009 -0500 @@ -191,13 +191,16 @@ def build_maf_index_species_chromosomes( filename, index_species = None ): species = [] species_chromosomes = {} - indexes = bx.interval_index_file.Indexes() + indexes = bx.interval_index_file.Indexes() + blocks = 0 try: maf_reader = bx.align.maf.Reader( open( filename ) ) while True: pos = maf_reader.file.tell() block = maf_reader.next() - if block is None: break + if block is None: + break + blocks += 1 for c in block.components: spec = c.src chrom = None @@ -225,11 +228,11 @@ #most likely a bad MAF log.debug( 'Building MAF index on %s failed: %s' % ( filename, e ) ) return ( None, [], {} ) - return ( indexes, species, species_chromosomes ) + return ( indexes, species, species_chromosomes, blocks ) #builds and returns ( index, index_filename ) for specified maf_file def build_maf_index( maf_file, species = None ): - indexes, found_species, species_chromosomes = build_maf_index_species_chromosomes( maf_file, species ) + indexes, found_species, species_chromosomes, blocks = build_maf_index_species_chromosomes( maf_file, species ) if indexes is not None: fd, index_filename = tempfile.mkstemp() out = os.fdopen( fd, 'w' ) diff -r 93474ea5c366 -r 495dda4c0693 test-data/1.bam Binary file test-data/1.bam has changed diff -r 93474ea5c366 -r 495dda4c0693 test-data/2.sam --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test-data/2.sam Fri Dec 04 19:51:46 2009 -0500 @@ -0,0 +1,10 @@ +HWI-EAS91_1_30788AAXX:1:1:1095:605 0 chrM 23 25 36M * 0 0 AAGCAAGNNACTGAAAATGCCTAGATGAGTATTCTT IIIIIII""IIIIIIIIIIIIIIIEIIIIIIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1650:1185 0 chrM 14956 25 36M * 0 0 ACCCCAGNNAACCCTCTCAGCACTCCCCCTCATATT IIIIIII""IIIIIIIIIIII6IIIIIIIII5I-II NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:799:192 16 chrM 8421 25 36M * 0 0 CCTGTAGCCCTAGCCGTGCGGCTAACCNNTAACATT II%::I<IIIIIEIII8IIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1082:719 16 chrM 7191 25 36M * 0 0 TAAATTAACCCATACCAGCACCATAGANNCTCAAGA <III0EII3+3I29I>III8AIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1746:1180 16 chrM 12013 25 36M * 0 0 CCTAAGCTTCAAACTAGATTACTTCTCNNTAATTTT IIIIIIIIFIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:606:460 0 chrM 4552 25 36M * 0 0 TTAATTTNNATTATAATAACACTCACAATATTCATA IIIIIII""IIIIIIIIIIIIIIIIII?I6IIIII6 NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1059:362 16 chrM 7348 25 36M * 0 0 GGCCACCAATGATACTGAAGCTACGAGNNTACCGAT II/<)2IIIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1483:1161 16 chrM 15080 25 36M * 0 0 TCCTGATCCTAGCACTCATCCCCACCCNNCACATAT HIIIIIFIIAIHIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1273:600 16 chrM 13855 25 36M * 0 0 GTATTAGACACCCATACCTCAGGATACNNCTCAGTA IIIIIIIIIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1190:1283 16 chrM 15338 25 36M * 0 0 TATATCGCACATTACCCTGGTCTTGTANNCCAGAAA EIII?-IIIIIAIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 diff -r 93474ea5c366 -r 495dda4c0693 test-data/3.sam --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test-data/3.sam Fri Dec 04 19:51:46 2009 -0500 @@ -0,0 +1,10 @@ +HWI-EAS91_1_30788AAXX:1:1:1513:715 16 chrM 9563 25 36M * 0 0 CTGACTACCACAACTAAACATCTATGCNNAAAAAAC I+-II?IDIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1698:516 16 chrM 2735 25 36M * 0 0 TTTACACTCAGAGGTTCAACTCCTCTCNNTAACAAC I9IIIII5IIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1491:637 16 chrM 10864 25 36M * 0 0 TGTAGAAGCCCCAATTGCCGGATCCATNNTGCTAGC DBAIIIIIIIIIIIFIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1711:249 16 chrM 10617 25 36M * 0 0 ACCAAACAGAACGCCTGAACGCAGGCCNNTACTTCC IIIIIIIIIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1634:211 0 chrM 9350 25 36M * 0 0 GAAGCAGNNGCTTGATACTGACACTTCGTCGACGTA IIIIIII""IIIIIIIIIIIIIIIIIIIIII9IIDF NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1218:141 16 chrM 14062 25 36M * 0 0 ACAAAACTAACAACAAAAATAACACTCNNAATAAAC I+IIII1IIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1398:854 16 chrM 3921 25 36M * 0 0 CACCCTTCCCGTACTAATAAATCCCCTNNTCTTCAC IIIII=AIIIIIIIIIIIIIIBIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1310:991 16 chrM 10002 25 36M * 0 0 CTCCTATGCCTAGAAGGAATAATACTANNACTATTC I:2IEI:IIDIIIIII4IIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1716:413 0 chrM 6040 25 36M * 0 0 GATCCAANNCTTTATCAACACCTATTCTGATTCTTC IIIIIII""IIIIIIIIIIIIIIIIIIIIIIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 +HWI-EAS91_1_30788AAXX:1:1:1630:59 16 chrM 12387 25 36M * 0 0 TCATACTCGACCCCAACCTTACCAACCNNCCGCTCC FIIHII;IIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 diff -r 93474ea5c366 -r 495dda4c0693 test-data/chrM.fa --- a/test-data/chrM.fa Thu Dec 03 16:16:12 2009 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,335 +0,0 @@ ->chrM -GTTAATGTAGCTTAATAATATAAAGCAAGGCACTGAAAATGCCTAGATGA -GTATTCTTACTCCATAAACACATAGGCTTGGTCCTAGCCTTTTTATTAGT -TATTAATAGAATTACACATGCAAGTATCCGCACCCCAGTGAGAATGCCCT -CTAAATCACGTCTCTACGATTAAAAGGAGCAGGTATCAAGCACACTAGAA -AGTAGCTCATAACACCTTGCTCAGCCACACCCCCACGGGACACAGCAGTG -ATAAAAATTAAGCTATGAACGAAAGTTCGACTAAGTCATATTAAATAAGG -GTTGGTAAATTTCGTGCCAGCCACCGCGGTCATACGATTAACCCAAATTA -ATAAATCTCCGGCGTAAAGCGTGTCAAAGACTAATACCAAAATAAAGTTA -AAACCCAGTTAAGCCGTAAAAAGCTACAACCAAAGTAAAATAGACTACGA -AAGTGACTTTAATACCTCTGACTACACGATAGCTAAGACCCAAACTGGGA -TTAGATACCCCACTATGCTTAGCCCTAAACTAAAATAGCTTACCACAACA -AAGCTATTCGCCAGAGTACTACTAGCAACAGCCTAAAACTCAAAGGACTT -GGCGGTGCTTTACATCCCTCTAGAGGAGCCTGTTCCATAATCGATAAACC -CCGATAAACCCCACCATCCCTTGCTAATTCAGCCTATATACCGCCATCTT -CAGCAAACCCTAAACAAGGTACCGAAGTAAGCACAAATATCCAACATAAA -AACGTTAGGTCAAGGTGTAGCCCATGGGATGGAGAGAAATGGGCTACATT -TTCTACCCTAAGAACAAGAACTTTAACCCGGACGAAAGTCTCCATGAAAC -TGGAGACTAAAGGAGGATTTAGCAGTAAATTAAGAATAGAGAGCTTAATT -GAATCAGGCCATGAAGCGCGCACACACCGCCCGTCACCCTCCTTAAATAT -CACAAATCATAACATAACATAAAACCGTGACCCAAACATATGAAAGGAGA -CAAGTCGTAACAAGGTAAGTATACCGGAAGGTGTACTTGGATAACCAAAG -TGTAGCTTAAACAAAGCATCCAGCTTACACCTAGAAGATTTCACTCAAAA -TGAACACTTTGAACTAAAGCTAGCCCAAACAATACCTAATTCAATTACCC -TTAGTCACTTAACTAAAACATTCACCAAACCATTAAAGTATAGGAGATAG -AAATTTTAACTTGGCGCTATAGAGAAAGTACCGTAAGGGAACGATGAAAG -ATGCATTAAAAGTACTAAACAGCAAAGCTTACCCCTTTTACCTTTTGCAT -AATGATTTAACTAGAATAAACTTAGCAAAGAGAACTTAAGCTAAGCACCC -CGAAACCAGACGAGCTACCTATGAACAGTTACAAATGAACCAACTCATCT -ATGTCGCAAAATAGTGAGAAGATTCGTAGGTAGAGGTGAAAAGCCCAACG -AGCCTGGTGATAGCTGGTTGTCCAGAAACAGAATTTCAGTTCAAATTTAA -ATTTACCTAAAAACTACTCAATTCTAATGTAAATTTAAATTATAGTCTAA -AAAGGTACAGCTTTTTAGATACAGGTTACAACCTTCATTAGAGAGTAAGA -ACAAGATAAACCCATAGTTGGCTTAAAAGCAGCCATCAATTAAGAAAGCG -TTCAAGCTCAACGACACATCTATCTTAATCCCAACAATCAACCCAAACTA -ACTCCTAATCTCATACTGGACTATTCTATCAACACATAGAAGCAATAATG -TTAATATGAGTAACAAGAATTATTTCTCCTTGCATAAGCTTATATCAGAA -CGAATACTCACTGATAGTTAACAACAAGATAGGGATAATCCAAAAACTAA -TCATCTATTTAAACCATTGTTAACCCAACACAGGCATGCATCTATAAGGA -AAGATTAAAAGAAGTAAAAGGAACTCGGCAAACACAAACCCCGCCTGTTT -ACCAAAAACATCACCTCTAGCATTTCCAGTATTAGAGGCACTGCCTGCCC -AGTGACATCTGTTtaaacggccgcggtatcctaaccgtgcaaaggtagca -taatcacttgttccctaaatagggacttgtatgaatggccacacgagggt -tttactgtctcttacttccaatcagtgaaattgaccttcccgtgaagagg -cgggaatgactaaataagacgagaagaccctatggagcttTAATTAACTG -ATTCACAAAAAACAACACACAAACCTTAACCTTCAGGGACAACAAAACTT -TTGATTGAATCAGCAATTTCGGTTGGGGTGACCTCGGAGAACAAAACAAC -CTCCGAGTGATTTAAATCCAGACTAACCAGTCAAAATATATAATCACTTA -TTGATCCAAACCATTGATCAACGGAACAAGTTACCCTAGGGATAACAGCG -CAATCCTATTCCAGAGTCCATATCGACAATTAGGGTTTACGACCTCGATG -TTGGATCAAGACATCCTAATGGTGCAACCGCTATTAAGGGTTCGTTTGTT -CAACGATTAAAGTCTTACGTGATCTGAGTTCAGACCGGAGTAATCCAGGT -CGGTTTCTATCTATTCTATACTTTTCCCAGTACGAAAGGACAAGAAAAGT -AGGGCCCACTTTACAAGAAGCGCCCTCAAACTAATAGATGACATAATCTA -AATCTAACTAATTTATAACTTCTACCGCCCTAGAACAGGGCTCgttaggg -tggcagagcccggaaattgcataaaacttaaacctttacactcagaggtt -caactcctctccctaacaacaTGTTCATAATTAACGTCCTCCTCCTAATT -GTCCCAATCTTGCTCGCCGTAGCATTCCTCACACTAGTTGAACGAAAAGT -CTTAGGCTATATGCAACTTCGCAAAGGACCCAACATCGTAGGCCCCTATG -GCCTACTACAACCTATTGCCGATGCCCTCAAACTATTTATCAAAGAGCCA -CTACAACCACTAACATCATCGACATCCATATTCATCATCGCACCAATCCT -AGCCCTAACCCTGGCCTTAACCATATGAATCCCTCTGCCCATACCATACC -CACTAATCAACATAAACCTAGGAATTCTATTCATACTAGCCATGTCCAGC -CTAGCTGTCTACTCAATCCTTTGATCAGGATGGGCCTCAAACTCAAAATA -CGCCCTAATTGGAGCTCTACGAGCAGTAGCACAAACCATCTCATACGAAG -TAACTCTAGCAATCATCCTACTCTCAGTCCTCCTAATAAGCGGATCATTC -ACATTATCAACACTTATTATTACCCAAGAATACCTCTGATTAATCTTCCC -ATCATGACCCTTAGCCATAATGTGATTCATCTCAACATTAGCCGAAACCA -ACCGAGCTCCATTTGACCTAACAGAAGGAGAATCAGAACTCGTCTCTGGA -TTCAACGTTGAATACGCAGCCGGCCCATTTGCTCTATTCTTCCTAGCAGA -ATACGCAAACATCATCATGATAAACATCTTCACAACAACCCTATTTCTAG -GAGCATTTCACAACCCCTACCTGCCAGAACTCTACTCAATTAATTTCACC -ATTAAAGCTCTCCTTCTAACATGTTCCTTCCTATGAATCCGAGCATCCTA -CCCACGATTCCGATATGACCAACTTATACACCTCCTATGAAAGAACTTCC -TACCACTCACACTAGCCCTCTGCATATGACACGTCTCACTTCCAATCATA -CTATCCAGCATCCCACCACAAACATAGGAAATATGTCTGACAAAAGAGTT -ACTTTGATAGAGTAAAACATAGAGGCTCAAACCCTCTTATTTctagaact -acaggaattgaacctgctcctgagaattcaaaatcctccgtgctaccgaa -ttacaccatgtcctaCAAGTAAGGTCAGCTAAATAAGCTATCGGGCCCAT -ACCCCGAAAATGTTGGATTACACCCTTCCCGTACTAATAAATCCCCTTAT -CTTCACAACTATTCTAATAACAGTTCTTCTAGGAACTATAATCGTTATAA -TAAGCTCACACTGACTAATAATCTGAATCGGATTTGAAATAAATCTACTA -GCCATTATCCCTATCCTAATAAAAAAGTACAATCCCCGAACCATAGAAGC -CTCCACCAAATATTTTCTAACCCAAGCCACCGCATCAATACTCCTCATAA -TAGCGATCATCATTAACCTCATACACTCAGGCCAATGAACAATCACAAAA -GTCTTCAACCCCACAGCGTCCATCATTATAACTTCAGCTCTCGCCATAAA -ACTTGGACTCACACCATTCCACTTCTGAGTACCCGAAGTCACACAGGGCA -TCTCATTAACATCAGGTCTCATCCTACTTACATGACAAAAACTAGCCCCA -ATATCAATCCTATATCAAATCTCACCCTCAATTAACCTAAATATCTTATT -AACTATAGCCGTACTGTCAATCCTAGTAGGAGGCTGAGGCGGTCTCAACC -AAACCCAACTACGAAAAATCATAGCATACTCGTCAATCGCGCATATAGGA -TGAATAACAGCTGTCCTAGTATATAACCCAACACTAACAATACTAAACAT -ATTAATTTACATTATAATAACACTCACAATATTCATACTATTTATCCACA -GCTCCTCTACTACAACACTATCACTCTCCCACACATGAAACAAAATACCT -CTAACCACTACACTAATCTTAATTACCTTACTATCCATAGGAGGCCTCCC -CCCACTATCAGGATTCATACCCAAATGAATAATCATTCAAGAGCTCACCA -AAAATAGCAGCATCATCCTCCCCACACTAATAGCCATTATAGCACTACTC -AACCTCTACTTCTACATACGACTAACCTATTCCACCTCACTGACCATATT -CCCATCCACAAACAACATAAAAATAAAATGACAATTCGAAACCAAACGAA -TTACTCTCTTACCCCCGTTAATTGTTATATCCTCCCTACTCCTCCCCCTA -ACCCCCATACTATCAATTTTGGACTAGGAATTTAGGTTAACATCCCAGAC -CAAGAGCCTTCAAAGCTCTAAGCAAGTGAATCCACTTAATTCCTGCATAC -TAAGGACTGCGAGACTCTATCTCACATCAATTGAACGCAAATCAAACTCT -TTTATTAAGCTAAGCCCTTACTAGATTGGTGGGCTACCATCCCACGAAAT -TTTAGTTAACAGCTAAATACCCTAATCAACTGGCTTCAATCTACTTCTCC -CGCCGCCTAGAAAAAAAGGCGGGAGAAGCCCCGGCAGAAATTGAAGCTGC -TCCTTTGAATTTGCAATTCAATGTGAAAATTCACCACGGGACTTGATAAG -AAGAGGATTCCAACCCCTGTCTTTAGATTTACAGTCTAATGCTTACTCAG -CCATCTTACCTATGTTCATCAACCGCTGACTATTTTCAACTAACCACAAA -GACATCGGCACTCTGTACCTCCTATTCGGCGCTTGAGCTGGAATAGTAGG -AACTGCCCTAAGCCTCCTAATCCGTGCTGAATTAGGCCAACCTGGGACCC -TACTAGGAGATGATCAGATCTACAATGTCATTGTAACCGCCCATGCATTC -GTAATAATTTTCTTTATGGTCATACCCATTATAATCGGAGGATTCGGAAA -CTGATTAGTCCCCCTGATAATTGGAGCACCTGATATAGCTTTCCCCCGAA -TAAACAACATAAGCTTCTGATTACTTCCCCCATCATTCCTACTTCTTCTC -GCTTCCTCAATAATTGAAGCAGGTGCCGGAACAGGCTGAACCGTATATCC -TCCTCTAGCTGGAAATCTGGCGCATGCAGGAGCCTCTGTTGACTTAACCA -TTTTCTCTCTCCACCTAGCTGGGGTGTCCTCGATTTTAGGTGCCATCAAC -TTTATTACCACAATCATTAACATAAAACCACCAGCCCTATCCCAATATCA -AACCCCCCTATTCGTTTGATCTGTCCTTATTACGGCAGTACTCCTTCTCC -TAGCCCTCCCGGTCCTAGCAGCAGGCATTACCATGCTTCTCACAGACCGT -AACCTGAACACTACTTTCTTCGACCCCGCAGGAGGAGGGGATCCAATCCT -TTATCAACACCTATTCTGATTCTTCGGACACCCCGAAGTCTATATTCTTA -TCCTACCAGGCTTCGGTATAATCTCACACATCGTCACATACTACTCAGGT -AAAAAGGAACCTTTTGGCTACATGGGTATAGTGTGAGCTATAATATCCAT -TGGCTTTCTAGGCTTCATCGTATGGGCTCACCACATGTTTACAGTAGGGA -TAGACGTTGACACACGAGCATACTTCACATCAGCTACCATAATCATCGCT -ATCCCTACTGGTGTAAAAGTATTCAGCTGACTAGCCACCCTGCACGGAGG -AAATATCAAATGATCTCCAGCTATACTCTGAGCTCTAGGCTTCATCTTCT -TATTCACAGTAGGAGGTCTAACAGGAATCGTCCTAGCTAACTCATCCCTA -GATATTGTTCTCCACGATACTTATTATGTAGTAGCACATTTCCATTATGT -CCTGTCTATAGGAGCAGTCTTCGCCATTATGGGGGGATTTGTACACTGAT -TCCCTCTATTCTCAGGATACACACTCAACCAAACCTGAGCAAAAATCCAC -TTTACAATTATATTCGTAGGGGTAAATATAACCTTCTTCCCACAACATTT -CCTTGGCCTCTCAGGAATGCCACGACGCTATTCTGATTATCCAGACGCAT -ATACAACATGAAATACCATCTCATCCATAGGATCTTTTATCTCACTTACA -GCAGTGATACTAATAATTTTCATAATTTGAGAAGCGTTCGCATCCAAACG -AGAAGTGTCTACAGTAGAATTAACCTCAACTAATCTGGAATGACTACACG -GATGCCCCCCACCATACCACACATTTGAAGAACCCACCTACGTAAACCTA -AAAtaagaaaggaaggaatcgaaccccctctaactggtttcaagccaata -tcataaccactatgtctttctcCATCAATTGAGGTATTAGTAAAAATTAC -ATGACTTTGTCAAAGTTAAATTATAGGTTAAACCCCTATATACCTCTATG -GCCTACCCCTTCCAACTAGGATTCCAAGACGCAACATCCCCTATTATAGA -AGAACTCCTACACTTCCACGACCACACACTAATAATCGTATTCCTAATTA -GCTCTCTAGTATTATATATTATCTCATCAATACTAACAACTAAATTAACC -CATACCAGCACCATAGATGCTCAAGAAGTAGAGACAATTTGAACGATTTT -ACCAGCCATCATCCTTATTCTAATCGCCCTCCCATCCCTACGAATTCTAT -ATATAATAGATGAAATCAATAATCCGTCCCTCACAGTCAAAACAATAGGC -CACCAATGATACTGAAGCTACGAGTATACCGATTACGAAGACTTGACCTT -TGACTCCTACATGATCCCCACATCAGACCTAAAACCAGGAGAATTACGTC -TTCTAGAAGTCGACAATCGAGTGGTTCTCCCCATAGAAATAACCATCCGA -ATGCTAATTTCATCCGAAGACGTCCTACACTCATGAGCTGTGCCCTCCCT -AGGCCTAAAAACAGACGCTATCCCTGGGCGCCTAAATCAGACAACTCTCG -TGGCCTCTCGACCAGGACTTTACTACGGTCAATGCTCAGAGATCTGCGGA -TCAAACCACAGCTTTATACCAATTGTCCTTGAACTAGTTCCACTGAAACA -CTTCGAAGAATGATCTGCATCAATATTATAAAGTCACTAAGAAGCTATTA -TAGCATTAACCTTTTAAGTTAAAGATTGAGGGTTCAACCCCCTCCCTAGT -GATATGCCACAGTTGGATACATCAACATGATTTATTAATATCGTCTCAAT -AATCCTAACTCTATTTATTGTATTTCAACTAAAAATCTCAAAGCACTCCT -ATCCGACACACCCAGAAGTAAAGACAACCAAAATAACAAAACACTCTGCC -CCTTGAGAATCAAAATGAACGAAAATCTATTCGCCTCTTTCGCTACCCCA -ACAATAGTAGGCCTCCCTATTGTAATTCTGATCATCATATTTCCCAGCAT -CCTATTCCCCTCACCCAACCGACTAATCAACAATCGCCTAATCTCAATTC -AACAATGGCTAGTCCAACTTACATCAAAACAAATAATAGCTATCCATAAC -AGCAAAGGACAAACCTGAACTCTTATACTCATATCACTGATCCTATTCAT -TGGCTCAACAAACTTATTAGGCCTACTACCTCACTCATTTACACCAACAA -CACAACTATCAATAAACCTAGGCATAGCTATTCCCCTATGGGCAGGGACA -GTATTCATAGGCTTTCGTCACAAAACAAAAGCAGCCCTAGCCCACTTTCT -ACCTCAAGGGACGCCCATTTTCCTCATCCCCATACTAGTAATTATCGAGA -CTATCAGCCTATTTATTCAACCTGTAGCCCTAGCCGTGCGGCTAACCGCT -AACATTACCGCCGGACACCTCCTAATACACCTCATCGGAGGGGCAACACT -AGCCCTCATAAGCATCAGCCCCTCAACAGCCCTTATTACGTTTATCATCC -TAATTCTACTAACTATCCTCGAATTCGCAGTAGCTATAATCCAAGCCTAC -GTATTCACTCTCCTGGTAAGCCTTTACTTACACGACAACACCTAATGACC -CACCAAACCCACGCTTACCACATAGTAAACCCCAGCCCATGACCACTTAC -AGGAGCCCTATCAGCCCTCCTGATAACATCAGGACTAGCCATGTGATTTC -ACTTTAACTCAACCTTACTTCTAGCTATAGGGCTATTAACTAACATCCTT -ACCATATATCAATGATGACGAGACATCATCCGAGAAAGCACATTCCAAGG -CCATCACACATCAATCGTTCAAAAGGGACTCCGATATGGCATAATCCTTT -TTATTATCTCAGAAGTCTTCTTCTTCTCTGGCTTCTTCTGAGCCTTTTAC -CACTCAAGCCTAGCCCCCACACCCGAACTAGGCGGCTGCTGACCACCCAC -AGGTATCCACCCCTTAAACCCCCTAGAAGTCCCCTTACTCAACACCTCAG -TGCTCCTAGCATCTGGAGTCTCTATCACCTGAGCCCACCATAGCCTAATA -GAAGGAAACCGTAAAAATATGCTCCAAGGCCTATTCATCACAATTTCACT -AGGCGTATACTTCACCCTTCTCCAAGCCTCAGAATACTATGAAGCCTCAT -TTACTATTTCAGATGGAGTATACGGATCAACATTTTTCGTAGCAACAGGG -TTCCACGGACTACACGTAATTATCGGATCTACCTTCCTCATTGTATGTTT -CCTACGCCAACTAAAATTCCACTTTACATCCAGCCACCACTTCGGATTCG -AAGCAGCCGCTTGATACTGACACTTCGTCGACGTAGTCTGACTATTCTTG -TACGTCTCTATTTATTGATGAGGATCCTATTCTTTTAGTATTGACCAGTA -CAATTGACTTCCAATCAATCAGCTTCGGTATAACCCGAAAAAGAATAATA -AACCTCATACTGACACTCCTCACTAACACATTACTAGCCTCGCTACTCGT -ACTCATCGCATTCTGACTACCACAACTAAACATCTATGCAGAAAAAACCA -GCCCATATGAATGCGGATTTGACCCTATAGGGTCAGCACGCCTCCCCTTC -TCAATAAAATTTTTCTTAGTGGCCATTACATTTCTGCTATTCGACTTAGA -AATTGCCCTCCTATTACCCCTTCCATGAGCATCCCAAACAACTAACCTAA -ACACTATACTTATCATAGCACTAGTCCTAATCTCTCTTCTAGCCATCAGC -CTAGCCTACGAATGAACCCAAAAAGGACTAGAATGAACTGAGTATGGTAA -TTAGTTTAAACCAAAACAAATGATTTCGACTCATTAAACTATGATTAACT -TCATAATTACCAACATGTCACTAGTCCATATTAATATCTTCCTAGCATTC -ACAGTATCCCTCGTAGGCCTACTAATGTACCGATCCCACCTAATATCCTC -ACTCCTATGCCTAGAAGGAATAATACTATCACTATTCGTCATAGCAACCA -TAATAGTCCTAAACACCCACTTCACACTAGCTAGTATAATACCTATCATC -TTACTAGTATTTGCTGCCTGCGAACGAGCTCTAGGATTATCCCTACTAGT -CATAGTCTCCAATACTTATGGAGTAGACCACGTACAAAACCTTAACCTCC -TCCAATGCTAAAAATTATCATTCCCACAATCATACTTATGCCCCTTACAT -GACTATCAAAAAAGAATATAATCTGAATCAACACTACAACCTATAGTCTA -TTAATCAGCCTTATCAGCCTATCCCTCCTAAACCAACCTAGCAACAATAG -CCTAAACTTCTCACTAATATTCTTCTCCGATCCCCTATCAGCCCCACTTC -TGGTGTTGACAACATGACTACTGCCACTAATACTCATAGCCAGCCAACAC -CATCTATCTAAGGAACCACTAATCCGAAAAAAACTCTACATCACCATGCT -AACCATACTTCAAACTTTCCTAATCATGACTTTTACCGCCACAGAACTAA -TCTCCTTCTACATCCTATTTGAAGCCACATTAGTTCCAACACTAATTATC -ATCACCCGCTGAGGCAACCAAACAGAACGCCTGAACGCAGGCCTCTACTT -CCTATTCTACACACTAATAGGTTCCCTCCCACTCTTAGTTGCACTAATCT -CTATCCAAAACCTAACAGGCTCACTAAACTTCCTATTAATTCAATACTGA -AACCAAGCACTACCCGACTCTTGATCCAATATTTTCCTATGACTAGCATG -TATAATAGCATTCATAGTCAAAATACCGGTATATGGTCTTCACCTCTGAC -TCCCAAAAGCCCATGTAGAAGCCCCAATTGCCGGATCCATAGTGCTAGCA -GCCATTCTACTAAAACTAGGAGGCTACGGAATACTACGAATTACAACAAT -ACTAAACCCCCAAACTAGCTTTATAGCCTACCCCTTCCTCATACTATCCC -TGTGAGGAATAATCATAACTAGTTCCATCTGCTTGCGACAAACCGATCTA -AAATCACTTATTGCATACTCCTCTGTCAGCCACATAGCCCTAGTAATCGT -AGCCGTCCTCATCCAAACACCATGAAGTTATATAGGAGCTACAGCCCTAA -TAATCGCTCACGGCCTTACATCATCAATACTATTCTGCCTGGCAAACTCA -AATTACGAACGTACCCATAGCCGAACTATAATCCTAGCCCGCGGGCTTCA -AACACTTCTTCCCCTTATAGCAGCCTGATGACTATTAGCCAGCCTAACCA -ACCTGGCCCTCCCTCCCAGCATTAACCTAATTGGAGAGCTATTCGTAGTA -ATATCATCATTCTCATGATCAAATATTACCATTATCCTAATAGGAGCCAA -TATCACCATCACCGCCCTCTACTCCCTATACATACTAATCACAACACAAC -GAGGGAAATACACACACCATATCAACAGCATTAAACCTTCATTTACACGA -GAAAACGCACTCATGGCCCTCCACATGACTCCCCTACTACTCCTATCACT -TAACCCTAAAATTATCCTAGGCTTTACGTACTGTAAATATAGTTTAACAA -AAACACTAGATTGTGGATCTAGAAACAGAAACTTAATATTTCTTATTTAC -CGAGAAAGTATGCAAGAACTGCTAATTCATGCCCCCATGTCCAACAAACA -TGGCTCTCTCAAACTTTTAAAGGATAGGAGCTATCCGTTGGTCTTAGGAA -CCAAAAAATTGGTGCAACTCCAAATAAAAGTAATCAACATGTTCTCCTCC -CTCATACTAGTTTCACTATTAGTACTAACCCTCCCAATCATATTATCAAT -CTTCAATACCTACAAAAACAGCACGTTCCCGCATCATGTAAAAAACACTA -TCTCATATGCCTTCATTACTAGCCTAATTCCCACTATAATATTTATTCAC -TCTGGACAAGAAACAATTATCTCAAACTGACACTGAATAACCATACAAAC -CCTCAAACTATCCCTAAGCTTCAAACTAGATTACTTCTCAATAATTTTCG -TACCAGTAGCCCTATTCGTAACATGATCTATTATGGAATTCTCCCTATGA -TACATGCACTCAGATCCTTACATTACTCGATTTTTTAAATACTTACTTAC -ATTCCTCATCACTATAATAATTCTAGTCACAGCTAACAACCTTTTCCAAC -TGTTCATCGGATGGGAGGGAGTAGGCATCATGTCATTCTTACTAATCGGA -TGATGATACGGCCGAACAGATGCCAACACCGCGGCCCTTCAAGCAATCCT -TTATAACCGCATCGGGGATATCGGCTTCATCATGGCCATAGCCTGATTCC -TATTCAACACCAACACATGAGACCTCCAACAAATCTTCATACTCGACCCC -AACCTTACCAACCTCCCGCTCCTAGGCCTCCTCCTAGCCGCAACTGGCAA -ATCCGCTCAATTTGGACTCCACCCATGACTTCCTTCAGCCATAGAGGGCC -CTACACCAGTCTCAGCCCTACTCCACTCCAGCACAATAGTTGTAGCAGGC -GTCTTCCTGCTAATCCGCTTCCATCCACTAATAGAAAACAACAAAACAAT -CCAGTCACTTACCCTATGCCTAGGAGCCATCACCACACTATTCACAGCAA -TCTGCGCACTCACTCAAAACGATATCAAAAAAATCATTGCTTTCTCCACC -TCCAGCCAACTAGGCCTGATAATCGTAACCATCGGTATCAATCAACCCTA -CCTAGCATTCCTCCACATTTGCACTCACGCATTCTTCAAAGCTATACTAT -TTATATGTTCCGGATCCATTATCCACAGCCTAAATGACGAGCAAGATATC -CGAAAAATAGGCGGACTATTTAATGCAATACCCTTCACCACCACATCTCT -AATTATTGGCAGCCTTGCACTCACCGGAATTCCTTTCCTCACAGGCTTCT -ACTCCAAAGACCTCATCATCGAAACCGCCAACACATCGTACACCAACGCC -TGAGCCCTACTAATAACTCTCATTGCCACATCCCTCACAGCTGTCTACAG -TACCCGAATCATCTTCTTTGCACTCCTAGGGCAACCCCGCTTCCTCCCTC -TGACCTCAATCAACGAAAATAACCCCTTTCTAATTAACTCCATCAAACGC -CTCTTAATTGGCAGCATTTTTGCCGGATTCTTCATCTCCAACAATATCTA -CCCCACAACCGTCCCAGAAATAACCATACCTACTTACATAAAACTCACCG -CCCTCGCAGTAACCATCCTAGGATTTACACTAGCCCTAGAACTAAGCTTG -ATAACCCATAACTTAAAACTAGAACACTCCACCAACGTATTCAAATTCTC -CAACCTCCTAGGATACTACCCAACAATTATACACCGACTCCCACCGCTCG -CTAACCTATCAATAAGCCAAAAATCAGCATCACTTCTACTAGACTCAATC -TGACTAGAAAACATCCTGCCAAAATCTATCTCCCAGTTCCAAATAAAAAC -CTCGATCCTAATTTCCACCCAAAAAGGACAAATCAAATTATATTTCCTCT -CATTCCTCATCACCCTTACCCTAAGCATACTACTTTTTAATCTCCACGAG -TAACCTCTAAAATTACCAAGACCCCAACAAGCAACGATCAACCAGTCACA -ATCACAACCCAAGCCCCATAACTATACAATGCAGCAGCCCCTATAATTTC -CTCACTAAACGCCCCAGAATCTCCAGTATCATAAATAGCTCAAGCCCCCA -CACCACTAAACTTAAACACTACCCCCACTTCCTCACTCTTCAGAACATAT -AAAACCAACATAACCTCCATCAACAACCCTAAAAGAAATACCCCCATAAC -AGTCGTATTAGACACCCATACCTCAGGATACTGCTCAGTAGCCATAGCCG -TTGTATAACCAAAAACAACCAACATTCCTCCCAAATAAATCAAAAACACC -ATCAACCCCAAAAAGGACCCTCCAAAATTCATAATAATACCACAACCTAC -CCCTCCACTTACAATCAGCACTAAACCCCCATAAATAGGTGAAGGTTTTG -AAGAAAACCCCACAAAACTAACAACAAAAATAACACTCAAAATAAACACA -ATATATGTCATCATTATTCCCACGTGGAATCTAACCACGACCAATGACAT -GAAAAATCATCGTTGTATTTCAACTATAAGAACACCAATGACAAACATCC -GGAAATCTCACCCACTAATTAAAATCATCAATCACTCTTTTATTGACCTA -CCAGCCCCCTCAAACATTTCATCATGATGAAACTTCGGCTCCCTCCTAGG -AATCTGCCTAATCCTCCAAATCTTAACAGGCCTATTCCTAGCCATACACT -ACACATCAGACACGACAACTGCCTTCTCATCCGTCACTCACATCTGCCGA -GACGTTAACTACGGATGAATTATTCGCTACCTCCATGCCAACGGAGCATC -AATATTTTTTATCTGCCTCTTCATTCACGTAGGACGCGGCCTCTACTACG -GCTCTTACACATTCCTAGAGACATGAAACATTGGAATCATCCTACTTTTC -ACAGTTATAGCTACAGCATTCATGGGCTATGTCCTACCATGAGGCCAAAT -ATCCTTTTGAGGAGCAACAGTCATCACGAACCTCCTATCAGCAATTCCCT -ACATCGGTACTACCCTCGTCGAGTGAATCTGAGGTGGATTCTCAGTAGAC -AAAGCCACCCTTACCCGATTTTTTGCTTTCCACTTCATCCTACCCTTCAT -CATCACAGCCCTGGTAGTCGTACATTTACTATTTCTTCACGAAACAGGAT -CTAATAACCCCTCAGGAATCCCATCCGATATGGACAAAATCCCATTCCAC -CCATATTATACAATTAAAGACATCCTAGGACTCCTCCTCCTGATCTTGCT -CCTACTAACTCTAGTATTATTCTCCCCCGACCTCCTAGGAGACCCAGACA -ACTACACCCCAGCTAACCCTCTCAGCACTCCCCCTCATATTAAACCAGAA -TGGTACTTCCTGTTTGCCTACGCCATCCTACGCTCCATTCCCAACAAACT -AGGCGGCGTATTAGCCCTAATCCTCTCCATCCTGATCCTAGCACTCATCC -CCACCCTCCACATATCAAAACAACGAAGCATAATATTCCGGCCTCTCAGC -CAATGCGTATTCTGACTCTTAGTGGCAGACTTACTGACACTAACATGAAT -CGGCGGACAGCCAGTGGAACACCCATACGTAATTATCGGCCAACTGGCCT -CAATCCTCTACTTCTCCCTAATTCTCATTTTTATACCACTCGCAAGCACC -ATCGAAAACAATCTTCTAAAATGAAGAGTCCCTGTAGTATATCGCACATT -ACCCTGGTCTTGTAAACCAGAAAAGGGGGAAAACGTTTCCTCCCAAGGAC -TATCAAGGAAGAAGCTCTAGCTCCACCATCAACACCCAAAGCTGAAATTC -TACTTAAACTATTCCTTGATTTCTTCCCCTAAACGACAACAATTTACCCT -CATGTGCTATGTCAGTATCAGATTATACCCCCACATAACACCATACCCAC -CTGACATGCAATATCTTATGAATGGCCTATGTACGTCGTGCATTAAATTG -TCTGCCCCATGAATAATAAGCATGTACATAATATCATTTATCTTACATAA -GTACATTATATTATTGATCGTGCATACCCCATCCAAGTCAAATCATTTCC -AGTCAACACGCATATCACAGCCCATGTTCCACGAGCTTAATCACCAAGCC -GCGGGAAATCAGCAACCCTCCCAACTACGTGTCCCAATCCTCGCTCCGGG -CCCATCCAAACGTGGGGGTTTCTACAATGAAACTATACCTGGCATCTGGT -TCTTTCTTCAGGGCCATTCCCACCCAACCTCGCCCATTCTTTCCCCTTAA -ATAAGACATCTCGATGGACTAATGACTAATCAGCCCATGCTCACACATAA -CTGTGATTTCATGCATTTGGTATCTTTTTATATTTGGGGATGCTATGACT -CAGCTATGGCCGTCAAAGGCCTCGACGCAGTCAATTAAATTGAAGCTGGA -CTTAAATTGAACGTTATTCCTCCGCATCAGCAACCATAAGGTGTTATTCA -GTCCATGGTAGCGGGACATAGGAAACAAgtgcacctgtgcacctgtgcac -ctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacct -gtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgt -gcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgc -acctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcac -ctgtgcacctACCCGCGCAGTAAGCAAGTAATATAGCTTTCTTAATCAAA -CCCCCCCTACCCCCCATTAAACTCCACATATGTACATTCAACACAATCTT -GCCAAACCCCAAAAACAAGACTAAACAATGCACAATACTTCATGAAGCTT -AACCCTCGCATGCCAACCATAATAACTCAACACACCTAACAATCTTAACA -GAACTTTCCCCCCGCCATTAATACCAACATGCTACTTTAATCAATAAAAT -TTCCATAGACAGGCATCCCCCTAGATCTAATTTTCTAAATCTGTCAACCC -TTCTTCCCCC diff -r 93474ea5c366 -r 495dda4c0693 test-data/chr_m.fasta --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/test-data/chr_m.fasta Fri Dec 04 19:51:46 2009 -0500 @@ -0,0 +1,335 @@ +>chrM +GTTAATGTAGCTTAATAATATAAAGCAAGGCACTGAAAATGCCTAGATGA +GTATTCTTACTCCATAAACACATAGGCTTGGTCCTAGCCTTTTTATTAGT +TATTAATAGAATTACACATGCAAGTATCCGCACCCCAGTGAGAATGCCCT +CTAAATCACGTCTCTACGATTAAAAGGAGCAGGTATCAAGCACACTAGAA +AGTAGCTCATAACACCTTGCTCAGCCACACCCCCACGGGACACAGCAGTG +ATAAAAATTAAGCTATGAACGAAAGTTCGACTAAGTCATATTAAATAAGG +GTTGGTAAATTTCGTGCCAGCCACCGCGGTCATACGATTAACCCAAATTA +ATAAATCTCCGGCGTAAAGCGTGTCAAAGACTAATACCAAAATAAAGTTA +AAACCCAGTTAAGCCGTAAAAAGCTACAACCAAAGTAAAATAGACTACGA +AAGTGACTTTAATACCTCTGACTACACGATAGCTAAGACCCAAACTGGGA +TTAGATACCCCACTATGCTTAGCCCTAAACTAAAATAGCTTACCACAACA +AAGCTATTCGCCAGAGTACTACTAGCAACAGCCTAAAACTCAAAGGACTT +GGCGGTGCTTTACATCCCTCTAGAGGAGCCTGTTCCATAATCGATAAACC +CCGATAAACCCCACCATCCCTTGCTAATTCAGCCTATATACCGCCATCTT +CAGCAAACCCTAAACAAGGTACCGAAGTAAGCACAAATATCCAACATAAA +AACGTTAGGTCAAGGTGTAGCCCATGGGATGGAGAGAAATGGGCTACATT +TTCTACCCTAAGAACAAGAACTTTAACCCGGACGAAAGTCTCCATGAAAC +TGGAGACTAAAGGAGGATTTAGCAGTAAATTAAGAATAGAGAGCTTAATT +GAATCAGGCCATGAAGCGCGCACACACCGCCCGTCACCCTCCTTAAATAT +CACAAATCATAACATAACATAAAACCGTGACCCAAACATATGAAAGGAGA +CAAGTCGTAACAAGGTAAGTATACCGGAAGGTGTACTTGGATAACCAAAG +TGTAGCTTAAACAAAGCATCCAGCTTACACCTAGAAGATTTCACTCAAAA +TGAACACTTTGAACTAAAGCTAGCCCAAACAATACCTAATTCAATTACCC +TTAGTCACTTAACTAAAACATTCACCAAACCATTAAAGTATAGGAGATAG +AAATTTTAACTTGGCGCTATAGAGAAAGTACCGTAAGGGAACGATGAAAG +ATGCATTAAAAGTACTAAACAGCAAAGCTTACCCCTTTTACCTTTTGCAT +AATGATTTAACTAGAATAAACTTAGCAAAGAGAACTTAAGCTAAGCACCC +CGAAACCAGACGAGCTACCTATGAACAGTTACAAATGAACCAACTCATCT +ATGTCGCAAAATAGTGAGAAGATTCGTAGGTAGAGGTGAAAAGCCCAACG +AGCCTGGTGATAGCTGGTTGTCCAGAAACAGAATTTCAGTTCAAATTTAA +ATTTACCTAAAAACTACTCAATTCTAATGTAAATTTAAATTATAGTCTAA +AAAGGTACAGCTTTTTAGATACAGGTTACAACCTTCATTAGAGAGTAAGA +ACAAGATAAACCCATAGTTGGCTTAAAAGCAGCCATCAATTAAGAAAGCG +TTCAAGCTCAACGACACATCTATCTTAATCCCAACAATCAACCCAAACTA +ACTCCTAATCTCATACTGGACTATTCTATCAACACATAGAAGCAATAATG +TTAATATGAGTAACAAGAATTATTTCTCCTTGCATAAGCTTATATCAGAA +CGAATACTCACTGATAGTTAACAACAAGATAGGGATAATCCAAAAACTAA +TCATCTATTTAAACCATTGTTAACCCAACACAGGCATGCATCTATAAGGA +AAGATTAAAAGAAGTAAAAGGAACTCGGCAAACACAAACCCCGCCTGTTT +ACCAAAAACATCACCTCTAGCATTTCCAGTATTAGAGGCACTGCCTGCCC +AGTGACATCTGTTtaaacggccgcggtatcctaaccgtgcaaaggtagca +taatcacttgttccctaaatagggacttgtatgaatggccacacgagggt +tttactgtctcttacttccaatcagtgaaattgaccttcccgtgaagagg +cgggaatgactaaataagacgagaagaccctatggagcttTAATTAACTG +ATTCACAAAAAACAACACACAAACCTTAACCTTCAGGGACAACAAAACTT +TTGATTGAATCAGCAATTTCGGTTGGGGTGACCTCGGAGAACAAAACAAC +CTCCGAGTGATTTAAATCCAGACTAACCAGTCAAAATATATAATCACTTA +TTGATCCAAACCATTGATCAACGGAACAAGTTACCCTAGGGATAACAGCG +CAATCCTATTCCAGAGTCCATATCGACAATTAGGGTTTACGACCTCGATG +TTGGATCAAGACATCCTAATGGTGCAACCGCTATTAAGGGTTCGTTTGTT +CAACGATTAAAGTCTTACGTGATCTGAGTTCAGACCGGAGTAATCCAGGT +CGGTTTCTATCTATTCTATACTTTTCCCAGTACGAAAGGACAAGAAAAGT +AGGGCCCACTTTACAAGAAGCGCCCTCAAACTAATAGATGACATAATCTA +AATCTAACTAATTTATAACTTCTACCGCCCTAGAACAGGGCTCgttaggg +tggcagagcccggaaattgcataaaacttaaacctttacactcagaggtt +caactcctctccctaacaacaTGTTCATAATTAACGTCCTCCTCCTAATT +GTCCCAATCTTGCTCGCCGTAGCATTCCTCACACTAGTTGAACGAAAAGT +CTTAGGCTATATGCAACTTCGCAAAGGACCCAACATCGTAGGCCCCTATG +GCCTACTACAACCTATTGCCGATGCCCTCAAACTATTTATCAAAGAGCCA +CTACAACCACTAACATCATCGACATCCATATTCATCATCGCACCAATCCT +AGCCCTAACCCTGGCCTTAACCATATGAATCCCTCTGCCCATACCATACC +CACTAATCAACATAAACCTAGGAATTCTATTCATACTAGCCATGTCCAGC +CTAGCTGTCTACTCAATCCTTTGATCAGGATGGGCCTCAAACTCAAAATA +CGCCCTAATTGGAGCTCTACGAGCAGTAGCACAAACCATCTCATACGAAG +TAACTCTAGCAATCATCCTACTCTCAGTCCTCCTAATAAGCGGATCATTC +ACATTATCAACACTTATTATTACCCAAGAATACCTCTGATTAATCTTCCC +ATCATGACCCTTAGCCATAATGTGATTCATCTCAACATTAGCCGAAACCA +ACCGAGCTCCATTTGACCTAACAGAAGGAGAATCAGAACTCGTCTCTGGA +TTCAACGTTGAATACGCAGCCGGCCCATTTGCTCTATTCTTCCTAGCAGA +ATACGCAAACATCATCATGATAAACATCTTCACAACAACCCTATTTCTAG +GAGCATTTCACAACCCCTACCTGCCAGAACTCTACTCAATTAATTTCACC +ATTAAAGCTCTCCTTCTAACATGTTCCTTCCTATGAATCCGAGCATCCTA +CCCACGATTCCGATATGACCAACTTATACACCTCCTATGAAAGAACTTCC +TACCACTCACACTAGCCCTCTGCATATGACACGTCTCACTTCCAATCATA +CTATCCAGCATCCCACCACAAACATAGGAAATATGTCTGACAAAAGAGTT +ACTTTGATAGAGTAAAACATAGAGGCTCAAACCCTCTTATTTctagaact +acaggaattgaacctgctcctgagaattcaaaatcctccgtgctaccgaa +ttacaccatgtcctaCAAGTAAGGTCAGCTAAATAAGCTATCGGGCCCAT +ACCCCGAAAATGTTGGATTACACCCTTCCCGTACTAATAAATCCCCTTAT +CTTCACAACTATTCTAATAACAGTTCTTCTAGGAACTATAATCGTTATAA +TAAGCTCACACTGACTAATAATCTGAATCGGATTTGAAATAAATCTACTA +GCCATTATCCCTATCCTAATAAAAAAGTACAATCCCCGAACCATAGAAGC +CTCCACCAAATATTTTCTAACCCAAGCCACCGCATCAATACTCCTCATAA +TAGCGATCATCATTAACCTCATACACTCAGGCCAATGAACAATCACAAAA +GTCTTCAACCCCACAGCGTCCATCATTATAACTTCAGCTCTCGCCATAAA +ACTTGGACTCACACCATTCCACTTCTGAGTACCCGAAGTCACACAGGGCA +TCTCATTAACATCAGGTCTCATCCTACTTACATGACAAAAACTAGCCCCA +ATATCAATCCTATATCAAATCTCACCCTCAATTAACCTAAATATCTTATT +AACTATAGCCGTACTGTCAATCCTAGTAGGAGGCTGAGGCGGTCTCAACC +AAACCCAACTACGAAAAATCATAGCATACTCGTCAATCGCGCATATAGGA +TGAATAACAGCTGTCCTAGTATATAACCCAACACTAACAATACTAAACAT +ATTAATTTACATTATAATAACACTCACAATATTCATACTATTTATCCACA +GCTCCTCTACTACAACACTATCACTCTCCCACACATGAAACAAAATACCT +CTAACCACTACACTAATCTTAATTACCTTACTATCCATAGGAGGCCTCCC +CCCACTATCAGGATTCATACCCAAATGAATAATCATTCAAGAGCTCACCA +AAAATAGCAGCATCATCCTCCCCACACTAATAGCCATTATAGCACTACTC +AACCTCTACTTCTACATACGACTAACCTATTCCACCTCACTGACCATATT +CCCATCCACAAACAACATAAAAATAAAATGACAATTCGAAACCAAACGAA +TTACTCTCTTACCCCCGTTAATTGTTATATCCTCCCTACTCCTCCCCCTA +ACCCCCATACTATCAATTTTGGACTAGGAATTTAGGTTAACATCCCAGAC +CAAGAGCCTTCAAAGCTCTAAGCAAGTGAATCCACTTAATTCCTGCATAC +TAAGGACTGCGAGACTCTATCTCACATCAATTGAACGCAAATCAAACTCT +TTTATTAAGCTAAGCCCTTACTAGATTGGTGGGCTACCATCCCACGAAAT +TTTAGTTAACAGCTAAATACCCTAATCAACTGGCTTCAATCTACTTCTCC +CGCCGCCTAGAAAAAAAGGCGGGAGAAGCCCCGGCAGAAATTGAAGCTGC +TCCTTTGAATTTGCAATTCAATGTGAAAATTCACCACGGGACTTGATAAG +AAGAGGATTCCAACCCCTGTCTTTAGATTTACAGTCTAATGCTTACTCAG +CCATCTTACCTATGTTCATCAACCGCTGACTATTTTCAACTAACCACAAA +GACATCGGCACTCTGTACCTCCTATTCGGCGCTTGAGCTGGAATAGTAGG +AACTGCCCTAAGCCTCCTAATCCGTGCTGAATTAGGCCAACCTGGGACCC +TACTAGGAGATGATCAGATCTACAATGTCATTGTAACCGCCCATGCATTC +GTAATAATTTTCTTTATGGTCATACCCATTATAATCGGAGGATTCGGAAA +CTGATTAGTCCCCCTGATAATTGGAGCACCTGATATAGCTTTCCCCCGAA +TAAACAACATAAGCTTCTGATTACTTCCCCCATCATTCCTACTTCTTCTC +GCTTCCTCAATAATTGAAGCAGGTGCCGGAACAGGCTGAACCGTATATCC +TCCTCTAGCTGGAAATCTGGCGCATGCAGGAGCCTCTGTTGACTTAACCA +TTTTCTCTCTCCACCTAGCTGGGGTGTCCTCGATTTTAGGTGCCATCAAC +TTTATTACCACAATCATTAACATAAAACCACCAGCCCTATCCCAATATCA +AACCCCCCTATTCGTTTGATCTGTCCTTATTACGGCAGTACTCCTTCTCC +TAGCCCTCCCGGTCCTAGCAGCAGGCATTACCATGCTTCTCACAGACCGT +AACCTGAACACTACTTTCTTCGACCCCGCAGGAGGAGGGGATCCAATCCT +TTATCAACACCTATTCTGATTCTTCGGACACCCCGAAGTCTATATTCTTA +TCCTACCAGGCTTCGGTATAATCTCACACATCGTCACATACTACTCAGGT +AAAAAGGAACCTTTTGGCTACATGGGTATAGTGTGAGCTATAATATCCAT +TGGCTTTCTAGGCTTCATCGTATGGGCTCACCACATGTTTACAGTAGGGA +TAGACGTTGACACACGAGCATACTTCACATCAGCTACCATAATCATCGCT +ATCCCTACTGGTGTAAAAGTATTCAGCTGACTAGCCACCCTGCACGGAGG +AAATATCAAATGATCTCCAGCTATACTCTGAGCTCTAGGCTTCATCTTCT +TATTCACAGTAGGAGGTCTAACAGGAATCGTCCTAGCTAACTCATCCCTA +GATATTGTTCTCCACGATACTTATTATGTAGTAGCACATTTCCATTATGT +CCTGTCTATAGGAGCAGTCTTCGCCATTATGGGGGGATTTGTACACTGAT +TCCCTCTATTCTCAGGATACACACTCAACCAAACCTGAGCAAAAATCCAC +TTTACAATTATATTCGTAGGGGTAAATATAACCTTCTTCCCACAACATTT +CCTTGGCCTCTCAGGAATGCCACGACGCTATTCTGATTATCCAGACGCAT +ATACAACATGAAATACCATCTCATCCATAGGATCTTTTATCTCACTTACA +GCAGTGATACTAATAATTTTCATAATTTGAGAAGCGTTCGCATCCAAACG +AGAAGTGTCTACAGTAGAATTAACCTCAACTAATCTGGAATGACTACACG +GATGCCCCCCACCATACCACACATTTGAAGAACCCACCTACGTAAACCTA +AAAtaagaaaggaaggaatcgaaccccctctaactggtttcaagccaata +tcataaccactatgtctttctcCATCAATTGAGGTATTAGTAAAAATTAC +ATGACTTTGTCAAAGTTAAATTATAGGTTAAACCCCTATATACCTCTATG +GCCTACCCCTTCCAACTAGGATTCCAAGACGCAACATCCCCTATTATAGA +AGAACTCCTACACTTCCACGACCACACACTAATAATCGTATTCCTAATTA +GCTCTCTAGTATTATATATTATCTCATCAATACTAACAACTAAATTAACC +CATACCAGCACCATAGATGCTCAAGAAGTAGAGACAATTTGAACGATTTT +ACCAGCCATCATCCTTATTCTAATCGCCCTCCCATCCCTACGAATTCTAT +ATATAATAGATGAAATCAATAATCCGTCCCTCACAGTCAAAACAATAGGC +CACCAATGATACTGAAGCTACGAGTATACCGATTACGAAGACTTGACCTT +TGACTCCTACATGATCCCCACATCAGACCTAAAACCAGGAGAATTACGTC +TTCTAGAAGTCGACAATCGAGTGGTTCTCCCCATAGAAATAACCATCCGA +ATGCTAATTTCATCCGAAGACGTCCTACACTCATGAGCTGTGCCCTCCCT +AGGCCTAAAAACAGACGCTATCCCTGGGCGCCTAAATCAGACAACTCTCG +TGGCCTCTCGACCAGGACTTTACTACGGTCAATGCTCAGAGATCTGCGGA +TCAAACCACAGCTTTATACCAATTGTCCTTGAACTAGTTCCACTGAAACA +CTTCGAAGAATGATCTGCATCAATATTATAAAGTCACTAAGAAGCTATTA +TAGCATTAACCTTTTAAGTTAAAGATTGAGGGTTCAACCCCCTCCCTAGT +GATATGCCACAGTTGGATACATCAACATGATTTATTAATATCGTCTCAAT +AATCCTAACTCTATTTATTGTATTTCAACTAAAAATCTCAAAGCACTCCT +ATCCGACACACCCAGAAGTAAAGACAACCAAAATAACAAAACACTCTGCC +CCTTGAGAATCAAAATGAACGAAAATCTATTCGCCTCTTTCGCTACCCCA +ACAATAGTAGGCCTCCCTATTGTAATTCTGATCATCATATTTCCCAGCAT +CCTATTCCCCTCACCCAACCGACTAATCAACAATCGCCTAATCTCAATTC +AACAATGGCTAGTCCAACTTACATCAAAACAAATAATAGCTATCCATAAC +AGCAAAGGACAAACCTGAACTCTTATACTCATATCACTGATCCTATTCAT +TGGCTCAACAAACTTATTAGGCCTACTACCTCACTCATTTACACCAACAA +CACAACTATCAATAAACCTAGGCATAGCTATTCCCCTATGGGCAGGGACA +GTATTCATAGGCTTTCGTCACAAAACAAAAGCAGCCCTAGCCCACTTTCT +ACCTCAAGGGACGCCCATTTTCCTCATCCCCATACTAGTAATTATCGAGA +CTATCAGCCTATTTATTCAACCTGTAGCCCTAGCCGTGCGGCTAACCGCT +AACATTACCGCCGGACACCTCCTAATACACCTCATCGGAGGGGCAACACT +AGCCCTCATAAGCATCAGCCCCTCAACAGCCCTTATTACGTTTATCATCC +TAATTCTACTAACTATCCTCGAATTCGCAGTAGCTATAATCCAAGCCTAC +GTATTCACTCTCCTGGTAAGCCTTTACTTACACGACAACACCTAATGACC +CACCAAACCCACGCTTACCACATAGTAAACCCCAGCCCATGACCACTTAC +AGGAGCCCTATCAGCCCTCCTGATAACATCAGGACTAGCCATGTGATTTC +ACTTTAACTCAACCTTACTTCTAGCTATAGGGCTATTAACTAACATCCTT +ACCATATATCAATGATGACGAGACATCATCCGAGAAAGCACATTCCAAGG +CCATCACACATCAATCGTTCAAAAGGGACTCCGATATGGCATAATCCTTT +TTATTATCTCAGAAGTCTTCTTCTTCTCTGGCTTCTTCTGAGCCTTTTAC +CACTCAAGCCTAGCCCCCACACCCGAACTAGGCGGCTGCTGACCACCCAC +AGGTATCCACCCCTTAAACCCCCTAGAAGTCCCCTTACTCAACACCTCAG +TGCTCCTAGCATCTGGAGTCTCTATCACCTGAGCCCACCATAGCCTAATA +GAAGGAAACCGTAAAAATATGCTCCAAGGCCTATTCATCACAATTTCACT +AGGCGTATACTTCACCCTTCTCCAAGCCTCAGAATACTATGAAGCCTCAT +TTACTATTTCAGATGGAGTATACGGATCAACATTTTTCGTAGCAACAGGG +TTCCACGGACTACACGTAATTATCGGATCTACCTTCCTCATTGTATGTTT +CCTACGCCAACTAAAATTCCACTTTACATCCAGCCACCACTTCGGATTCG +AAGCAGCCGCTTGATACTGACACTTCGTCGACGTAGTCTGACTATTCTTG +TACGTCTCTATTTATTGATGAGGATCCTATTCTTTTAGTATTGACCAGTA +CAATTGACTTCCAATCAATCAGCTTCGGTATAACCCGAAAAAGAATAATA +AACCTCATACTGACACTCCTCACTAACACATTACTAGCCTCGCTACTCGT +ACTCATCGCATTCTGACTACCACAACTAAACATCTATGCAGAAAAAACCA +GCCCATATGAATGCGGATTTGACCCTATAGGGTCAGCACGCCTCCCCTTC +TCAATAAAATTTTTCTTAGTGGCCATTACATTTCTGCTATTCGACTTAGA +AATTGCCCTCCTATTACCCCTTCCATGAGCATCCCAAACAACTAACCTAA +ACACTATACTTATCATAGCACTAGTCCTAATCTCTCTTCTAGCCATCAGC +CTAGCCTACGAATGAACCCAAAAAGGACTAGAATGAACTGAGTATGGTAA +TTAGTTTAAACCAAAACAAATGATTTCGACTCATTAAACTATGATTAACT +TCATAATTACCAACATGTCACTAGTCCATATTAATATCTTCCTAGCATTC +ACAGTATCCCTCGTAGGCCTACTAATGTACCGATCCCACCTAATATCCTC +ACTCCTATGCCTAGAAGGAATAATACTATCACTATTCGTCATAGCAACCA +TAATAGTCCTAAACACCCACTTCACACTAGCTAGTATAATACCTATCATC +TTACTAGTATTTGCTGCCTGCGAACGAGCTCTAGGATTATCCCTACTAGT +CATAGTCTCCAATACTTATGGAGTAGACCACGTACAAAACCTTAACCTCC +TCCAATGCTAAAAATTATCATTCCCACAATCATACTTATGCCCCTTACAT +GACTATCAAAAAAGAATATAATCTGAATCAACACTACAACCTATAGTCTA +TTAATCAGCCTTATCAGCCTATCCCTCCTAAACCAACCTAGCAACAATAG +CCTAAACTTCTCACTAATATTCTTCTCCGATCCCCTATCAGCCCCACTTC +TGGTGTTGACAACATGACTACTGCCACTAATACTCATAGCCAGCCAACAC +CATCTATCTAAGGAACCACTAATCCGAAAAAAACTCTACATCACCATGCT +AACCATACTTCAAACTTTCCTAATCATGACTTTTACCGCCACAGAACTAA +TCTCCTTCTACATCCTATTTGAAGCCACATTAGTTCCAACACTAATTATC +ATCACCCGCTGAGGCAACCAAACAGAACGCCTGAACGCAGGCCTCTACTT +CCTATTCTACACACTAATAGGTTCCCTCCCACTCTTAGTTGCACTAATCT +CTATCCAAAACCTAACAGGCTCACTAAACTTCCTATTAATTCAATACTGA +AACCAAGCACTACCCGACTCTTGATCCAATATTTTCCTATGACTAGCATG +TATAATAGCATTCATAGTCAAAATACCGGTATATGGTCTTCACCTCTGAC +TCCCAAAAGCCCATGTAGAAGCCCCAATTGCCGGATCCATAGTGCTAGCA +GCCATTCTACTAAAACTAGGAGGCTACGGAATACTACGAATTACAACAAT +ACTAAACCCCCAAACTAGCTTTATAGCCTACCCCTTCCTCATACTATCCC +TGTGAGGAATAATCATAACTAGTTCCATCTGCTTGCGACAAACCGATCTA +AAATCACTTATTGCATACTCCTCTGTCAGCCACATAGCCCTAGTAATCGT +AGCCGTCCTCATCCAAACACCATGAAGTTATATAGGAGCTACAGCCCTAA +TAATCGCTCACGGCCTTACATCATCAATACTATTCTGCCTGGCAAACTCA +AATTACGAACGTACCCATAGCCGAACTATAATCCTAGCCCGCGGGCTTCA +AACACTTCTTCCCCTTATAGCAGCCTGATGACTATTAGCCAGCCTAACCA +ACCTGGCCCTCCCTCCCAGCATTAACCTAATTGGAGAGCTATTCGTAGTA +ATATCATCATTCTCATGATCAAATATTACCATTATCCTAATAGGAGCCAA +TATCACCATCACCGCCCTCTACTCCCTATACATACTAATCACAACACAAC +GAGGGAAATACACACACCATATCAACAGCATTAAACCTTCATTTACACGA +GAAAACGCACTCATGGCCCTCCACATGACTCCCCTACTACTCCTATCACT +TAACCCTAAAATTATCCTAGGCTTTACGTACTGTAAATATAGTTTAACAA +AAACACTAGATTGTGGATCTAGAAACAGAAACTTAATATTTCTTATTTAC +CGAGAAAGTATGCAAGAACTGCTAATTCATGCCCCCATGTCCAACAAACA +TGGCTCTCTCAAACTTTTAAAGGATAGGAGCTATCCGTTGGTCTTAGGAA +CCAAAAAATTGGTGCAACTCCAAATAAAAGTAATCAACATGTTCTCCTCC +CTCATACTAGTTTCACTATTAGTACTAACCCTCCCAATCATATTATCAAT +CTTCAATACCTACAAAAACAGCACGTTCCCGCATCATGTAAAAAACACTA +TCTCATATGCCTTCATTACTAGCCTAATTCCCACTATAATATTTATTCAC +TCTGGACAAGAAACAATTATCTCAAACTGACACTGAATAACCATACAAAC +CCTCAAACTATCCCTAAGCTTCAAACTAGATTACTTCTCAATAATTTTCG +TACCAGTAGCCCTATTCGTAACATGATCTATTATGGAATTCTCCCTATGA +TACATGCACTCAGATCCTTACATTACTCGATTTTTTAAATACTTACTTAC +ATTCCTCATCACTATAATAATTCTAGTCACAGCTAACAACCTTTTCCAAC +TGTTCATCGGATGGGAGGGAGTAGGCATCATGTCATTCTTACTAATCGGA +TGATGATACGGCCGAACAGATGCCAACACCGCGGCCCTTCAAGCAATCCT +TTATAACCGCATCGGGGATATCGGCTTCATCATGGCCATAGCCTGATTCC +TATTCAACACCAACACATGAGACCTCCAACAAATCTTCATACTCGACCCC +AACCTTACCAACCTCCCGCTCCTAGGCCTCCTCCTAGCCGCAACTGGCAA +ATCCGCTCAATTTGGACTCCACCCATGACTTCCTTCAGCCATAGAGGGCC +CTACACCAGTCTCAGCCCTACTCCACTCCAGCACAATAGTTGTAGCAGGC +GTCTTCCTGCTAATCCGCTTCCATCCACTAATAGAAAACAACAAAACAAT +CCAGTCACTTACCCTATGCCTAGGAGCCATCACCACACTATTCACAGCAA +TCTGCGCACTCACTCAAAACGATATCAAAAAAATCATTGCTTTCTCCACC +TCCAGCCAACTAGGCCTGATAATCGTAACCATCGGTATCAATCAACCCTA +CCTAGCATTCCTCCACATTTGCACTCACGCATTCTTCAAAGCTATACTAT +TTATATGTTCCGGATCCATTATCCACAGCCTAAATGACGAGCAAGATATC +CGAAAAATAGGCGGACTATTTAATGCAATACCCTTCACCACCACATCTCT +AATTATTGGCAGCCTTGCACTCACCGGAATTCCTTTCCTCACAGGCTTCT +ACTCCAAAGACCTCATCATCGAAACCGCCAACACATCGTACACCAACGCC +TGAGCCCTACTAATAACTCTCATTGCCACATCCCTCACAGCTGTCTACAG +TACCCGAATCATCTTCTTTGCACTCCTAGGGCAACCCCGCTTCCTCCCTC +TGACCTCAATCAACGAAAATAACCCCTTTCTAATTAACTCCATCAAACGC +CTCTTAATTGGCAGCATTTTTGCCGGATTCTTCATCTCCAACAATATCTA +CCCCACAACCGTCCCAGAAATAACCATACCTACTTACATAAAACTCACCG +CCCTCGCAGTAACCATCCTAGGATTTACACTAGCCCTAGAACTAAGCTTG +ATAACCCATAACTTAAAACTAGAACACTCCACCAACGTATTCAAATTCTC +CAACCTCCTAGGATACTACCCAACAATTATACACCGACTCCCACCGCTCG +CTAACCTATCAATAAGCCAAAAATCAGCATCACTTCTACTAGACTCAATC +TGACTAGAAAACATCCTGCCAAAATCTATCTCCCAGTTCCAAATAAAAAC +CTCGATCCTAATTTCCACCCAAAAAGGACAAATCAAATTATATTTCCTCT +CATTCCTCATCACCCTTACCCTAAGCATACTACTTTTTAATCTCCACGAG +TAACCTCTAAAATTACCAAGACCCCAACAAGCAACGATCAACCAGTCACA +ATCACAACCCAAGCCCCATAACTATACAATGCAGCAGCCCCTATAATTTC +CTCACTAAACGCCCCAGAATCTCCAGTATCATAAATAGCTCAAGCCCCCA +CACCACTAAACTTAAACACTACCCCCACTTCCTCACTCTTCAGAACATAT +AAAACCAACATAACCTCCATCAACAACCCTAAAAGAAATACCCCCATAAC +AGTCGTATTAGACACCCATACCTCAGGATACTGCTCAGTAGCCATAGCCG +TTGTATAACCAAAAACAACCAACATTCCTCCCAAATAAATCAAAAACACC +ATCAACCCCAAAAAGGACCCTCCAAAATTCATAATAATACCACAACCTAC +CCCTCCACTTACAATCAGCACTAAACCCCCATAAATAGGTGAAGGTTTTG +AAGAAAACCCCACAAAACTAACAACAAAAATAACACTCAAAATAAACACA +ATATATGTCATCATTATTCCCACGTGGAATCTAACCACGACCAATGACAT +GAAAAATCATCGTTGTATTTCAACTATAAGAACACCAATGACAAACATCC +GGAAATCTCACCCACTAATTAAAATCATCAATCACTCTTTTATTGACCTA +CCAGCCCCCTCAAACATTTCATCATGATGAAACTTCGGCTCCCTCCTAGG +AATCTGCCTAATCCTCCAAATCTTAACAGGCCTATTCCTAGCCATACACT +ACACATCAGACACGACAACTGCCTTCTCATCCGTCACTCACATCTGCCGA +GACGTTAACTACGGATGAATTATTCGCTACCTCCATGCCAACGGAGCATC +AATATTTTTTATCTGCCTCTTCATTCACGTAGGACGCGGCCTCTACTACG +GCTCTTACACATTCCTAGAGACATGAAACATTGGAATCATCCTACTTTTC +ACAGTTATAGCTACAGCATTCATGGGCTATGTCCTACCATGAGGCCAAAT +ATCCTTTTGAGGAGCAACAGTCATCACGAACCTCCTATCAGCAATTCCCT +ACATCGGTACTACCCTCGTCGAGTGAATCTGAGGTGGATTCTCAGTAGAC +AAAGCCACCCTTACCCGATTTTTTGCTTTCCACTTCATCCTACCCTTCAT +CATCACAGCCCTGGTAGTCGTACATTTACTATTTCTTCACGAAACAGGAT +CTAATAACCCCTCAGGAATCCCATCCGATATGGACAAAATCCCATTCCAC +CCATATTATACAATTAAAGACATCCTAGGACTCCTCCTCCTGATCTTGCT +CCTACTAACTCTAGTATTATTCTCCCCCGACCTCCTAGGAGACCCAGACA +ACTACACCCCAGCTAACCCTCTCAGCACTCCCCCTCATATTAAACCAGAA +TGGTACTTCCTGTTTGCCTACGCCATCCTACGCTCCATTCCCAACAAACT +AGGCGGCGTATTAGCCCTAATCCTCTCCATCCTGATCCTAGCACTCATCC +CCACCCTCCACATATCAAAACAACGAAGCATAATATTCCGGCCTCTCAGC +CAATGCGTATTCTGACTCTTAGTGGCAGACTTACTGACACTAACATGAAT +CGGCGGACAGCCAGTGGAACACCCATACGTAATTATCGGCCAACTGGCCT +CAATCCTCTACTTCTCCCTAATTCTCATTTTTATACCACTCGCAAGCACC +ATCGAAAACAATCTTCTAAAATGAAGAGTCCCTGTAGTATATCGCACATT +ACCCTGGTCTTGTAAACCAGAAAAGGGGGAAAACGTTTCCTCCCAAGGAC +TATCAAGGAAGAAGCTCTAGCTCCACCATCAACACCCAAAGCTGAAATTC +TACTTAAACTATTCCTTGATTTCTTCCCCTAAACGACAACAATTTACCCT +CATGTGCTATGTCAGTATCAGATTATACCCCCACATAACACCATACCCAC +CTGACATGCAATATCTTATGAATGGCCTATGTACGTCGTGCATTAAATTG +TCTGCCCCATGAATAATAAGCATGTACATAATATCATTTATCTTACATAA +GTACATTATATTATTGATCGTGCATACCCCATCCAAGTCAAATCATTTCC +AGTCAACACGCATATCACAGCCCATGTTCCACGAGCTTAATCACCAAGCC +GCGGGAAATCAGCAACCCTCCCAACTACGTGTCCCAATCCTCGCTCCGGG +CCCATCCAAACGTGGGGGTTTCTACAATGAAACTATACCTGGCATCTGGT +TCTTTCTTCAGGGCCATTCCCACCCAACCTCGCCCATTCTTTCCCCTTAA +ATAAGACATCTCGATGGACTAATGACTAATCAGCCCATGCTCACACATAA +CTGTGATTTCATGCATTTGGTATCTTTTTATATTTGGGGATGCTATGACT +CAGCTATGGCCGTCAAAGGCCTCGACGCAGTCAATTAAATTGAAGCTGGA +CTTAAATTGAACGTTATTCCTCCGCATCAGCAACCATAAGGTGTTATTCA +GTCCATGGTAGCGGGACATAGGAAACAAgtgcacctgtgcacctgtgcac +ctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacct +gtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgt +gcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgc +acctgtgcacctgtgcacctgtgcacctgtgcacctgtgcacctgtgcac +ctgtgcacctACCCGCGCAGTAAGCAAGTAATATAGCTTTCTTAATCAAA +CCCCCCCTACCCCCCATTAAACTCCACATATGTACATTCAACACAATCTT +GCCAAACCCCAAAAACAAGACTAAACAATGCACAATACTTCATGAAGCTT +AACCCTCGCATGCCAACCATAATAACTCAACACACCTAACAATCTTAACA +GAACTTTCCCCCCGCCATTAATACCAACATGCTACTTTAATCAATAAAAT +TTCCATAGACAGGCATCCCCCTAGATCTAATTTTCTAAATCTGTCAACCC +TTCTTCCCCC diff -r 93474ea5c366 -r 495dda4c0693 test-data/sam_to_bam_in1.sam --- a/test-data/sam_to_bam_in1.sam Thu Dec 03 16:16:12 2009 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,10 +0,0 @@ -HWI-EAS91_1_30788AAXX:1:1:1513:715 16 chrM 9563 25 36M * 0 0 CTGACTACCACAACTAAACATCTATGCNNAAAAAAC I+-II?IDIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1698:516 16 chrM 2735 25 36M * 0 0 TTTACACTCAGAGGTTCAACTCCTCTCNNTAACAAC I9IIIII5IIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1491:637 16 chrM 10864 25 36M * 0 0 TGTAGAAGCCCCAATTGCCGGATCCATNNTGCTAGC DBAIIIIIIIIIIIFIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1711:249 16 chrM 10617 25 36M * 0 0 ACCAAACAGAACGCCTGAACGCAGGCCNNTACTTCC IIIIIIIIIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1634:211 0 chrM 9350 25 36M * 0 0 GAAGCAGNNGCTTGATACTGACACTTCGTCGACGTA IIIIIII""IIIIIIIIIIIIIIIIIIIIII9IIDF NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1218:141 16 chrM 14062 25 36M * 0 0 ACAAAACTAACAACAAAAATAACACTCNNAATAAAC I+IIII1IIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1398:854 16 chrM 3921 25 36M * 0 0 CACCCTTCCCGTACTAATAAATCCCCTNNTCTTCAC IIIII=AIIIIIIIIIIIIIIBIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1310:991 16 chrM 10002 25 36M * 0 0 CTCCTATGCCTAGAAGGAATAATACTANNACTATTC I:2IEI:IIDIIIIII4IIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1716:413 0 chrM 6040 25 36M * 0 0 GATCCAANNCTTTATCAACACCTATTCTGATTCTTC IIIIIII""IIIIIIIIIIIIIIIIIIIIIIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1630:59 16 chrM 12387 25 36M * 0 0 TCATACTCGACCCCAACCTTACCAACCNNCCGCTCC FIIHII;IIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 diff -r 93474ea5c366 -r 495dda4c0693 test-data/sam_to_bam_in2.sam --- a/test-data/sam_to_bam_in2.sam Thu Dec 03 16:16:12 2009 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,10 +0,0 @@ -HWI-EAS91_1_30788AAXX:1:1:1095:605 0 chrM 23 25 36M * 0 0 AAGCAAGNNACTGAAAATGCCTAGATGAGTATTCTT IIIIIII""IIIIIIIIIIIIIIIEIIIIIIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1650:1185 0 chrM 14956 25 36M * 0 0 ACCCCAGNNAACCCTCTCAGCACTCCCCCTCATATT IIIIIII""IIIIIIIIIIII6IIIIIIIII5I-II NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:799:192 16 chrM 8421 25 36M * 0 0 CCTGTAGCCCTAGCCGTGCGGCTAACCNNTAACATT II%::I<IIIIIEIII8IIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1082:719 16 chrM 7191 25 36M * 0 0 TAAATTAACCCATACCAGCACCATAGANNCTCAAGA <III0EII3+3I29I>III8AIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1746:1180 16 chrM 12013 25 36M * 0 0 CCTAAGCTTCAAACTAGATTACTTCTCNNTAATTTT IIIIIIIIFIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:606:460 0 chrM 4552 25 36M * 0 0 TTAATTTNNATTATAATAACACTCACAATATTCATA IIIIIII""IIIIIIIIIIIIIIIIII?I6IIIII6 NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1059:362 16 chrM 7348 25 36M * 0 0 GGCCACCAATGATACTGAAGCTACGAGNNTACCGAT II/<)2IIIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1483:1161 16 chrM 15080 25 36M * 0 0 TCCTGATCCTAGCACTCATCCCCACCCNNCACATAT HIIIIIFIIAIHIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1273:600 16 chrM 13855 25 36M * 0 0 GTATTAGACACCCATACCTCAGGATACNNCTCAGTA IIIIIIIIIIIIIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 -HWI-EAS91_1_30788AAXX:1:1:1190:1283 16 chrM 15338 25 36M * 0 0 TATATCGCACATTACCCTGGTCTTGTANNCCAGAAA EIII?-IIIIIAIIIIIIIIIIIIIII""IIIIIII NM:i:1 X1:i:1 MD:Z:7N0N27 diff -r 93474ea5c366 -r 495dda4c0693 test-data/sam_to_bam_out1.bam Binary file test-data/sam_to_bam_out1.bam has changed diff -r 93474ea5c366 -r 495dda4c0693 test-data/sam_to_bam_out2.bam Binary file test-data/sam_to_bam_out2.bam has changed diff -r 93474ea5c366 -r 495dda4c0693 tools/data_source/data_source.py --- a/tools/data_source/data_source.py Thu Dec 03 16:16:12 2009 -0500 +++ b/tools/data_source/data_source.py Fri Dec 04 19:51:46 2009 -0500 @@ -12,6 +12,7 @@ sys.exit() def check_gzip( filename ): + # TODO: This needs to check for BAM files since they are compressed and must remain so ( see upload.py ) temp = open( filename, "U" ) magic_check = temp.read( 2 ) temp.close() @@ -66,6 +67,7 @@ out.write( chunk ) out.close() if check_gzip( filename ): + # TODO: This needs to check for BAM files since they are compressed and must remain so ( see upload.py ) fd, uncompressed = tempfile.mkstemp() gzipped_file = gzip.GzipFile( filename ) while 1: diff -r 93474ea5c366 -r 495dda4c0693 tools/data_source/upload.py --- a/tools/data_source/upload.py Thu Dec 03 16:16:12 2009 -0500 +++ b/tools/data_source/upload.py Fri Dec 04 19:51:46 2009 -0500 @@ -4,7 +4,7 @@ # WARNING: Changes in this tool (particularly as related to parsing) may need # to be reflected in galaxy.web.controllers.tool_runner and galaxy.tools -import urllib, sys, os, gzip, tempfile, shutil, re, gzip, zipfile, codecs +import urllib, sys, os, gzip, tempfile, shutil, re, gzip, zipfile, codecs, binascii from galaxy import eggs # need to import model before sniff to resolve a circular import dependency import galaxy.model @@ -18,7 +18,6 @@ def stop_err( msg, ret=1 ): sys.stderr.write( msg ) sys.exit( ret ) - def file_err( msg, dataset, json_file ): json_file.write( to_json_string( dict( type = 'dataset', ext = 'data', @@ -28,7 +27,6 @@ os.remove( dataset.path ) except: pass - def safe_dict(d): """ Recursively clone json structure with UTF-8 dictionary keys @@ -40,7 +38,6 @@ return [safe_dict(x) for x in d] else: return d - def check_html( temp_name, chunk=None ): if chunk is None: temp = open(temp_name, "U") @@ -64,7 +61,6 @@ if chunk is None: temp.close() return False - def check_binary( temp_name, chunk=None ): if chunk is None: temp = open( temp_name, "U" ) @@ -85,21 +81,42 @@ if chunk is None: temp.close() return False - def check_gzip( temp_name ): + # This is sort of hacky. BAM is compressed in the BGZF format, and must + # not be uncompressed in upon upload ( it will be detected as gzipped ). + # The tuple we're returning from here contains boolean values for + # ( is_compressed, is_valid, is_bam ). temp = open( temp_name, "U" ) magic_check = temp.read( 2 ) temp.close() if magic_check != util.gzip_magic: - return ( False, False ) + return ( False, False, False ) CHUNK_SIZE = 2**15 # 32Kb gzipped_file = gzip.GzipFile( temp_name ) chunk = gzipped_file.read( CHUNK_SIZE ) gzipped_file.close() - if check_html( temp_name, chunk=chunk ) or check_binary( temp_name, chunk=chunk ): - return( True, False ) - return ( True, True ) - + if check_html( temp_name, chunk=chunk ): + return ( True, False, False ) + if check_binary( temp_name, chunk=chunk ): + # We do support some binary data types, so check if the compressed binary file is valid + # We currently only check for [ 'sff', 'bam' ] + # TODO: this should be fixed to more easily support future-supported binary data types. + # This is currently just copied from the sniff methods. + # The first 4 bytes of any bam file is 'BAM\1', and the file is binary. + try: + header = gzip.open( temp_name ).read(4) + if binascii.b2a_hex( header ) == binascii.hexlify( 'BAM\1' ): + return ( True, True, True ) + except: + pass + try: + header = gzip.open( temp_name ).read(4) + if binascii.b2a_hex( header ) == binascii.hexlify( '.sff' ): + return ( True, True, False ) + except: + pass + return ( True, False, False ) + return ( True, True, False ) def check_zip( temp_name ): if not zipfile.is_zipfile( temp_name ): return ( False, False, None ) @@ -116,14 +133,12 @@ if ext != test_ext: return ( True, False, test_ext ) return ( True, True, test_ext ) - def parse_outputs( args ): rval = {} for arg in args: id, files_path, path = arg.split( ':', 2 ) rval[int( id )] = ( path, files_path ) return rval - def add_file( dataset, json_file, output_path ): data_type = None line_count = None @@ -153,15 +168,19 @@ ext = sniff.guess_ext( dataset.path, is_multi_byte=True ) else: # See if we have a gzipped file, which, if it passes our restrictions, we'll uncompress - is_gzipped, is_valid = check_gzip( dataset.path ) + is_gzipped, is_valid, is_bam = check_gzip( dataset.path ) if is_gzipped and not is_valid: file_err( 'The uploaded file contains inappropriate content', dataset, json_file ) return - elif is_gzipped and is_valid: - # We need to uncompress the temp_name file + elif is_gzipped and is_valid and is_bam: + ext = 'bam' + data_type = 'bam' + elif is_gzipped and is_valid and not is_bam: + # We need to uncompress the temp_name file, but BAM files must remain compressed + # in order for samtools to function on them CHUNK_SIZE = 2**20 # 1Mb - fd, uncompressed = tempfile.mkstemp( prefix='data_id_%s_upload_gunzip_' % dataset.dataset_id, dir=os.path.dirname( dataset.path ) ) - gzipped_file = gzip.GzipFile( dataset.path ) + fd, uncompressed = tempfile.mkstemp( prefix='data_id_%s_upload_gunzip_' % dataset.dataset_id, dir=os.path.dirname( dataset.path ), text=False ) + gzipped_file = gzip.GzipFile( dataset.path, 'rb' ) while 1: try: chunk = gzipped_file.read( CHUNK_SIZE ) @@ -229,7 +248,7 @@ if check_html( dataset.path ): file_err( 'The uploaded file contains inappropriate content', dataset, json_file ) return - if data_type != 'binary' and data_type != 'zip': + if data_type != 'bam' and data_type != 'binary' and data_type != 'zip': if dataset.space_to_tab: line_count = sniff.convert_newlines_sep2tabs( dataset.path ) else: diff -r 93474ea5c366 -r 495dda4c0693 tools/samtools/sam_to_bam.py --- a/tools/samtools/sam_to_bam.py Thu Dec 03 16:16:12 2009 -0500 +++ b/tools/samtools/sam_to_bam.py Fri Dec 04 19:51:46 2009 -0500 @@ -1,31 +1,27 @@ #! /usr/bin/python - """ -Converts SAM data to BAM format. - -usage: %prog [options] - -i, --input1=i: SAM file to be converted - -d, --dbkey=d: dbkey value - -r, --ref_file=r: Reference file if choosing from history - -o, --output1=o: BAM output - -x, --index_dir=x: Index directory - -usage: %prog input_file dbkey ref_list output_file +Converts SAM data to sorted BAM data. +usage: sam_to_bam.py [options] + --input1: SAM file to be converted + --dbkey: dbkey value + --ref_file: Reference file if choosing from history + --output1: output dataset in bam format + --index_dir: GALAXY_DATA_INDEX_DIR """ -import os, sys, tempfile +import optparse, os, sys, subprocess, tempfile, shutil, gzip from galaxy import eggs import pkg_resources; pkg_resources.require( "bx-python" ) from bx.cookbook import doc_optparse +from galaxy import util def stop_err( msg ): sys.stderr.write( "%s\n" % msg ) sys.exit() -def check_seq_file( dbkey, GALAXY_DATA_INDEX_DIR ): - seq_file = "%s/sam_fa_indices.loc" % GALAXY_DATA_INDEX_DIR +def check_seq_file( dbkey, cached_seqs_pointer_file ): seq_path = '' - for line in open( seq_file ): + for line in open( cached_seqs_pointer_file ): line = line.rstrip( '\r\n' ) if line and not line.startswith( "#" ) and line.startswith( 'index' ): fields = line.split( '\t' ) @@ -38,48 +34,80 @@ def __main__(): #Parse Command Line - options, args = doc_optparse.parse( __doc__ ) - seq_path = check_seq_file( options.dbkey, options.index_dir ) + parser = optparse.OptionParser() + parser.add_option( '', '--input1', dest='input1', help='The input SAM dataset' ) + parser.add_option( '', '--dbkey', dest='dbkey', help='The build of the reference dataset' ) + parser.add_option( '', '--ref_file', dest='ref_file', help='The reference dataset from the history' ) + parser.add_option( '', '--output1', dest='output1', help='The output BAM dataset' ) + parser.add_option( '', '--index_dir', dest='index_dir', help='GALAXY_DATA_INDEX_DIR' ) + ( options, args ) = parser.parse_args() + + cached_seqs_pointer_file = "%s/sam_fa_indices.loc" % options.index_dir + if not os.path.exists( cached_seqs_pointer_file ): + stop_err( "The required file (%s) does not exist." % cached_seqs_pointer_file ) + # If found for the dbkey, seq_path will look something like /depot/data2/galaxy/equCab2/sam_index/equCab2.fa, + # and the equCab2.fa file will contain fasta sequences. + seq_path = check_seq_file( options.dbkey, cached_seqs_pointer_file ) tmp_dir = tempfile.gettempdir() - os.chdir(tmp_dir) - tmpf1 = tempfile.NamedTemporaryFile(dir=tmp_dir) - tmpf1fai = '%s.fai' % tmpf1.name - tmpf2 = tempfile.NamedTemporaryFile(dir=tmp_dir) - tmpf3 = tempfile.NamedTemporaryFile(dir=tmp_dir) - tmpf3bam = '%s.bam' % tmpf3.name if options.ref_file == "None": - full_path = "%s.fai" % seq_path - if not os.path.exists( full_path ): - stop_err( "No sequences are available for '%s', request them by reporting this error." % options.dbkey ) - cmd1 = "cp %s %s; cp %s %s" % (seq_path, tmpf1.name, full_path, tmpf1fai) + # We're using locally cached reference sequences( e.g., /depot/data2/galaxy/equCab2/sam_index/equCab2.fa ). + # The indexes for /depot/data2/galaxy/equCab2/sam_index/equCab2.fa will be contained in + # a file named /depot/data2/galaxy/equCab2/sam_index/equCab2.fa.fai + fai_index_file_path = "%s.fai" % seq_path + if not os.path.exists( fai_index_file_path ): + stop_err( "No sequences are available for build (%s), request them by reporting this error." % options.dbkey ) else: - cmd1 = "cp %s %s; samtools faidx %s 2>/dev/null" % (options.ref_file, tmpf1.name, tmpf1.name) - cmd2 = "samtools view -bt %s -o %s %s 2>/dev/null" % (tmpf1fai, tmpf2.name, options.input1) - cmd3 = "samtools sort %s %s 2>/dev/null" % (tmpf2.name, tmpf3.name) - cmd4 = "cp %s %s" % (tmpf3bam, options.output1) - # either create index based on fa file or copy provided index to temp directory + try: + # Create indexes for history reference ( e.g., ~/database/files/000/dataset_1.dat ) using samtools faidx, which will: + # - index reference sequence in the FASTA format or extract subsequence from indexed reference sequence + # - if no region is specified, faidx will index the file and create <ref.fasta>.fai on the disk + # - if regions are specified, the subsequences will be retrieved and printed to stdout in the FASTA format + # - the input file can be compressed in the RAZF format. + # IMPORTANT NOTE: a real weakness here is that we are creating indexes for the history dataset + # every time we run this tool. It would be nice if we could somehow keep track of user's specific + # index files so they could be re-used. + fai_index_file_path = os.path.join( tmp_dir, os.path.basename( options.ref_file ) ) + # At this point, fai_index_file_path will look something like /tmp/dataset_13.dat + os.symlink( options.ref_file, fai_index_file_path ) + command = "samtools faidx %s 2>/dev/null" % fai_index_file_path + proc = subprocess.Popen( args=command, shell=True ) + proc.wait() + except Exception, e: + stop_err( 'Error creating indexes from reference (%s), %s' % ( options.ref_file, str( e ) ) ) try: - os.system(cmd1) - except Exception, eq: - stop_err("Error creating the reference list index.\n" + str(eq)) - # create original bam file + # Extract all alignments from the input SAM file to BAM format ( since no region is specified, all the alignments will be extracted ). + tmp_aligns_file = tempfile.NamedTemporaryFile() + tmp_aligns_file_name = tmp_aligns_file.name + tmp_aligns_file.close() + # IMPORTANT NOTE: for some reason the samtools view command gzips the resulting bam file without warning, + # and the docs do not currently state that this occurs ( very bad ). + command = "samtools view -bt %s -o %s %s 2>/dev/null" % ( fai_index_file_path, tmp_aligns_file_name, options.input1 ) + proc = subprocess.Popen( args=command, shell=True ) + proc.wait() + except Exception, e: + stop_err( 'Error extracting alignments from (%s), %s' % ( options.input1, str( e ) ) ) try: - os.system(cmd2) - except Exception, eq: - stop_err("Error running view command.\n" + str(eq)) - # sort original bam file to produce sorted output bam file - try: - os.system(cmd3) - os.system(cmd4) - except Exception, eq: - stop_err("Error sorting data and creating output file.\n" + str(eq)) - # cleanup temp files - tmpf1.close() - tmpf2.close() - tmpf3.close() - if os.path.exists(tmpf1fai): - os.remove(tmpf1fai) - if os.path.exists(tmpf3bam): - os.remove(tmpf3bam) + # Sort alignments by leftmost coordinates. File <out.prefix>.bam will be created. This command + # may also create temporary files <out.prefix>.%d.bam when the whole alignment cannot be fitted + # into memory ( controlled by option -m ). + tmp_sorted_aligns_file = tempfile.NamedTemporaryFile() + tmp_sorted_aligns_file_name = tmp_sorted_aligns_file.name + tmp_sorted_aligns_file.close() + command = "samtools sort %s %s 2>/dev/null" % ( tmp_aligns_file_name, tmp_sorted_aligns_file_name ) + proc = subprocess.Popen( args=command, shell=True ) + proc.wait() + except Exception, e: + stop_err( 'Error sorting alignments from (%s), %s' % ( tmp_aligns_file_name, str( e ) ) ) + # Move tmp_aligns_file_name to our output dataset location + sorted_bam_file = '%s.bam' % tmp_sorted_aligns_file_name + shutil.move( sorted_bam_file, options.output1 ) + if options.ref_file != "None": + # Remove the symlink from /tmp/dataset_13.dat to ~/database/files/000/dataset_13.dat + os.unlink( fai_index_file_path ) + # Remove the index file + index_file_name = '%s.fai' % fai_index_file_path + os.unlink( index_file_name ) + # Remove the tmp_aligns_file_name + os.unlink( tmp_aligns_file_name ) if __name__=="__main__": __main__() diff -r 93474ea5c366 -r 495dda4c0693 tools/samtools/sam_to_bam.xml --- a/tools/samtools/sam_to_bam.xml Thu Dec 03 16:16:12 2009 -0500 +++ b/tools/samtools/sam_to_bam.xml Fri Dec 04 19:51:46 2009 -0500 @@ -1,32 +1,29 @@ <tool id="sam_to_bam" name="SAM-to-BAM" version="1.0.0"> <description>converts SAM format to BAM format</description> <command interpreter="python"> - sam_to_bam.py - --input1=$source.input1 - --dbkey=${input1.metadata.dbkey} - #if $source.indexSource == "history": - --ref_file=$ref_file - #else - --ref_file="None" - #end if - --output1=$output1 - --index_dir=${GALAXY_DATA_INDEX_DIR} +sam_to_bam.py --input1=$source.input1 --dbkey=${input1.metadata.dbkey} +#if $source.index_source == "history": +--ref_file=$source.ref_file +#else +--ref_file="None" +#end if +--output1=$output1 --index_dir=${GALAXY_DATA_INDEX_DIR} </command> <inputs> <conditional name="source"> - <param name="indexSource" type="select" label="Choose the source for the reference list"> - <option value="built_in">Built-in</option> + <param name="index_source" type="select" label="Choose the source for the reference list"> + <option value="cached">Locally cached</option> <option value="history">History</option> </param> - <when value="built_in"> + <when value="cached"> <param name="input1" type="data" format="sam" label="SAM File to Convert"> <validator type="unspecified_build" /> <validator type="dataset_metadata_in_file" filename="sam_fa_indices.loc" metadata_name="dbkey" metadata_column="1" message="Sequences are not currently available for the specified build." line_startswith="index" /> </param> </when> <when value="history"> - <param name="input1" type="data" format="sam" label="SAM File to Convert" /> - <param name="ref_file" type="data" format="fasta" label="Choose the reference file" /> + <param name="input1" type="data" format="sam" label="Convert SAM file" /> + <param name="ref_file" type="data" format="fasta" label="Using reference file" /> </when> </conditional> </inputs> @@ -34,19 +31,16 @@ <data name="output1" format="bam"/> </outputs> <tests> +  <test> - <param name="indexSource" value="history" /> - <param name="input1" value="sam_to_bam_in1.sam" ftype="sam" /> - <param name="ref_file" value="chrM.fa" ftype="fasta" /> + <param name="index_source" value="history" /> + <param name="input1" value="3.sam" ftype="sam" /> + <param name="ref_file" value="chr_m.fasta" ftype="fasta" /> <output name="output1" file="sam_to_bam_out1.bam" /> </test> - </tests> <help> diff -r 93474ea5c366 -r 495dda4c0693 tools/sr_mapping/bowtie_wrapper.xml --- a/tools/sr_mapping/bowtie_wrapper.xml Thu Dec 03 16:16:12 2009 -0500 +++ b/tools/sr_mapping/bowtie_wrapper.xml Fri Dec 04 19:51:46 2009 -0500 @@ -338,7 +338,7 @@ </test> <test> <param name="genomeSource" value="history" /> - <param name="ownFile" value="chrM.fa" /> + <param name="ownFile" value="chr_m.fasta" /> <param name="index_settings" value="index_pre_set" /> <param name="sPaired" value="paired" /> <param name="input1" ftype="fastqsanger" value="bowtie_in2.fastq" /> @@ -349,7 +349,7 @@ </test> <test> <param name="genomeSource" value="history" /> - <param name="ownFile" value="chrM.fa" /> + <param name="ownFile" value="chr_m.fasta" /> <param name="index_settings" value="index_full" /> <param name="auto_b" value="set" /> <param name="packed" value="unpacked" /> diff -r 93474ea5c366 -r 495dda4c0693 tools/sr_mapping/lastz_wrapper.xml --- a/tools/sr_mapping/lastz_wrapper.xml Thu Dec 03 16:16:12 2009 -0500 +++ b/tools/sr_mapping/lastz_wrapper.xml Fri Dec 04 19:51:46 2009 -0500 @@ -125,10 +125,13 @@ <requirement type="binary">lastz</requirement> </requirements> <tests> - <test>  + <test> +  <param name="input2" value="b1.fasta" ftype="fasta" /> <param name="ref_source" value="cached" /> <param name="input1_2bit" value="phiX" /> @@ -156,10 +159,13 @@ <param name="num_threads" value="4" /> <output name="output1" file="lastz_wrapper_out2.sam" /> </test> - <test>  + <test> +  <param name="input2" value="b1.fasta" ftype="fasta" /> <param name="ref_source" value="history" /> <param name="input1" value="phiX.fasta" ftype="fasta" /> @@ -173,14 +179,7 @@ <param name="num_threads" value="4" /> <output name="output1" file="lastz_wrapper_out3.tabular" /> </test> - <test>  + <test> <param name="input2" value="b1.fasta" ftype="fasta" /> <param name="ref_source" value="history" /> <param name="input1" value="phiX_split.fasta" ftype="fasta" />

1 0

[hg] galaxy 3145: Requests in any state can be deleted by an admin
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/93474ea5c366 changeset: 3145:93474ea5c366 user: rc date: Thu Dec 03 16:16:12 2009 -0500 description: Requests in any state can be deleted by an admin diffstat: lib/galaxy/web/controllers/requests.py | 2 +- lib/galaxy/web/controllers/requests_admin.py | 22 ++++++---------------- 2 files changed, 7 insertions(+), 17 deletions(-) diffs (58 lines): diff -r 29b6fac01d78 -r 93474ea5c366 lib/galaxy/web/controllers/requests.py --- a/lib/galaxy/web/controllers/requests.py Thu Dec 03 15:33:15 2009 -0500 +++ b/lib/galaxy/web/controllers/requests.py Thu Dec 03 16:16:12 2009 -0500 @@ -542,7 +542,7 @@ action='new', **kwd) ) request = self.__save_request(trans, None, **kwd) - msg = 'The new request named %s has been created' % request.name + msg = 'The new request named %s has been created' % request.name if params.get('create_request_button', False) == 'Save': return trans.response.send_redirect( web.url_for( controller='requests', action='list', diff -r 29b6fac01d78 -r 93474ea5c366 lib/galaxy/web/controllers/requests_admin.py --- a/lib/galaxy/web/controllers/requests_admin.py Thu Dec 03 15:33:15 2009 -0500 +++ b/lib/galaxy/web/controllers/requests_admin.py Thu Dec 03 16:16:12 2009 -0500 @@ -149,7 +149,7 @@ confirm="More samples cannot be added to this request once it is submitted. Click OK to submit." ), grids.GridOperation( "Edit", allow_multiple=False, condition=( lambda item: not item.deleted ) ), grids.GridOperation( "Reject", allow_multiple=False, condition=( lambda item: not item.deleted and item.submitted() ) ), - grids.GridOperation( "Delete", allow_multiple=True, condition=( lambda item: not item.deleted and item.new() ) ), + grids.GridOperation( "Delete", allow_multiple=True, condition=( lambda item: not item.deleted ) ), grids.GridOperation( "Undelete", condition=( lambda item: item.deleted ) ), ] global_actions = [ @@ -387,7 +387,6 @@ return self.__show_request_form(trans) def __delete_request(self, trans, **kwd): id_list = util.listify( kwd['id'] ) - delete_failed = [] for id in id_list: try: request = trans.sa_session.query( trans.app.model.Request ).get( trans.security.decode_id(id) ) @@ -399,20 +398,11 @@ status='error', message=msg, **kwd) ) - # a request cannot be deleted once its submitted - if not request.new(): - delete_failed.append(request.name) - else: - request.deleted = True - trans.sa_session.add( request ) - trans.sa_session.flush() - if not len(delete_failed): - msg = '%i request(s) has been deleted.' % len(id_list) - status = 'done' - else: - msg = '%i request(s) has been deleted. %i request %s could not be deleted as they have been submitted.' % (len(id_list)-len(delete_failed), - len(delete_failed), str(delete_failed)) - status = 'warning' + request.deleted = True + trans.sa_session.add( request ) + trans.sa_session.flush() + msg = '%i request(s) has been deleted.' % len(id_list) + status = 'done' return trans.response.send_redirect( web.url_for( controller='requests_admin', action='list', status=status,

1 0

[hg] galaxy 3144: Add datasource tool configs for modMine and Ra...
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/29b6fac01d78 changeset: 3144:29b6fac01d78 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Dec 03 15:33:15 2009 -0500 description: Add datasource tool configs for modMine and Ratmine. diffstat: tool_conf.xml.main | 2 ++ tool_conf.xml.sample | 6 ++++-- tools/data_source/modmine.xml | 30 ++++++++++++++++++++++++++++++ tools/data_source/ratmine.xml | 29 +++++++++++++++++++++++++++++ 4 files changed, 65 insertions(+), 2 deletions(-) diffs (97 lines): diff -r 7aaa791a6283 -r 29b6fac01d78 tool_conf.xml.main --- a/tool_conf.xml.main Thu Dec 03 14:39:21 2009 -0500 +++ b/tool_conf.xml.main Thu Dec 03 15:33:15 2009 -0500 @@ -8,6 +8,8 @@ <tool file="data_source/biomart.xml" /> <tool file="data_source/gramene_mart.xml" /> <tool file="data_source/flymine.xml" /> + <tool file="data_source/modmine.xml" /> + <tool file="data_source/ratmine.xml" /> <tool file="data_source/eupathdb.xml" /> <tool file="data_source/encode_db.xml" /> <tool file="data_source/epigraph_import.xml" /> diff -r 7aaa791a6283 -r 29b6fac01d78 tool_conf.xml.sample --- a/tool_conf.xml.sample Thu Dec 03 14:39:21 2009 -0500 +++ b/tool_conf.xml.sample Thu Dec 03 15:33:15 2009 -0500 @@ -9,10 +9,12 @@ <tool file="data_source/biomart.xml" /> <tool file="data_source/biomart_test.xml" /> <tool file="data_source/gramene_mart.xml" /> + <tool file="data_source/flymine.xml" /> + <tool file="data_source/flymine_test.xml" /> + <tool file="data_source/modmine.xml" /> + <tool file="data_source/ratmine.xml" /> <tool file="data_source/wormbase.xml" /> <tool file="data_source/wormbase_test.xml" /> - <tool file="data_source/flymine.xml" /> - <tool file="data_source/flymine_test.xml" /> <tool file="data_source/eupathdb.xml" /> <tool file="data_source/encode_db.xml" /> <tool file="data_source/epigraph_import.xml" /> diff -r 7aaa791a6283 -r 29b6fac01d78 tools/data_source/modmine.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/data_source/modmine.xml Thu Dec 03 15:33:15 2009 -0500 @@ -0,0 +1,30 @@ +<?xml version="1.0"?> + +<tool name="modMine" id="modmine" tool_type="data_source" URL_method="post"> + <description>server</description> + <command interpreter="python">data_source.py $output $__app__.config.output_size_limit</command> + <inputs action="http://intermine.modencode.org/" check_values="false" method="get" target="_top"> + <display>go to modMine server $GALAXY_URL</display> + <param name="GALAXY_URL" type="baseurl" value="/tool_runner?tool_id=modmine" /> + </inputs> + <request_param_translation> + <request_param galaxy_name="URL" remote_name="URL" missing="" /> + <request_param galaxy_name="dbkey" remote_name="db" missing="?" /> + <request_param galaxy_name="organism" remote_name="organism" missing="" /> + <request_param galaxy_name="table" remote_name="table" missing="" /> + <request_param galaxy_name="description" remote_name="description" missing="" /> + <request_param galaxy_name="name" remote_name="name" missing="modMine query" /> + <request_param galaxy_name="info" remote_name="info" missing="" /> + <request_param galaxy_name="data_type" remote_name="data_type" missing="txt" /> + </request_param_translation> + <uihints minwidth="800"/> + <outputs> + <data name="output" format="txt" /> + </outputs> + <options sanitize="False" refresh="True"/> +</tool> + diff -r 7aaa791a6283 -r 29b6fac01d78 tools/data_source/ratmine.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/data_source/ratmine.xml Thu Dec 03 15:33:15 2009 -0500 @@ -0,0 +1,29 @@ +<?xml version="1.0"?> + +<tool name="Ratmine" id="ratmine" tool_type="data_source" URL_method="post"> + <description>server</description> + <command interpreter="python">data_source.py $output $__app__.config.output_size_limit</command> + <inputs action="http://ratmine.mcw.edu/ratmine/begin.do" check_values="false" method="get" target="_top"> + <display>go to Ratmine server $GALAXY_URL</display> + <param name="GALAXY_URL" type="baseurl" value="/tool_runner?tool_id=ratmine" /> + </inputs> + <request_param_translation> + <request_param galaxy_name="URL" remote_name="URL" missing="" /> + <request_param galaxy_name="dbkey" remote_name="db" missing="?" /> + <request_param galaxy_name="organism" remote_name="organism" missing="" /> + <request_param galaxy_name="table" remote_name="table" missing="" /> + <request_param galaxy_name="description" remote_name="description" missing="" /> + <request_param galaxy_name="name" remote_name="name" missing="Ratmine query" /> + <request_param galaxy_name="info" remote_name="info" missing="" /> + <request_param galaxy_name="data_type" remote_name="data_type" missing="txt" /> + </request_param_translation> + <uihints minwidth="800"/> + <outputs> + <data name="output" format="txt" /> + </outputs> + <options sanitize="False" refresh="True"/> +</tool>

1 0

[hg] galaxy 3142: Added ability for a user to start 64bit cloud ...
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/b636ff9d0f05 changeset: 3142:b636ff9d0f05 user: Enis Afgan <afgane(a)gmail.com> date: Thu Dec 03 14:16:50 2009 -0500 description: Added ability for a user to start 64bit cloud instances. diffstat: lib/galaxy/cloud/__init__.py | 1 + lib/galaxy/cloud/providers/ec2.py | 9 +++++---- templates/cloud/configure_cloud.mako | 5 +++++ 3 files changed, 11 insertions(+), 4 deletions(-) diffs (45 lines): diff -r 5e30e1ea6bbb -r b636ff9d0f05 lib/galaxy/cloud/__init__.py --- a/lib/galaxy/cloud/__init__.py Wed Dec 02 20:02:05 2009 -0500 +++ b/lib/galaxy/cloud/__init__.py Thu Dec 03 14:16:50 2009 -0500 @@ -397,6 +397,7 @@ return None store.error = error + store.status = store_status.ERROR self.sa_session.add( store ) self.sa_session.flush() diff -r 5e30e1ea6bbb -r b636ff9d0f05 lib/galaxy/cloud/providers/ec2.py --- a/lib/galaxy/cloud/providers/ec2.py Wed Dec 02 20:02:05 2009 -0500 +++ b/lib/galaxy/cloud/providers/ec2.py Thu Dec 03 14:16:50 2009 -0500 @@ -318,10 +318,11 @@ if count == len( vl ): uci_wrapper.set_deleted() else: - err = "Deleting following volume(s) failed: " + str( failedList ) + ". However, these volumes were successfully deleted: " \ - + str( deletedList ) + ". MANUAL intervention and processing needed." - log.error( err ) - uci_wrapper.set_error( err, True ) + if uci_wrapper.get_uci_state != uci_states.ERROR: + err = "Deleting following volume(s) failed: " + str( failedList ) + ". However, these volumes were successfully deleted: " \ + + str( deletedList ) + ". MANUAL intervention and processing needed." + log.error( err ) + uci_wrapper.set_error( err, True ) def snapshot_uci( self, uci_wrapper ): """ diff -r 5e30e1ea6bbb -r b636ff9d0f05 templates/cloud/configure_cloud.mako --- a/templates/cloud/configure_cloud.mako Wed Dec 02 20:02:05 2009 -0500 +++ b/templates/cloud/configure_cloud.mako Thu Dec 03 14:16:50 2009 -0500 @@ -321,6 +321,11 @@ <a class="action-button" href="${h.url_for( action='rename_uci', id=trans.security.encode_id(prevInstance.id) )}">Rename</a> <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='m1.small' )}"> Start m1.small</a> <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='c1.medium' )}"> Start c1.medium</a> + <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='m1.large' )}"> Start m1.large</a> + <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='m1.xlarge' )}"> Start m1.xlarge</a> + <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='c1.xlarge' )}"> Start c1.xlarge</a> + <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='m2.2xlarge' )}"> Start m2.2xlarge</a> + <a class="action-button" href="${h.url_for( action='start', id=trans.security.encode_id(prevInstance.id), type='m2.4xlarge' )}"> Start m2.4xlarge</a> <a class="action-button" href="${h.url_for( action='create_snapshot', id=trans.security.encode_id(prevInstance.id) )}">Create snapshot</a> <a class="action-button" href="${h.url_for( action='view_snapshots', id=trans.security.encode_id(prevInstance.id) )}">View snapshots</a> <a class="action-button" confirm="Are you sure you want to delete instance '${prevInstance.name}'? This will delete all of your data assocaiated with this instance!" href="${h.url_for( action='delete_uci', id=trans.security.encode_id(prevInstance.id) )}">Delete</a>

1 0

[hg] galaxy 3141: Add support for uploading BAM files.
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/5e30e1ea6bbb changeset: 3141:5e30e1ea6bbb user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Dec 02 20:02:05 2009 -0500 description: Add support for uploading BAM files. diffstat: datatypes_conf.xml.sample | 3 +- lib/galaxy/datatypes/binary.py | 55 ++++++++++++++++----------- lib/galaxy/datatypes/registry.py | 1 + lib/galaxy/datatypes/sniff.py | 3 + lib/galaxy/datatypes/test/1.bam | test-data/1.bam | test/functional/test_get_data.py | 18 ++++++++- tools/data_source/upload.xml | 6 +++ 8 files changed, 62 insertions(+), 24 deletions(-) diffs (183 lines): diff -r f4e6ed22a39e -r 5e30e1ea6bbb datatypes_conf.xml.sample --- a/datatypes_conf.xml.sample Wed Dec 02 15:23:15 2009 -0500 +++ b/datatypes_conf.xml.sample Wed Dec 02 20:02:05 2009 -0500 @@ -3,7 +3,7 @@ <registration converters_path="lib/galaxy/datatypes/converters"> <datatype extension="ab1" type="galaxy.datatypes.binary:Ab1" mimetype="application/octet-stream" display_in_upload="true"/> <datatype extension="axt" type="galaxy.datatypes.sequence:Axt" display_in_upload="true"/> - <datatype extension="bam" type="galaxy.datatypes.binary:Bam" mimetype="application/octet-stream"/> + <datatype extension="bam" type="galaxy.datatypes.binary:Bam" mimetype="application/octet-stream" display_in_upload="true"/> <datatype extension="bed" type="galaxy.datatypes.interval:Bed" display_in_upload="true"> <converter file="bed_to_gff_converter.xml" target_datatype="gff"/> <converter file="interval_to_coverage.xml" target_datatype="coverage"/> @@ -203,6 +203,7 @@ defined format first, followed by next-most rigidly defined, and so on. --> + <sniffer type="galaxy.datatypes.binary:Bam"/> <sniffer type="galaxy.datatypes.binary:Sff"/> <sniffer type="galaxy.datatypes.xml:BlastXml"/> <sniffer type="galaxy.datatypes.sequence:Maf"/> diff -r f4e6ed22a39e -r 5e30e1ea6bbb lib/galaxy/datatypes/binary.py --- a/lib/galaxy/datatypes/binary.py Wed Dec 02 15:23:15 2009 -0500 +++ b/lib/galaxy/datatypes/binary.py Wed Dec 02 20:02:05 2009 -0500 @@ -12,7 +12,7 @@ log = logging.getLogger(__name__) -sniffable_binary_formats = [ 'sff' ] +sniffable_binary_formats = [ 'sff', 'bam' ] # Currently these supported binary data types must be manually set on upload unsniffable_binary_formats = [ 'ab1', 'scf' ] @@ -26,6 +26,9 @@ else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' + def get_mime( self ): + """Returns the mime type of the datatype""" + return 'application/octet-stream' class Ab1( Binary ): """Class describing an ab1 binary sequence file""" @@ -48,29 +51,40 @@ """Class describing a BAM binary file""" file_ext = "bam" MetadataElement( name="bam_index", desc="BAM Index File", param=metadata.FileParameter, readonly=True, no_value=None, visible=False, optional=True ) + def init_meta( self, dataset, copy_from=None ): Binary.init_meta( self, dataset, copy_from=copy_from ) - def set_meta( self, dataset, overwrite = True, **kwd ): """ - Sets index for BAM file. + GVK 12/2/09: just noticed this - not good and doesn't work, so commenting out for now. + def set_meta( self, dataset, overwrite = True, **kwd ): + # Sets index for BAM file. + index_file = dataset.metadata.bam_index + if not index_file: + index_file = dataset.metadata.spec['bam_index'].param.new_file( dataset = dataset ) + tmp_dir = tempfile.gettempdir() + tmpf1 = tempfile.NamedTemporaryFile( dir=tmp_dir ) + tmpf1bai = '%s.bai' % tmpf1.name + try: + os.system( 'cd %s' % tmp_dir ) + os.system( 'cp %s %s' % ( dataset.file_name, tmpf1.name ) ) + os.system( 'samtools index %s' % tmpf1.name ) + os.system( 'cp %s %s' % ( tmpf1bai, index_file.file_name ) ) + except Exception, ex: + sys.stderr.write( 'There was a problem creating the index for the BAM file\n%s\n' + str( ex ) ) + tmpf1.close() + if os.path.exists( tmpf1bai ): + os.remove( tmpf1bai ) + dataset.metadata.bam_index = index_file """ - index_file = dataset.metadata.bam_index - if not index_file: - index_file = dataset.metadata.spec['bam_index'].param.new_file( dataset = dataset ) - tmp_dir = tempfile.gettempdir() - tmpf1 = tempfile.NamedTemporaryFile( dir=tmp_dir ) - tmpf1bai = '%s.bai' % tmpf1.name + def sniff( self, filename ): + # The first 4 bytes of any bam file is 'BAM\1', and the file is binary. try: - os.system( 'cd %s' % tmp_dir ) - os.system( 'cp %s %s' % ( dataset.file_name, tmpf1.name ) ) - os.system( 'samtools index %s' % tmpf1.name ) - os.system( 'cp %s %s' % ( tmpf1bai, index_file.file_name ) ) - except Exception, ex: - sys.stderr.write( 'There was a problem creating the index for the BAM file\n%s\n' + str( ex ) ) - tmpf1.close() - if os.path.exists( tmpf1bai ): - os.remove( tmpf1bai ) - dataset.metadata.bam_index = index_file + header = open( filename ).read(4) + if binascii.b2a_hex( header ) == binascii.hexlify( 'BAM\1' ): + return True + return False + except: + return False def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: export_url = "/history_add_to?" + urlencode( {'history_id':dataset.history_id,'ext':'bam','name':'bam alignments','info':'Alignments file','dbkey':dataset.dbkey} ) @@ -84,9 +98,6 @@ return dataset.peek except: return "Binary bam alignments file (%s)" % ( data.nice_size( dataset.get_size() ) ) - def get_mime( self ): - """Returns the mime type of the datatype""" - return 'application/octet-stream' class Binseq( Binary ): """Class describing a zip archive of binary sequence files""" diff -r f4e6ed22a39e -r 5e30e1ea6bbb lib/galaxy/datatypes/registry.py --- a/lib/galaxy/datatypes/registry.py Wed Dec 02 15:23:15 2009 -0500 +++ b/lib/galaxy/datatypes/registry.py Wed Dec 02 20:02:05 2009 -0500 @@ -174,6 +174,7 @@ # because some formats are much more flexibly defined than others. if len(self.sniff_order) < 1: self.sniff_order = [ + binary.Bam(), binary.Sff(), xml.BlastXml(), sequence.Maf(), diff -r f4e6ed22a39e -r 5e30e1ea6bbb lib/galaxy/datatypes/sniff.py --- a/lib/galaxy/datatypes/sniff.py Wed Dec 02 15:23:15 2009 -0500 +++ b/lib/galaxy/datatypes/sniff.py Wed Dec 02 20:02:05 2009 -0500 @@ -252,6 +252,9 @@ >>> fname = get_test_fname('1.sff') >>> guess_ext(fname) 'sff' + >>> fname = get_test_fname('1.bam') + >>> guess_ext(fname) + 'bam' """ if sniff_order is None: datatypes_registry = registry.Registry() diff -r f4e6ed22a39e -r 5e30e1ea6bbb lib/galaxy/datatypes/test/1.bam Binary file lib/galaxy/datatypes/test/1.bam has changed diff -r f4e6ed22a39e -r 5e30e1ea6bbb test-data/1.bam Binary file test-data/1.bam has changed diff -r f4e6ed22a39e -r 5e30e1ea6bbb test/functional/test_get_data.py --- a/test/functional/test_get_data.py Wed Dec 02 15:23:15 2009 -0500 +++ b/test/functional/test_get_data.py Wed Dec 02 20:02:05 2009 -0500 @@ -520,7 +520,23 @@ self.check_history_for_string( '1.axt format: axt, database: \? Info: uploaded file' ) self.check_metadata_for_string( 'value="1.axt" value="\?" Change data type selected value="axt" selected="yes"' ) self.delete_history( id=self.security.encode_id( history.id ) ) - def test_0150_url_paste( self ): + def test_0150_upload_file( self ): + """Test uploading 1.bam, NOT setting the file format""" + self.check_history_for_string( 'Your history is empty' ) + history = sa_session.query( galaxy.model.History ) \ + .filter( and_( galaxy.model.History.table.c.deleted==False, + galaxy.model.History.table.c.user_id==admin_user.id ) ) \ + .order_by( desc( galaxy.model.History.table.c.create_time ) ) \ + .first() + self.upload_file( '1.bam' ) + hda = sa_session.query( galaxy.model.HistoryDatasetAssociation ) \ + .order_by( desc( galaxy.model.HistoryDatasetAssociation.table.c.create_time ) ) \ + .first() + assert hda is not None, "Problem retrieving hda from database" + self.verify_dataset_correctness( '1.bam', hid=str( hda.hid ) ) + self.check_history_for_string( 'bam' ) + self.delete_history( id=self.security.encode_id( history.id ) ) + def test_0155_url_paste( self ): """Test url paste behavior""" # Logged in as admin_user # Deleting the current history should have created a new history diff -r f4e6ed22a39e -r 5e30e1ea6bbb tools/data_source/upload.xml --- a/tools/data_source/upload.xml Wed Dec 02 15:23:15 2009 -0500 +++ b/tools/data_source/upload.xml Wed Dec 02 20:02:05 2009 -0500 @@ -57,6 +57,12 @@ ----- +**Bam** + +A binary file compressed in the BGZF format with a '.bam' file extension. + +----- + **Binseq.zip** A zipped archive consisting of binary sequence files in either 'ab1' or 'scf' format. All files in this archive must have the same file extension which is one of '.ab1' or '.scf'. You must manually select this 'File Format' when uploading the file.

1 0

[hg] galaxy 3143: Bug fixes for 'insert history' operation in pa...
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/7aaa791a6283 changeset: 3143:7aaa791a6283 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Thu Dec 03 14:39:21 2009 -0500 description: Bug fixes for 'insert history' operation in pages. diffstat: lib/galaxy/web/controllers/history.py | 8 +++++--- templates/page/editor.mako | 28 +++++++++++++++++----------- 2 files changed, 22 insertions(+), 14 deletions(-) diffs (65 lines): diff -r b636ff9d0f05 -r 7aaa791a6283 lib/galaxy/web/controllers/history.py --- a/lib/galaxy/web/controllers/history.py Thu Dec 03 14:16:50 2009 -0500 +++ b/lib/galaxy/web/controllers/history.py Thu Dec 03 14:39:21 2009 -0500 @@ -387,12 +387,14 @@ def set_importable_async( self, trans, id=None, importable=False ): """ Set history's importable attribute. """ history = get_history( trans, id, True ) - - if history: + + # Only set if importable value would change; this prevents a change in the update_time unless attribute really changed. + importable = importable in ['True', 'true', 't']; + if history and history.importable != importable: history.importable = importable trans.sa_session.flush() - return + return result @web.expose def name_autocomplete_data( self, trans, q=None, limit=None, timestamp=None ): diff -r b636ff9d0f05 -r 7aaa791a6283 templates/page/editor.mako --- a/templates/page/editor.mako Thu Dec 03 14:16:50 2009 -0500 +++ b/templates/page/editor.mako Thu Dec 03 14:39:21 2009 -0500 @@ -232,23 +232,29 @@ // Insert link. wym._exec(WYMeditor.CREATE_LINK, sStamp); - if ( $("a[href=" + sStamp + "]", wym._doc.body).length != 0) + var link_text = $("a[href=" + sStamp + "]", wym._doc.body).text(); + if ( + link_text == "" // Firefox. + || + link_text == sStamp // Safari + ) + { + // User selected no text; create link from scratch and use default text. + + // Get history name. + $.get( '${h.url_for( controller='history', action='get_name_async' )}?id=' + item_id, function( history_name ) { + var href = '${h.url_for( controller='history', action='view' )}?id=' + item_id; + wym.insert("<a href='" + href + "'>History '" + history_name + "'</a>nbsp;"); + }); + } + else { // Link created from selected text; add href and title. $("a[href=" + sStamp + "]", wym._doc.body) .attr(WYMeditor.HREF, '${h.url_for( controller='history', action='view' )}' + '?id=' + item_id) .attr(WYMeditor.TITLE, "History" + item_id); } - else - { - // User selected no text; create link from scratch and use default text. - - // Get history name. - $.get( '${h.url_for( controller='history', action='get_name_async' )}?id=' + item_id, function( history_name ) { - var href = '${h.url_for( controller='history', action='view' )}?id=' + item_id; - wym.insert("<a href='" + href + "'>History '" + history_name + "'</a>"); - }); - } + }); hide_modal();

1 0

[hg] galaxy 3138: Bug fix - add a needed import to ~/datatypes/d...
by Greg Von Kuster 04 Dec '09

04 Dec '09

details: http://www.bx.psu.edu/hg/galaxy/rev/661ed3a56a4d changeset: 3138:661ed3a56a4d user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Dec 02 10:07:21 2009 -0500 description: Bug fix - add a needed import to ~/datatypes/data.py. diffstat: lib/galaxy/datatypes/data.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r ddd9d5ac0457 -r 661ed3a56a4d lib/galaxy/datatypes/data.py --- a/lib/galaxy/datatypes/data.py Tue Dec 01 20:16:41 2009 -0500 +++ b/lib/galaxy/datatypes/data.py Wed Dec 02 10:07:21 2009 -0500 @@ -4,6 +4,7 @@ from galaxy.util.bunch import Bunch from cgi import escape import metadata +import zipfile from metadata import MetadataElement #import directly to maintain ease of use in Datatype class definitions log = logging.getLogger(__name__)

1 0