- galaxy-dev - lists.galaxyproject.org

[hg] galaxy 2834: Allow root/my_data to take but ignore multiple...
by Greg Von Kuster 09 Oct '09

09 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/b14f99a4f736 changeset: 2834:b14f99a4f736 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Tue Oct 06 15:00:32 2009 -0400 description: Allow root/my_data to take but ignore multiple parameters in order to support route memory. 1 file(s) affected in this change: lib/galaxy/web/controllers/root.py diffs (12 lines): diff -r 40ab85150694 -r b14f99a4f736 lib/galaxy/web/controllers/root.py --- a/lib/galaxy/web/controllers/root.py Tue Oct 06 14:13:31 2009 -0400 +++ b/lib/galaxy/web/controllers/root.py Tue Oct 06 15:00:32 2009 -0400 @@ -48,7 +48,7 @@ ## ---- Root history display --------------------------------------------- @web.expose - def my_data( self, trans ): + def my_data( self, trans, **kwd ): """ Display user's data. """

1 0

Metadata changes in workflows
by Assaf Gordon 09 Oct '09

09 Oct '09

Hello, While browsing through Galaxy's wiki (where's the "search" button, BTW??), I noticed this issue: http://bitbucket.org/galaxy/galaxy-central/issue/34/metadata-handling-in-wo… We have two tools that do exactly that (allow users to change metadata columns and dbkey), so these can be changed during a workflow (goes great with AWK or CUT tools which changes the columns, or when users want to run mapping on library as two or more organisms). I'm attaching the tools and you're welcomed to modify and use them. -gordon

1 0

New tools: bigWig & bigBed
by Assaf Gordon 08 Oct '09

08 Oct '09

Hello, I'd like to suggest couple of tools to handle bigBed and bigWig formats. The bigBed and BigWig formats are new to the UCSC genome browser, and allow efficient uploading of user tracks. They are efficient not only because they are binary, but also because they don't need to be uploaded to the genome browser itself - the binary format support partial HTTP gets, so the bigWig/bigWig file can stay on your server, and the genome browser fetches only the relevant portion of the file (for the current viewing coordinates). The tools are: bedToBigBed bedGraphtoBigWig WigToBigWig BigBedtoBed bigWigToBedGraph bedClip I provide only the wrappers, the tools themselves are available for download from the UCSC genome browser. You can see example of the running tools here: http://cancan.cshl.edu/publicgalaxy/ (Bottom category - "bigBed/bigWig tools") You can download the XML wrappers from here: http://cancan.cshl.edu/labmembers/gordon/galaxy_bigbedwig_tools.tar.bz2 I'd like to point three advantages to the bigWig/bigBed formats: 1. They go hand-in-hand with next-gen sequencing files: A user uploads a huge FASTQ file, maps it with BWA/Bowtie (or other) and wants to visualize the results. Uploading the resulting textual BED/WIG file is too wasteful. 2. With some minor Galaxy modifications, adding custom tracks to the Genome Browser can be done easily, even faster than clicking on the "main" link of a BED file: Add to the dataset controller a method that returns something like: track name="User Track" type=bigWig bigDataURL=http://main.g2.bx.psu.edu/datasets/837890/display/index And point the genome browser to this URL - it will take the data directly from Galaxy - no need to upload anything. <Shameless plug> 3. The BedGraph format (which consists of chrom,start,end,value) is exactly what the 'fastx-collapser' was written for: assigning a multiplicity count for an interval... </Shameless Plug> But many other mappers/aligners programs also provided their own "collapsed" or multiplicity-value for intervals - this makes presenting data one the genome browser very easy. More information about bigWig & bigBed: http://genome.ucsc.edu/goldenPath/help/bigBed.html http://genome.ucsc.edu/goldenPath/help/bigWig.html Comments are welcomed, gordon.

2 1

Issue with sam_fa_indices.loc.sample
by Tam, Oliver 07 Oct '09

07 Oct '09

Hi, This message is long. If you wish to see how I resolved the issue, please scroll down to ³SOLUTION². I have been trying to set up the SAMTools program on Galaxy in my local instance, and I was getting stuck with the SAM-to-BAM tool. Looking at the XML file, it tells me that it requires the ³sam_fa_indices.loc² file. After copying the sam_fa_indices.loc.sample to make the required file, I then proceeded to put in the index. However, I found the instructions slightly confusing. ---Instruction at the top--- #This is a sample file distributed with Galaxy that enables tools #to use a directory of Samtools indexed sequences data files. You will need #to create these data files and then create a sam_fa_indices.loc file #similar to this one (store it in this directory ) that points to #the directories in which those files are stored. The sam_fa_indices.loc #file has this format (white space characters are TAB characters): # #<index> <seq> <location> This implies that it requires three columns, one with a variable that defines the index, one with a variable that defines the seq, and one that defines the location of the index file. However, in the example beneath, this is not the format that the line entry is given. ---Example given--- #So, for example, if you had hg18 indexed stored in #/depot/data2/galaxy/sam/, #then the sam_fa_indices.loc entry would look like this: # #hg18 /depot/data2/galaxy/sam/hg18.fa This entry has only two columns, it was not clear if hg18 is the index variable, and if the seq variable refers to the sequence file that was used to generate the index, or if it refers the path (containing the prefix) to find the index file. When I used the two-column entry as in the example, the tool failed, informing me that the sequence is not found. The path is correct, and the file was independently generated by SAMTools. Upon looking at the xml file, I noticed the following validator code: <validator type="dataset_metadata_in_file" filename="sam_fa_indices.loc" metadata_name="dbkey" metadata_column="1" message="Sequences are not currently available for the specified build." line_startswith="index" /> Two things popped up from reading this: 1. The ³metadata_column = 1², which given that column assignment is 0-based, implies that the dbkey variable should match column 2 of the sam_fa_indices.loc file. 2. The ³line_startswith=index², which led me to believe that every line must start with index. >>>>>SOLUTION<<<<< In sam_fa_indices.loc.sample: #This is a sample file distributed with Galaxy that enables tools #to use a directory of Samtools indexed sequences data files. You will need #to create these data files and then create a sam_fa_indices.loc file #similar to this one (store it in this directory ) that points to #the directories in which those files are stored. The sam_fa_indices.loc #file has this format (white space characters are TAB characters): # #index <seq> <location> [Change <index> to the word ³index²] #So, for example, if you had hg18 indexed stored in #/depot/data2/galaxy/sam/, #then the sam_fa_indices.loc entry would look like this: # #index hg18 /depot/data2/galaxy/sam/hg18.fa [Add the word ³index² to the beginning of line] # #and your /depot/data2/galaxy/sam/ directory #would contain hg18.fa and hg18.fa.fai files: # #-rw-r--r-- 1 james universe 830134 2005-09-13 10:12 hg18.fa #-rw-r--r-- 1 james universe 527388 2005-09-13 10:12 hg18.fa.fai # #Your sam_fa_indices.loc file should include an entry per line for #each index set you have stored. The file in the path does actually #exist, but it should never be directly used. Instead, the name serves #as a prefix for the index file. For example: # #index hg18 /depot/data2/galaxy/sam/hg18.fa [Add the word ³index² to the beginning of line] Once I made these changes, the tool now works. >>>>>END SOLUTION<<<<< Thanks for your time and patience. Cheers, Oliver

2 1

Feature request: counting job queue time
by Gordon, Assaf 07 Oct '09

07 Oct '09

Hello, I have a small feature request: I'd like to be able to measure the time jobs wait in "queued" state - that is - how much time passed between the job's creation and the start of execution. The reason is that I'm testing a move from the local runner to an SGE runner. I'm sure the local runner is much faster in executing (assuming there are free job threads), but the SGE runner will eventually scale better. My main concern is that because my Galaxy will compete for the SGE cluster with other users, the advantages of using the cluster will be outweighed by the wasted time the SGE jobs will be queued and not executed. Thanks, -gordon

2 1

tool_conf.xml question
by Assaf Gordon 07 Oct '09

07 Oct '09

Hello, a small question regarding 'tool_conf.xml' : Is it possible to see the tool_conf.xml that you use for the main galaxy ? it is not "tool_conf.xml.main" (which is missing the NGS tools), and not exactly "tool_conf.xml.sample" (which has different ordering, and also contains references to RGenetics tools). Since it is not monitored in the mercurial repository, Cloning a fresh galaxy copy and running "sh setup.sh" does not reproduce the same tools and categories as your main web site. Also, The "tool_conf.xml.sample" contains "CutWrapper.xml" twice, and, Is it possible to have the RGenetics XMLs (and tools) ? They are referenced in the tool_conf.xml.sample, but do not exist in the repository. My goal is to have my local galaxy look as closely as possible to the public one, so people would be comfortable going from one to the other (and vice versa). Thanks! -gordon.

2 1

[hg] galaxy 2829: Added method get_display_name() to History and...
by Greg Von Kuster 06 Oct '09

06 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e6324f395d26 changeset: 2829:e6324f395d26 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Mon Oct 05 17:38:54 2009 -0400 description: Added method get_display_name() to History and HistoryDatasetAssociation objects; these methods return unicode objects that can be used to display non-ascii characters. Fixes #181 4 file(s) affected in this change: lib/galaxy/model/__init__.py lib/galaxy/web/controllers/history.py templates/dataset/edit_attributes.mako templates/root/history.mako diffs (100 lines): diff -r f648d6dec544 -r e6324f395d26 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py Mon Oct 05 15:07:20 2009 -0400 +++ b/lib/galaxy/model/__init__.py Mon Oct 05 17:38:54 2009 -0400 @@ -245,6 +245,12 @@ def activatable_datasets( self ): # This needs to be a list return [ hda for hda in self.datasets if not hda.dataset.deleted ] + def get_display_name( self ): + ## History name can be either a string or a unicode object. If string, convert to unicode object assuming 'utf-8' format. + history_name = self.name + if isinstance(history_name, str): + history_name = unicode(history_name, 'utf-8') + return history_name class HistoryUserShareAssociation( object ): def __init__( self ): @@ -682,6 +688,13 @@ for assoc in self.implicitly_converted_datasets: if not metadata_safe or not assoc.metadata_safe: assoc.clear( purge = purge ) + def get_display_name( self ): + ## Name can be either a string or a unicode object. If string, convert to unicode object assuming 'utf-8' format. + hda_name = self.name + if isinstance(hda_name, str): + hda_name = unicode(hda_name, 'utf-8') + return hda_name + class HistoryDatasetAssociationDisplayAtAuthorization( object ): def __init__( self, hda=None, user=None, site=None ): diff -r f648d6dec544 -r e6324f395d26 lib/galaxy/web/controllers/history.py --- a/lib/galaxy/web/controllers/history.py Mon Oct 05 15:07:20 2009 -0400 +++ b/lib/galaxy/web/controllers/history.py Mon Oct 05 17:38:54 2009 -0400 @@ -18,6 +18,13 @@ class HistoryListGrid( grids.Grid ): # Custom column types + class NameColumn( grids.GridColumn ): + def __init( self, key, link, attach_popup ): + grids.GridColumn.__init__(self, key, link, attach_popup) + + def get_value( self, trans, grid, history ): + return history.get_display_name() + class DatasetsByStateColumn( grids.GridColumn ): def get_value( self, trans, grid, history ): rval = [] @@ -42,7 +49,7 @@ return dict( operation="sharing" ) return None class TagsColumn( grids.GridColumn ): - def __init__(self, col_name, key, filterable): + def __init__( self, col_name, key, filterable ): grids.GridColumn.__init__(self, col_name, key=key, filterable=filterable) # Tags cannot be sorted. self.sortable = False @@ -98,8 +105,8 @@ template='/history/grid.mako' default_sort_key = "-create_time" columns = [ - grids.GridColumn( "Name", key="name", - link=( lambda item: iff( item.deleted, None, dict( operation="switch", id=item.id ) ) ), + NameColumn( "Name", key="name", + link=( lambda history: iff( history.deleted, None, dict( operation="switch", id=history.id ) ) ), attach_popup=True ), DatasetsByStateColumn( "Datasets (by state)", ncells=4 ), TagsColumn( "Tags", key="tags", filterable=True), diff -r f648d6dec544 -r e6324f395d26 templates/dataset/edit_attributes.mako --- a/templates/dataset/edit_attributes.mako Mon Oct 05 15:07:20 2009 -0400 +++ b/templates/dataset/edit_attributes.mako Mon Oct 05 17:38:54 2009 -0400 @@ -35,7 +35,7 @@ Name: </label> <div style="float: left; width: 250px; margin-right: 10px;"> - <input type="text" name="name" value="${data.name}" size="40"/> + <input type="text" name="name" value="${data.get_display_name()}" size="40"/> </div> <div style="clear: both"></div> </div> @@ -155,7 +155,7 @@ %if trans.app.security_agent.can_manage_dataset( user_roles, data.dataset ): <%namespace file="/dataset/security_common.mako" import="render_permission_form" /> - ${render_permission_form( data.dataset, data.name, h.url_for( controller='root', action='edit', id=data.id ), user_roles )} + ${render_permission_form( data.dataset, data.get_display_name(), h.url_for( controller='root', action='edit', id=data.id ), user_roles )} %elif trans.user: <div class="toolForm"> <div class="toolFormTitle">View Permissions</div> diff -r f648d6dec544 -r e6324f395d26 templates/root/history.mako --- a/templates/root/history.mako Mon Oct 05 15:07:20 2009 -0400 +++ b/templates/root/history.mako Mon Oct 05 17:38:54 2009 -0400 @@ -290,7 +290,7 @@ <div id="history-name-area" class="historyLinks" style="color: gray; font-weight: bold;"> <div style="float: right"><a id="history-rename" title="Rename" class="icon-button edit" target="galaxy_main" href="${h.url_for( controller='history', action='rename' )}"></a></div> - <div id="history-name">${history.name}</div> + <div id="history-name">${history.get_display_name()}</div> </div> %if history.deleted:

1 0

[hg] galaxy 2831: bug fix for pilup_parser reported by user
by Greg Von Kuster 06 Oct '09

06 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/8f87205219ef changeset: 2831:8f87205219ef user: Anton Nekrutenko <anton(a)bx.psu.edu> date: Tue Oct 06 09:44:34 2009 -0400 description: bug fix for pilup_parser reported by user 1 file(s) affected in this change: tools/samtools/pileup_parser.pl diffs (35 lines): diff -r 50dbd8907d88 -r 8f87205219ef tools/samtools/pileup_parser.pl --- a/tools/samtools/pileup_parser.pl Mon Oct 05 17:39:29 2009 -0400 +++ b/tools/samtools/pileup_parser.pl Tue Oct 06 09:44:34 2009 -0400 @@ -37,17 +37,19 @@ die "Coverage column" . ($cvrg_column+1) . " contains non-numeric values. Check your input parameters as well as format of input dataset." if ( not isdigit $fields[ $cvrg_column ] ); next if $fields[ $cvrg_column ] < $cvrg_cutoff; my $base_quality = $fields[ $base_quality_column ]; - if ($read_bases =~ m/[\$\^\+-]/) { + $read_bases =~ s/\^.//g; #removing the start of the read segement mark $read_bases =~ s/\$//g; #removing end of the read segment mark - $read_bases =~ s/\^.//g; #removing the start of the read segement mark while ($read_bases =~ m/[\+-]{1}(\d+)/g) { my $indel_len = $1; $read_bases =~ s/[\+-]{1}$indel_len.{$indel_len}//; # remove indel info from read base field } } - die "Error parsing read bases and qualities in line $.. Last processed line conatined these values: " . join("\t", @fields) . "\n" if ( length($read_bases) != length($base_quality) ); - + if ( length($read_bases) != length($base_quality) ) { + $first_skipped_line = $. if $first_skipped_line eq ""; + ++$invalid_line_counter; + next; + } # after removing read block and indel data the length of read_base # field should identical to the length of base_quality field @@ -94,6 +96,6 @@ } -print STDERR "Could not parse $invalid_line_counter line(s) beginning with: $first_skipped_line\n" if $invalid_line_counter > 0; +print "Skipped $invalid_line_counter invalid line(s) beginning with line $first_skipped_line\n" if $invalid_line_counter > 0; close IN; close OUT;

1 0

[hg] galaxy 2832: small tweaks to bowtie & bwa interfaces
by Greg Von Kuster 06 Oct '09

06 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/a7a7376cf8e9 changeset: 2832:a7a7376cf8e9 user: Anton Nekrutenko <anton(a)bx.psu.edu> date: Tue Oct 06 13:20:24 2009 -0400 description: small tweaks to bowtie & bwa interfaces 2 file(s) affected in this change: tools/sr_mapping/bowtie_wrapper.xml tools/sr_mapping/bwa_wrapper.xml diffs (28 lines): diff -r 8f87205219ef -r a7a7376cf8e9 tools/sr_mapping/bowtie_wrapper.xml --- a/tools/sr_mapping/bowtie_wrapper.xml Tue Oct 06 09:44:34 2009 -0400 +++ b/tools/sr_mapping/bowtie_wrapper.xml Tue Oct 06 13:20:24 2009 -0400 @@ -158,9 +158,9 @@ <when value="history"> <param name="ownFile" type="data" format="fasta" metadata_name="dbkey" label="Select a reference genome" /> <conditional name="indexParams"> - <param name="index_settings" type="select" label="Choose whether to use default options or to set your own"> - <option value="index_pre_set">Commonly Used</option> - <option value="index_full">Full Parameter List</option> + <param name="index_settings" type="select" label="Choose whether to use default options for building indices or to set your own"> + <option value="index_pre_set">Default</option> + <option value="index_full">Set your own</option> </param> <when value="index_pre_set" /> <when value="index_full"> diff -r 8f87205219ef -r a7a7376cf8e9 tools/sr_mapping/bwa_wrapper.xml --- a/tools/sr_mapping/bwa_wrapper.xml Tue Oct 06 09:44:34 2009 -0400 +++ b/tools/sr_mapping/bwa_wrapper.xml Tue Oct 06 13:20:24 2009 -0400 @@ -103,7 +103,7 @@ </param> </when> <when value="history"> - <param name="ownFile" type="data" format="fasta" metadata_name="dbkey" label="Select a reference genome" /> + <param name="ownFile" type="data" format="fasta" metadata_name="dbkey" label="Select a reference from history" /> </when> </conditional> </when>

1 0

[hg] galaxy 2830: merge
by Greg Von Kuster 06 Oct '09

06 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/50dbd8907d88 changeset: 2830:50dbd8907d88 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Mon Oct 05 17:39:29 2009 -0400 description: merge 0 file(s) affected in this change: diffs (35 lines): diff -r e6324f395d26 -r 50dbd8907d88 tools/filters/compare.xml --- a/tools/filters/compare.xml Mon Oct 05 17:38:54 2009 -0400 +++ b/tools/filters/compare.xml Mon Oct 05 17:39:29 2009 -0400 @@ -1,4 +1,4 @@ -<tool id="comp1" name="Compare two Queries"> +<tool id="comp1" name="Compare two Queries" version="1.0.1"> <description>to find common or distinct rows</description> <command interpreter="python">joinWrapper.py $input1 $input2 $field1 $field2 $mode $out_file1</command> <inputs> diff -r e6324f395d26 -r 50dbd8907d88 tools/filters/joinWrapper.py --- a/tools/filters/joinWrapper.py Mon Oct 05 17:38:54 2009 -0400 +++ b/tools/filters/joinWrapper.py Mon Oct 05 17:39:29 2009 -0400 @@ -22,8 +22,8 @@ try: #Sort the two files based on specified fields - os.system("sort -k %d -o %s %s" %(field1, tmpfile1.name, infile1)) - os.system("sort -k %d -o %s %s" %(field2, tmpfile2.name, infile2)) + os.system("sort -t $'\t' -k %d -o %s %s" %(field1, tmpfile1.name, infile1)) + os.system("sort -t $'\t' -k %d -o %s %s" %(field2, tmpfile2.name, infile2)) except Exception, exc: stop_err( 'Initialization error -> %s' %str(exc) ) @@ -40,9 +40,9 @@ break if mode == "V": - cmdline = 'join -v 1 -o %s -1 %d -2 %d %s %s | tr " " "\t" > %s' %(option, field1, field2, tmpfile1.name, tmpfile2.name, outfile) + cmdline = "join -t $'\t' -v 1 -o %s -1 %d -2 %d %s %s > %s" %(option, field1, field2, tmpfile1.name, tmpfile2.name, outfile) else: - cmdline = 'join -o %s -1 %d -2 %d %s %s | tr " " "\t" > %s' %(option, field1, field2, tmpfile1.name, tmpfile2.name, outfile) + cmdline = "join -t $'\t' -o %s -1 %d -2 %d %s %s > %s" %(option, field1, field2, tmpfile1.name, tmpfile2.name, outfile) try: os.system(cmdline)

1 0