- galaxy-dev - lists.galaxyproject.org

GATK integration
by Jan Haas 27 May '11

27 May '11

Hey, I know, the GATK-tools is considered to be in alpha status, but maybe someone can help me. First, is there a documentation on which dependencies where to put (just to confirm I've done it correctly) ? Second, when I run the unified genotyper, I get the following error: [Wed May 25 15:57:04 CEST 2011] net.sf.picard.sam.CreateSequenceDictionary REFERENCE=/var/folders/gr/grLYUw45FFS4KKMXzK5R6++++TI/-Tmp-/tmp7m3sg0/gatk_input.fasta OUTPUT=/var/folders/gr/grLYUw45FFS4KKMXzK5R6++++TI/-Tmp-/tmp7m3sg0/dict6511818973877179087.tmp TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 TMP_DIR=/var/folders/gr/grLYUw45FFS4KKMXzK5R6++++TI/-Tmp-/ngs2 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false [Wed May 25 15:57:04 CEST 2011] net.sf.picard.sam.CreateSequenceDictionary done. Runtime.totalMemory()=129957888 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at net.sf.picard.reference.IndexedFastaSequenceFile.getSubsequenceAt(IndexedFastaSequenceFile.java:178) at net.sf.picard.reference.IndexedFastaSequenceFile.getSequence(IndexedFastaSequenceFile.java:157) at net.sf.picard.reference.IndexedFastaSequenceFile.nextSequence(IndexedFastaSequenceFile.java:234) at net.sf.picard.sam.CreateSequenceDictionary.makeSequenceDictionary(CreateSequenceDictionary.java:133) at net.sf.picard.sam.CreateSequenceDictionary.doWork(CreateSequenceDictionary.java:113) at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:165) at org.broadinstitute.sting.gatk.datasources.simpleDataSources.ReferenceDataSource.(ReferenceDataSource.java:131) at org.broadinstitute.sting.gatk.AbstractGenomeAnalysisEngine.openReferenceSequenceFile(AbstractGenomeAnalysisEngine.java:577) at org.broadinstitute.sting.gatk.AbstractGenomeAnalysisEngine.initializeDataSources(AbstractGenomeAnalysisEngine.java:318) at org.broadinstitute.sting.gatk.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:90) at org.broadinstitute.sting.gatk.CommandLineExecutable.execute(CommandLineExecutable.java:97) at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:244) at org.broadinstitute.sting.gatk.CommandLineGATK.main(CommandLineGATK.java:87) I assume, changing the java heap size will solve the problem. But how can I do it in galaxy ? Thanks for your help! Jan

1 0

stdout and stderr
by shashi shekhar 26 May '11

26 May '11

Hi all, I am using one tool which gives stdout and stderr . it is showing red color (job failed) in browser.How can i resolve such type of problem. Regards shashi shekhar

5 7

The powers of Cheetah, how?
by Kempenaar, M (med) 26 May '11

26 May '11

Hi list, I've been working on a single Galaxy tool for a while now, and I'm running into all sorts of difficulties that can probably all be fixed using Cheetah. However I've been trying several ways of utilizing the (enormous) powers of Cheetah but so far I'm not successful. I'll try to ask my questions as clear as possible, please let me know if anything is unclear. In my tool I use the following input parameter: ----------------------- <param name="report_options" type="select" multiple="True" display="checkboxes"> <option value="overview">phyla distribution</option> <option value="phyladend">phyla dendograms</option> <option value="heatmap">heatmap</option> <option value="dendogram">dendogram</option> <option value="pca">principal component analysis</option> <option value="mds">multi dimensional scaling</option> </param> ----------------------- Each of these options corresponds to a function in my R script, thus I want to do an if-else if-else to execute these functions that the user selected (note that multiple="True"). Now the problem is, I also use R Sweave (sweaving R output (text/graphics) into LaTeX) to generate a nice looking PDF report, and since that's not a 'real' programming/ scripting language, I can't for instance split the string (as I used to do in R) and check for the boxes a user selected, so my questions are: I planned on using Cheetah to include sections of Sweave LaTeX code for each box checked by the user, perfectly doable with an #if-#elif you'd think, however since the $report_options contains a comma-separated single string, I would need to split (or use regular expressions) to be able to do such an #if-#elif like: #if "heatmap" in $report_options.split(",") results in: "NotFound: cannot find 'split' while searching for 'split'" ----------------------- Questions: Q: How can I use this string with Cheetah, or is the low-level Python object (i.e. a list) also available (that is used to generate this string)? Q: You can use a <filter> for the <output> parameters, which doesn't work with $report_options in this case, so how to use the <filter> with 'multiple_select="True"'? Q: While trying out some Cheetah functionality (as explained in the Cheetah manual), I've came across a number of 'things' in the Cheetah documentation that aren't supported in Galaxy tool XML files. Is there some documentation available showing what is possible and what isn't? Q: Followup from previous question, where within the sections of my tool XML file can I actually use Cheetah? This of course has to do with the order in which a tool is executed. ----------------------- Any help or pointers / examples are greatly appreciated! Some links: - Cheetah Manual: http://www.cheetahtemplate.org/docs/users_guide_html/users_guide.html _ Excellent post about Cheetah on the mailing list: http://gmod.827538.n3.nabble.com/Bitten-by-a-Cheetah-long-td2760688.html Regards, Marcel ________________________________ De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht. The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

2 2

Re: [galaxy-dev] Popup menu population bug for data_column parameters
by Pieter Neerincx 26 May '11

26 May '11

Hi Galaxy team, It seems this one managed to fly below the radar: https://bitbucket.org/galaxy/galaxy-central/issue/491/data_column-parameter… Is there a chance I can re-enable automagic data format conversion in the near future? *Special note for Dan: GenePattern does not have this bug :) Cheers, Pi On Mar 3, 2011, at 7:32 PM, Pieter Neerincx wrote: > Hi Galaxy Devs, > > I think I found a bug in the handling of data_column parameters in combination with the data parameter they depend on. Unfortunately in some cases the popup menu to select a column number won't get populated. (I did a hg pull yesterday, but that didn't solve the problem.) > > Here's what I have to reproduce the problem: > > 1. Make a new histroy > 2. Upload a tabular file > 3. Upload a FASTA file > 4. Select Text Manipulation -> Merge Columns tool > 5. Tada, you cannot select a column from the popup menu... > > I've traced the problem to detection of a suitable initial value (history item) for the data parameter on which the data_column parameter depends. With the FASTA file uploaded last it will be selected as the initial value. It's not a tabular file, but it can be automagically converted to tabular (is_convertable( data ) == true). It won't show up though in the popup menu to select a history item for the data parameter. Apparently when the HTML form is generated the FASTA file is skipped and the tabular file is the only one in the popup menu. > > In galaxy/lib/galaxy/tools/parameters/basic.py: > > class DataToolParameter( ToolParameter ): > ... > def get_initial_value( self, trans, context ): > """ > NOTE: This is wasteful since dynamic options and dataset collection > happens twice (here and when generating HTML). > """ > ... > # The line below is number 1366 (in my version of this file.) > # This may have bad effects for data_column parameters that depend on this data parameter; removing the 'or is_convertable( data )' will 'fix' it, > # but is no good solution as it kills the ability to select data that can be converted automagically to a compatible data type. > # The note above may explain why the convertable history item does not show up in the popup menu... > #if data.visible and not data.deleted and data.state not in [data.states.ERROR, data.states.DISCARDED] and ( isinstance( data.datatype, self.formats) or is_convertable( data ) ): > if data.visible and not data.deleted and data.state not in [data.states.ERROR, data.states.DISCARDED] and isinstance( data.datatype, self.formats ): > > When the data_column parameter is initialized however class ColumnListParameter( SelectToolParameter ) -> def get_column_list( self, trans, other_values ) sees the FASTA file and hence there will be no dataset.metadata.columns and the popup menu remains empty... > > If you have another tabular file in your history selecting that one and then switching back will update and populate the popup menu for the data_column parameter. If the order of the history items in the above example is reversed you also won't see the issue as a data set of the correct format is the "initial value". > > Cheers, > > Pi > > > ------------------------------------------------------------------ > Biomolecular Mass Spectrometry & Proteomics group > Utrecht University > > phone: +31 6 143 66 783 > email: pieter.neerincx(a)gmail.com > skype: pieter.online > > visiting address: > H.R. Kruyt building // room O607 > Padualaan 8 // 3584 CH Utrecht // The Netherlands > > mail address: > P.O. box 80.082 // 3508 TB Utrecht // The Netherlands > ------------------------------------------------------------------ > ------------------------------------------------------------- mobile: +31 6 143 66 783 e-mail: pieter.neerincx(a)gmail.com skype: pieter.online -------------------------------------------------------------

1 0

Re: [galaxy-dev] Display column headers in data_column parameter
by Johannes Eichner 26 May '11

26 May '11

Hi Peter, I defined an additional metadata element for the column names analogously to the "column_types" element in the module "<galaxy_basedir>/lib/galaxy/tools/datatypes/tabular.py": MetadataElement( name="column_names", default=[], desc="Column names", param=metadata.ColumnTypesParameter, readonly=True, visible=False, no_value=[] ) Then I read the column names in the function "set.meta" and saved them in the new metadata element: dataset.metadata.column_names = column_names Finally, I changed a line of code in the function "get.column.list" in "<galaxy_basedir>/lib/galaxy/tools/parameters/basic.py", such that Galaxy displays the column names in the data_column parameter: original line: column_list.append( str( i + 1 ) ) changed line: column_list.append( str( i + 1 ) + ": " + dataset.metadata.column_names[i]) ) Kind regards, Johannes Dipl.-Inf. (Bioinf.) Johannes Eichner Univ. Tuebingen, WSI-RA, Sand 1, D-72076 Tuebingen, Germany Phone: (+49/0) 7071 / 29 70436 Fax: (+49/0) 7071 / 29 5091 Am 26.05.2011 15:27, schrieb Peter Cock: > Hi Johannes, > > Did you mean to send this to just me? Could you resent it CC'ing the > mailing list? > > I'd like to see your hack - it could be the basis for extending Galaxy itself. > > Peter > > On Thu, May 26, 2011 at 2:24 PM, Johannes Eichner > <johannes.eichner(a)uni-tuebingen.de> wrote: >> Hello Peter, >> >> thanks for your comments. I managed to display the column headers by >> defining a new datatype (inherited from tabular) with additional >> metadata for the column headers and by changing the python code in >> <galaxy_basedir>/lib/galaxy/tools/parameters/basic.py, such that the >> headers are displayed in the data_column parameter. >> >> This hack works for the present, but I am looking forward to seeing one >> of your ideas realized in a new version of Galaxy, which will hopefully >> supersede this preliminary hack. >> >> Kind regards, >> Johannes >> >> Dipl.-Inf. (Bioinf.) Johannes Eichner >> Univ. Tuebingen, WSI-RA, Sand 1, D-72076 Tuebingen, Germany >> Phone: (+49/0) 7071 / 29 70436 Fax: (+49/0) 7071 / 29 5091 >> >> Am 24.05.2011 16:54, schrieb Peter Cock: >>> On Tue, May 24, 2011 at 3:12 PM, Johannes Eichner >>> <johannes.eichner(a)uni-tuebingen.de> wrote: >>>> Dear Galaxy-Dev Team, >>>> >>>> is it possible to display the headers of the columns which can be >>>> selected in a data_column parameter? Currently, only the column index >>>> (e.g., c1, c2, etc.) is displayed in Galaxy. If it would be possible to >>>> additionally display the column headers, the selection of the desired >>>> columns would be easier for the user. Is there a way to do this using >>>> the current version of Galaxy? >>>> >>>> Kind regards, >>>> Johannes >>> >>> I completely agree that in general the UI for column parameter isn't >>> as user friendly as it could be. The filter data UI is even worse for >>> non-programmers, but due to its complexity and flexibility, that is >>> harder to improve too. >>> >>> This was one of the things I wanted to bring up with the Galaxy team >>> at the conference (starting tonight). Unfortunately due to the Grimsvotn >>> Volcano's cloud of ash I won't be there. >>> >>> In some cases yes, Galaxy knows the column meaning. In many of my >>> data files there is a # header line with tab separated column names, >>> which could be detected and shown. >>> >>> Given there may or not be suitable column meta data, what I was >>> picturing was an expanded column picking dialogue that could show >>> the first few rows of data in each column (not using a simple HTML >>> widget anymore), or simply the first value (less drastic change). >>> >>> So, rather than having as now something like: Pick column(s): >>> [ ] c1 >>> [ ] c2 >>> [ ] c3 >>> [ ] c4 >>> [ ] c5 >>> >>> If Galaxy has names in the column meta data you might have >>> Pick columns(s): >>> [ ] c1 - Identifier >>> [ ] c2 - Start >>> [ ] c3 - End >>> [ ] c4 - Strand >>> [ ] c5 - Sequence >>> >>> Or, showing the first data entry: Pick columns(s): >>> [ ] c1, e.g. "gi|12345678" >>> [ ] c2, e.g. "123" >>> [ ] c3, e.g. "456" >>> [ ] c4, e.g. "+" >>> [ ] c4, e.g. "ACGTACTGT..." >>> >>> In the final example I envision truncating very long values with >>> an ellipsis. >>> >>> Both the above proposals don't require drastic changes to the >>> HTML widget and layout. >>> >>> I guess we should file an enhancement issue on bitbucket: >>> https://bitbucket.org/galaxy/galaxy-central/issues/new >>> >>> Peter >>

2 1

Paper on workflow patterns
by Peter van Heusden 26 May '11

26 May '11

As discussed in the Workflows and APIs breakaway session, here is the paper on Workflow patterns: van der Aalst, ter Hofsted, Kiepuszewski and Barros, Distributed and Parallel Databases, Volume 14, Number 1, 5-51: Workflow Patterns PDF available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.1538&rep=rep1&t…

1 0

Display column headers in data_column parameter
by Johannes Eichner 26 May '11

26 May '11

Dear Galaxy-Dev Team, is it possible to display the headers of the columns which can be selected in a data_column parameter? Currently, only the column index (e.g., c1, c2, etc.) is displayed in Galaxy. If it would be possible to additionally display the column headers, the selection of the desired columns would be easier for the user. Is there a way to do this using the current version of Galaxy? Kind regards, Johannes -- Dipl.-Inf. (Bioinf.) Johannes Eichner Univ. Tuebingen, WSI-RA, Sand 1, D-72076 Tuebingen, Germany Phone: (+49/0) 7071 / 29 70436 Fax: (+49/0) 7071 / 29 5091

2 2

cluster files directory configuration
by Shantanu Pavgi 26 May '11

26 May '11

I have configured custom cluster_files_directory in universe_wsgi.ini file. The cluster jobs are running fine, however job scripts, stdout and stderr are being written in default database/pbs location. Am I missing something in the configuration here? -- Thanks, Shantanu.

2 3

Getting binary programs into Galaxy distribution?
by Duddy, John 25 May '11

25 May '11

There is a C program for merging Gzip files (gzjoin) that I'd love to rely on for a core Galaxy capability. Is there a standard way to get things like this included in Galaxy? Recoding it in Python would be a bit of a pain, and might be a lot slower due to the IO layer not allowing the reuse of buffers. Thanks - John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

3 4

NGS index files not showing up
by Jeremy Lipson 25 May '11

25 May '11

Hi, I've installed galaxy locally on my Mac, I've got everything running fine except Bowtie/Tophat/etc. indexes. I tried following the steps here: https://bitbucket.org/galaxy/galaxy-central/wiki/NGSLocalSetup So my bowtie_indices.loc file looks like this: "e_coli /Users/jel65/galaxy-central/tool-data/e_coli" (no quotes) (I was just testing the e_coli index that came with bowtie) So I saved it, restarted the galaxy server, but still nothing shows up in the drop down. Did I miss a step? I've read the documentation multiple times. Thanks Jeremy

1 0