May 2011 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Popup menu population bug for data_column parameters
by Pieter Neerincx 26 May '11

26 May '11

Hi Galaxy team, It seems this one managed to fly below the radar: https://bitbucket.org/galaxy/galaxy-central/issue/491/data_column-parameter… Is there a chance I can re-enable automagic data format conversion in the near future? *Special note for Dan: GenePattern does not have this bug :) Cheers, Pi On Mar 3, 2011, at 7:32 PM, Pieter Neerincx wrote: > Hi Galaxy Devs, > > I think I found a bug in the handling of data_column parameters in combination with the data parameter they depend on. Unfortunately in some cases the popup menu to select a column number won't get populated. (I did a hg pull yesterday, but that didn't solve the problem.) > > Here's what I have to reproduce the problem: > > 1. Make a new histroy > 2. Upload a tabular file > 3. Upload a FASTA file > 4. Select Text Manipulation -> Merge Columns tool > 5. Tada, you cannot select a column from the popup menu... > > I've traced the problem to detection of a suitable initial value (history item) for the data parameter on which the data_column parameter depends. With the FASTA file uploaded last it will be selected as the initial value. It's not a tabular file, but it can be automagically converted to tabular (is_convertable( data ) == true). It won't show up though in the popup menu to select a history item for the data parameter. Apparently when the HTML form is generated the FASTA file is skipped and the tabular file is the only one in the popup menu. > > In galaxy/lib/galaxy/tools/parameters/basic.py: > > class DataToolParameter( ToolParameter ): > ... > def get_initial_value( self, trans, context ): > """ > NOTE: This is wasteful since dynamic options and dataset collection > happens twice (here and when generating HTML). > """ > ... > # The line below is number 1366 (in my version of this file.) > # This may have bad effects for data_column parameters that depend on this data parameter; removing the 'or is_convertable( data )' will 'fix' it, > # but is no good solution as it kills the ability to select data that can be converted automagically to a compatible data type. > # The note above may explain why the convertable history item does not show up in the popup menu... > #if data.visible and not data.deleted and data.state not in [data.states.ERROR, data.states.DISCARDED] and ( isinstance( data.datatype, self.formats) or is_convertable( data ) ): > if data.visible and not data.deleted and data.state not in [data.states.ERROR, data.states.DISCARDED] and isinstance( data.datatype, self.formats ): > > When the data_column parameter is initialized however class ColumnListParameter( SelectToolParameter ) -> def get_column_list( self, trans, other_values ) sees the FASTA file and hence there will be no dataset.metadata.columns and the popup menu remains empty... > > If you have another tabular file in your history selecting that one and then switching back will update and populate the popup menu for the data_column parameter. If the order of the history items in the above example is reversed you also won't see the issue as a data set of the correct format is the "initial value". > > Cheers, > > Pi > > > ------------------------------------------------------------------ > Biomolecular Mass Spectrometry & Proteomics group > Utrecht University > > phone: +31 6 143 66 783 > email: pieter.neerincx(a)gmail.com > skype: pieter.online > > visiting address: > H.R. Kruyt building // room O607 > Padualaan 8 // 3584 CH Utrecht // The Netherlands > > mail address: > P.O. box 80.082 // 3508 TB Utrecht // The Netherlands > ------------------------------------------------------------------ > ------------------------------------------------------------- mobile: +31 6 143 66 783 e-mail: pieter.neerincx(a)gmail.com skype: pieter.online -------------------------------------------------------------

1 0

Re: [galaxy-dev] Display column headers in data_column parameter
by Johannes Eichner 26 May '11

26 May '11

Hi Peter, I defined an additional metadata element for the column names analogously to the "column_types" element in the module "<galaxy_basedir>/lib/galaxy/tools/datatypes/tabular.py": MetadataElement( name="column_names", default=[], desc="Column names", param=metadata.ColumnTypesParameter, readonly=True, visible=False, no_value=[] ) Then I read the column names in the function "set.meta" and saved them in the new metadata element: dataset.metadata.column_names = column_names Finally, I changed a line of code in the function "get.column.list" in "<galaxy_basedir>/lib/galaxy/tools/parameters/basic.py", such that Galaxy displays the column names in the data_column parameter: original line: column_list.append( str( i + 1 ) ) changed line: column_list.append( str( i + 1 ) + ": " + dataset.metadata.column_names[i]) ) Kind regards, Johannes Dipl.-Inf. (Bioinf.) Johannes Eichner Univ. Tuebingen, WSI-RA, Sand 1, D-72076 Tuebingen, Germany Phone: (+49/0) 7071 / 29 70436 Fax: (+49/0) 7071 / 29 5091 Am 26.05.2011 15:27, schrieb Peter Cock: > Hi Johannes, > > Did you mean to send this to just me? Could you resent it CC'ing the > mailing list? > > I'd like to see your hack - it could be the basis for extending Galaxy itself. > > Peter > > On Thu, May 26, 2011 at 2:24 PM, Johannes Eichner > <johannes.eichner(a)uni-tuebingen.de> wrote: >> Hello Peter, >> >> thanks for your comments. I managed to display the column headers by >> defining a new datatype (inherited from tabular) with additional >> metadata for the column headers and by changing the python code in >> <galaxy_basedir>/lib/galaxy/tools/parameters/basic.py, such that the >> headers are displayed in the data_column parameter. >> >> This hack works for the present, but I am looking forward to seeing one >> of your ideas realized in a new version of Galaxy, which will hopefully >> supersede this preliminary hack. >> >> Kind regards, >> Johannes >> >> Dipl.-Inf. (Bioinf.) Johannes Eichner >> Univ. Tuebingen, WSI-RA, Sand 1, D-72076 Tuebingen, Germany >> Phone: (+49/0) 7071 / 29 70436 Fax: (+49/0) 7071 / 29 5091 >> >> Am 24.05.2011 16:54, schrieb Peter Cock: >>> On Tue, May 24, 2011 at 3:12 PM, Johannes Eichner >>> <johannes.eichner(a)uni-tuebingen.de> wrote: >>>> Dear Galaxy-Dev Team, >>>> >>>> is it possible to display the headers of the columns which can be >>>> selected in a data_column parameter? Currently, only the column index >>>> (e.g., c1, c2, etc.) is displayed in Galaxy. If it would be possible to >>>> additionally display the column headers, the selection of the desired >>>> columns would be easier for the user. Is there a way to do this using >>>> the current version of Galaxy? >>>> >>>> Kind regards, >>>> Johannes >>> >>> I completely agree that in general the UI for column parameter isn't >>> as user friendly as it could be. The filter data UI is even worse for >>> non-programmers, but due to its complexity and flexibility, that is >>> harder to improve too. >>> >>> This was one of the things I wanted to bring up with the Galaxy team >>> at the conference (starting tonight). Unfortunately due to the Grimsvotn >>> Volcano's cloud of ash I won't be there. >>> >>> In some cases yes, Galaxy knows the column meaning. In many of my >>> data files there is a # header line with tab separated column names, >>> which could be detected and shown. >>> >>> Given there may or not be suitable column meta data, what I was >>> picturing was an expanded column picking dialogue that could show >>> the first few rows of data in each column (not using a simple HTML >>> widget anymore), or simply the first value (less drastic change). >>> >>> So, rather than having as now something like: Pick column(s): >>> [ ] c1 >>> [ ] c2 >>> [ ] c3 >>> [ ] c4 >>> [ ] c5 >>> >>> If Galaxy has names in the column meta data you might have >>> Pick columns(s): >>> [ ] c1 - Identifier >>> [ ] c2 - Start >>> [ ] c3 - End >>> [ ] c4 - Strand >>> [ ] c5 - Sequence >>> >>> Or, showing the first data entry: Pick columns(s): >>> [ ] c1, e.g. "gi|12345678" >>> [ ] c2, e.g. "123" >>> [ ] c3, e.g. "456" >>> [ ] c4, e.g. "+" >>> [ ] c4, e.g. "ACGTACTGT..." >>> >>> In the final example I envision truncating very long values with >>> an ellipsis. >>> >>> Both the above proposals don't require drastic changes to the >>> HTML widget and layout. >>> >>> I guess we should file an enhancement issue on bitbucket: >>> https://bitbucket.org/galaxy/galaxy-central/issues/new >>> >>> Peter >>

2 1

Paper on workflow patterns
by Peter van Heusden 26 May '11

26 May '11

As discussed in the Workflows and APIs breakaway session, here is the paper on Workflow patterns: van der Aalst, ter Hofsted, Kiepuszewski and Barros, Distributed and Parallel Databases, Volume 14, Number 1, 5-51: Workflow Patterns PDF available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.1538&rep=rep1&t…

1 0

Display column headers in data_column parameter
by Johannes Eichner 26 May '11

26 May '11

Dear Galaxy-Dev Team, is it possible to display the headers of the columns which can be selected in a data_column parameter? Currently, only the column index (e.g., c1, c2, etc.) is displayed in Galaxy. If it would be possible to additionally display the column headers, the selection of the desired columns would be easier for the user. Is there a way to do this using the current version of Galaxy? Kind regards, Johannes -- Dipl.-Inf. (Bioinf.) Johannes Eichner Univ. Tuebingen, WSI-RA, Sand 1, D-72076 Tuebingen, Germany Phone: (+49/0) 7071 / 29 70436 Fax: (+49/0) 7071 / 29 5091

2 2

cluster files directory configuration
by Shantanu Pavgi 26 May '11

26 May '11

I have configured custom cluster_files_directory in universe_wsgi.ini file. The cluster jobs are running fine, however job scripts, stdout and stderr are being written in default database/pbs location. Am I missing something in the configuration here? -- Thanks, Shantanu.

2 3

Getting binary programs into Galaxy distribution?
by Duddy, John 25 May '11

25 May '11

There is a C program for merging Gzip files (gzjoin) that I'd love to rely on for a core Galaxy capability. Is there a standard way to get things like this included in Galaxy? Recoding it in Python would be a bit of a pain, and might be a lot slower due to the IO layer not allowing the reuse of buffers. Thanks - John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jduddy(a)illumina.com<mailto:jduddy@illumina.com>

3 4

NGS index files not showing up
by Jeremy Lipson 25 May '11

25 May '11

Hi, I've installed galaxy locally on my Mac, I've got everything running fine except Bowtie/Tophat/etc. indexes. I tried following the steps here: https://bitbucket.org/galaxy/galaxy-central/wiki/NGSLocalSetup So my bowtie_indices.loc file looks like this: "e_coli /Users/jel65/galaxy-central/tool-data/e_coli" (no quotes) (I was just testing the e_coli index that came with bowtie) So I saved it, restarted the galaxy server, but still nothing shows up in the drop down. Did I miss a step? I've read the documentation multiple times. Thanks Jeremy

1 0

multiple cluster job runners?
by Shantanu Pavgi 25 May '11

25 May '11

I am looking at options to configure settings such as wall-time, memory, and number of cores etc. for each tool and wondering if this can be done using multiple cluster job runner configurations. Is it possible to define multiple cluster job runner configurations for each tool? In the universe_wsgi.ini file it is mentioned that individual per-tool job runners can be specified, however I don't see any example other than 'default_cluster_job_runner'. So is there any key-value syntax for defining it in the ini file? I appreciate any comments or suggestions regarding this issue. -- Thanks, Shantanu.

2 3

Re: [galaxy-dev] Galaxy BLAST XML to tabular fix
by Peter Cock 25 May '11

25 May '11

On Wed, May 18, 2011 at 6:40 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hi Kanwei, > > Could you merge this branch or transplant this fix please? > > https://bitbucket.org/peterjc/galaxy-central/changeset/3c0b903c86a9 > > https://bitbucket.org/peterjc/galaxy-central/src/blast_2011_05_18 > > New branch, blast_2011_05_18 > > There is no change to the logic, just to the assert statement where > the sanity test didn't consider doubly masked sequences. > > Should I also update the XML wrapper's version number as well? > > Thanks, > > Peter Hi Kanwei, Did this email reach you? Let me know if you want a new unit test to go with it and/or to bump the tool's version number. Regards, Peter

2 2

Problem with file's extension
by shashi shekhar 25 May '11

25 May '11

Hi All, I have some problem with extension of file.some executables file will recognize the only the specific extension like autodock . it takes one input file ( extension with ".dpf" ) and one outpt file (extension with ".dlg"). how can we resolve such type of problem . Regards Shashi shekhar

3 2