Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool
by Pedersen Edvard
My PhD work may be of interest for this subject, although the primary focus has been on generating databases comprising the changes from a specific timeframe, and was not designed specifically for Galaxy. The similarities between my system and the system you are proposing are that it can generate a BLAST database from any date (that has been added to the system), as well as "diffs" between two dates, and supports FASTA, the Uniprot EMBL variant, full files (which does not give compression benefits) and several others. The system uses delta compression to make sure that non-updated fields do not take up extra space. It uses the Hadoop stack (HBase, HDFS and MapReduce) for parallelism in generating the databases (the blast database generation from FASTA files is not parallel).
You can find one of the publications here: <http://bdps.cs.uit.no/papers/hibb13.pdf> http://bdps.cs.uit.no/papers/hibb13.pdf
I hope this can be of some use to you.
Regards,
Edvard Pedersen
5 years, 5 months
Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool
by Dooley, Damion
Earlier on in the project analysis I was pursuing a Git solution because it seemed all its features would work with documents/code/files of any kind and so would be perfect for scientific reproducibility. But its ability to efficiently archive non-documents is quite hit and miss, and the file size limitation becomes a major problem on top of that when it doesn't.
I will try to design the system so that handlers for different types of databases/files can be called into play to retrieve versioned content.
Its just that this fall I'll only have time to provide the handlers for fasta file archiving (the key-value database update approach enables fasta versioning and all the spinoff data from that.).
The next priority would be a handler for any type of file that needs to be replaced as a whole from version to version (one just needs hard drive space to accommodate this, since caching is pointless).
A git handler for well-behaved document content would also be a possibility.
Typo: I said yesterday "I wasn't going to leave that as just "fasta" datatype since it seems tools like makeblastdb don't allow anything else ..." - but I meant 'I WAS going to leave that as just "fasta"...'
d.
5 years, 5 months
Examples of Galaxy tools in the toolsheds that install and run JAR files properly?
by Melissa Cline
Hi folks,
I'm attempting something that should be straightforward, but it's not. I
have a tool that runs a JAR file, which I have bundled with the tool. I
simply want to run the JAR file. And to paraphrase Thomas Edison, I've
tried several thousand things that do not work (at least for me), from
setting the JAVA_JAR_PATH environment variable in the tool_dependencies.xml
file to trying to copy the JAR file into the tool-data/shared/jars
subdirectories (which is the closest thing I've got to working). So, at
long last I'm doing the sensible thing and looking for one simple working
example that I can use as a template. Who can suggest a good toolshed tool
(either main or test) that involves running its own JAR file, and that
works?
Thanks!
Melissa
5 years, 5 months
Startup error after restoring Galaxy DB from backup
by Graeme Grimes
I have tried to restart Galaxy after restoring my database from a backup.
Here is the error message I get in the log file. Any idea what is wrong
and how to fix this problem?
------------------------------
galaxy.jobs DEBUG 2014-09-03 08:33:46,367 Loading job configuration from
/export/users/galaxy/galaxy-test/universe_wsgi.ini
galaxy.jobs DEBUG 2014-09-03 08:33:46,367 Done loading job configuration
Traceback (most recent call last):
File
"/export/users/galaxy/galaxy-test/lib/galaxy/webapps/galaxy/buildapp.py", line
35, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File "/export/users/galaxy/galaxy-test/lib/galaxy/app.py", line 102,
in __init__
self.toolbox = tools.ToolBox( tool_configs, self.config.tool_path,
self )
File "/export/users/galaxy/galaxy-test/lib/galaxy/tools/__init__.py",
line 118, in __init__
self.load_integrated_tool_panel_keys()
File "/export/users/galaxy/galaxy-test/lib/galaxy/tools/__init__.py",
line 283, in load_integrated_tool_panel_keys
tree = parse_xml( self.integrated_tool_panel_config )
File "/export/users/galaxy/galaxy-test/lib/galaxy/util/__init__.py",
line 132, in parse_xml
tree = ElementTree.parse(fname)
File
"/export/users/galaxy/galaxy-test/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py",
line 859, in parse
tree.parse(source, parser)
File
"/export/users/galaxy/galaxy-test/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py",
line 583, in parse
parser.feed(data)
File
"/export/users/galaxy/galaxy-test/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py",
line 1242, in feed
self._parser.Parse(data, 0)
ExpatError: not well-formed (invalid token): line 117, column 1
Removing PID file /var/run/paster.pid
-----------------
Thanks,
Graeme
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
5 years, 5 months
Concept for a Galaxy Versioned Fasta Data Retrieval Tool
by Dooley, Damion
We are about to implement a fasta database (file) versioning system as a Galaxy tool. I wanted to get interested people's feedback first before we roll ahead with the prototype implementation. The versioning system aims to:
* Enable reproducible research: To recreate a search result at a certain point in time we need versioning so that search and mapping tools can look at sequence reference databases corresponding to a particular past date. This recall can also explain the difference between what was known in the past vs. currently.
* Reduce hard drive space. Some databases are too big to keep N copies around, e.g. 5 years of 16S, updated monthly, is say, 670Mb + 668Mb + 665Mb + .... But occasionally we want to access past archives fairly quickly.
* Integrate database versioning into Galaxy without adding a lot of complexity.
A bonus would be to enable the efficient sharing of version databases between computers/servers.
The solution we think would work centres around a "Versioned Data Retrieval" tool (draft image attached) that would work as follows:
1) User selects from a list of databases provided by "Shared Data > Data Libraries > Versioned Data".
- Each database has a master file that keeps its various versions as a list of time-stamped insert/delete transactions of key (fasta id) value (description & sequence) pairs.
- Each master file is managed outside of galaxy via a triggered process on regular fasta file imports from data sources like NCBI or other niche sources.
- We're expecting, due to the nature of fasta archived sequence updates, that our master file would only be about 1.1x the latest version in size (uncompressed).
2) User enters date / version id to retrieve (validated)
3) If a cached version of that database exists, it is linked into user's history.
4) Otherwise a new version of it is created, placed in cache, and linked into history.
- The cached version itself then shows up as linked data under a Data Library > Versioned Data subfolder.
5) User can select preconfigured workflow(s) to execute on the selected retreived fasta file to regenerate any database products they need.
- Workflow output data would also be cached in the same way the fasta data is - by linking the Galaxy Data Library to it.
- Workflow execution will be skipped if end data already exists in cache.
- Simple makeblastdb or bowtie-build commands, or more specific workflows that include dustmasker etc can be implemented.
Does this sound attractive?
We're hoping such a vision could handle Fasta databases from 12mb to e.g. 200Gb (probably requires makeblastdb in parallel at that scale).
Preliminary work suggests this project is doable via the Galaxy API without galaxy customization - does that sound right?!
Feedback really appreciated!
Regards,
Damion Dooley
Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
5 years, 5 months
Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool
by Dooley, Damion
About the datatype. So you are thinking of a new datatype that applies to files that hold the versioned database contents (in this case a structured key-value fasta identifier/sequence pairs, right?) Then the fasta archive versioning tool would take only files of that datatype for input. That sounds good.
I was just going to have one folder (and its subfolders) in the data library that hold all the versioned databases to choose from. So the versioned database tool would just populate its input list based on that subfolder tree. But ensuring that it lists only files of a certain datatype sounds beneficial.
Output in any case would be a fasta file that other tools are already expecting; I wasn't going to leave that as just "fasta" datatype since it seems tools like makeblastdb don't allow anything else as input from user history.
I'm hoping that a global (admin) API key can be used by the tool so that all users can get versioned data, but maybe that is a pipe dream.
Sure I'd like to see old patches!
d.
________________________________________
From: Bj?rn Gr?ning [bjoern.gruening(a)gmail.com]
Sent: Saturday, August 23, 2014 12:17 AM
To: Dooley, Damion; galaxy-dev(a)lists.bx.psu.edu
Cc: Hsiao, William
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool
Hi Damion,
the idea sounds fantastic!
Can we go a step further and use a specific datatype that keeps entire
fasta files versioned and the user can choose which version he wants to
use, in any tool? Please have a look at my talk at GCC2012. Maybe you
are interested in the (old) patches. I would be very interested to
restart this old project.
5 years, 5 months
"when else" in <conditional> ? RE: refresh_on_change : is this a valid attribute? Any other ideas/options??
by Lukasse, Pieter
So I need to refresh on change....I see that if I have a conditional item in my form, this causes a refresh of the page and a (re)evaluation of my dynamic_options methods....so I could misuse this "feature". However, it seems that when I have a <conditional> I must have a <when> entry for every item in my select box. There is no "when else" option?
Thanks,
Pieter
From: galaxy-dev-bounces(a)lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Lukasse, Pieter
Sent: woensdag 27 augustus 2014 22:37
To: galaxy-dev(a)lists.bx.psu.edu
Subject: [galaxy-dev] refresh_on_change : is this a valid attribute? Any other ideas/options??
Hi,
I'm trying to get a wrapper from someone else working and I found this "refresh_on_change" attribute in his select boxes which are filled using the dynamic_options feature:
<param name="col_type" type="select" label="Select column type" refresh_on_change="true"
display="radio" dynamic_options='get_column_type(library_file)'
help="" />
<param name="polarity" type="select" label="Select polarity" refresh_on_change="true"
display="radio" dynamic_options='filter_column(library_file,col_type)'
help="" />
...
When searching the documentation/wiki I do not find a reference to this, but it would be a nice option to have ;)
Question: is there any way I can force a refresh when the user selects another option from such a select box. As you can see in the example above, this is needed because the next select box has its dynamic options built up by a function that takes the value from the previous select (col_type - highlighted above) as an input parameter. Currently this tool only works by showing each select in its own <page> , which is a deprecated option and prevents the tool from being used in a workflow... :(
Thanks for your help!
Best regards,
Pieter Lukasse
Wageningen UR, Plant Research International
Department of Bioinformatics (Bioscience)
Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB,
Wageningen, the Netherlands
T: +31-317481122;
M: +31-628189540;
skype: pieter.lukasse.wur
http://www.pri.wur.nl<http://www.pri.wur.nl/>
5 years, 5 months
directory as an input file
by Philippe Moncuquet
Hi,
I am trying to write a wrapper for a tool that take a directory containing
SAM/BAM files as an input. I am not sure how to do that, is there another
tool that implements this and that I can have a look at ? Any suggestions
would be greatly appreciated.
Regards,
Philip
5 years, 5 months
millions for rows in galaxy_session
by Robert Baertsch
Is there anyway to have galaxy not insert a row in the galaxy_session file for failed logins. Every ping from china seems to generate a row in galaxy_session. We have anonymous login turned off
Robert
5 years, 5 months
Re: [galaxy-dev] error with multi dataset tool run
by Robert Baertsch
>
yes, I was the only user but I guess it is too fragile for this kind of thing. postgres wasn’t queuing age jobs correctly so I guess I should track down why.
>
> From: Hans-Rudolf Hotz <hrh(a)fmi.ch>
> Subject: Re: [galaxy-dev] error with multi dataset tool run
> Date: September 1, 2014 at 5:54:10 AM PDT
> To: Robert Baertsch <baertsch(a)soe.ucsc.edu>, <galaxy-dev(a)lists.bx.psu.edu>
>
>
> Hi Robert
>
> Are you using the built in SQLite database ?
>
> Hans-Rudolf
>
> On 08/31/2014 01:27 AM, Robert Baertsch wrote:
>> I submitted 13 fastq files to tophat2 using DRMAA and got this error.
>> Is it fatal? BTW: This is a super cool feature.
>>
>> I’m running the following version of galaxy-dist.
>>
>> changeset: 14212:91547729ffde
>> branch: stable
>> tag: tip
>> user: Nate Coraor <nate(a)bx.psu.edu <mailto:nate@bx.psu.edu>>
>> date: Fri Aug 29 14:00:23 2014 -0400
>> summary: Update tag latest_2014.08.11 for changeset ea12550fbc34
>>
>> -Robert
>>
>>
>> There were errors setting up 2 submitted job(s):
>>
>> * *Error executing tool: (OperationalError) database is locked
>> u'UPDATE history_dataset_association SET update_time=?, name=?,
>> blurb=? WHERE history_dataset_association.id = ?' ('2014-08-30
>> 23:14:44.683957', 'Tophat2 on data 7: insertions', 'queued', 137)*
>> * *Error executing tool: (OperationalError) database is locked
>> u'UPDATE dataset SET update_time=?, state=? WHERE dataset.id = ?'
>> ('2014-08-30 23:15:57.204718', 'queued', 446)*
>>
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/mailinglists/
>>
>
>
>
>
> From: Sandra Derozier <sandra.derozier(a)jouy.inra.fr>
> Subject: [galaxy-dev] Error with functional tests on cluster
> Date: September 1, 2014 at 6:54:41 AM PDT
> To: galaxy-dev(a)bx.psu.edu
>
>
> Hi all,
>
> I try to set up functional tests on different tools on my Galaxy portal.
>
> When I run functional tests locally everything works fine. But when I run them on the cluster it failed with this message :
>
> /bin/sh: module: line 1: syntax error: unexpected end of file
> /bin/sh: error importing function definition for `module'
>
> The execution on the cluster is ok but the dataset state is set to ERROR.
>
> The DIFF between the expected result and the obtained result is null. Indeed, this two files are the same.
>
> As I do not know what the problem is: do you have an idea?
>
> Thanks,
> Sandra DEROZIER
>
> Sandra DEROZIER
> Unité Mathèmatique, Informatique et Génome (MIG)
> Plateforme MIGALE
> Bâtiment 233
> Domaine de Vilvert
> 78352 Jouy-en-Josas Cedex
>
>
>
> From: Hans-Rudolf Hotz <hrh(a)fmi.ch>
> Subject: [galaxy-dev] Solved - Re: testing the visualization plugins
> Date: September 1, 2014 at 8:13:30 AM PDT
> To: "<galaxy-dev(a)bx.psu.edu>" <galaxy-dev(a)bx.psu.edu>
>
>
> Hi all
>
> First of all, a big Thanks to Carl who helped me fixing this problem.
>
> So as a summary for all, the problem was caused by a datatype (an extension to tabular), I manually added to "datatypes_conf.xml"
>
> Removing the datatype fixed the problem. I couldn't identify a syntax problem, neither in "datatypes_conf.xml" nor in "~/lib/galaxy/datatypes/registry.py" and "~/lib/galaxy/datatypes/tabular.py". However, renaming it (in all three files) fixed it as well.
>
> Regards, Hans-Rudolf
>
>
>
> _______________________________________________
> galaxy-dev mailing list
> galaxy-dev(a)lists.bx.psu.edu
> http://lists.bx.psu.edu/listinfo/galaxy-dev
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
5 years, 5 months
