Am 25.08.2014 um 18:05 schrieb Dooley, Damion:
Ok, I'll be very happy to see what you've accomplished there. I will read through what you've done when I return from vacation in a week!
A key need is to have whatever data comes in show up as linked data in one's history to avoid server overhead; a second objective was to not need to modify existing workflows - as long as they could work of data in history that is typed appropriately. So your 'select type' solution sounds intreguing!
And certainly interested in your use of git - I tried using git, using a 1-line fasta data format, but git seemed to choke on protein fasta files? And did it run into performance problems with larger files? That was my experience. I think I read its authors say that its upper limit was 15gb.
This is probably true for one large file. I'm storing the entire PDB in git since a few years. One entry one file and it works fine. Do you know git annex? https://git-annex.branchable.com/
That was the motivation for writing a simple key-value master file diff system that seems to have the same I/O as git on smaller files, but more reliable for the fasta data case, and no problems with larger files - it outputs a new version in the same time it takes to read a master file. It has drawbacks though - incoming data to compare master with must be sorted in 1 line fasta format first.
My intention was to create a universal solution for database tracking. So if you can please design your system in such a way that you can store arbitrary data, not only fasta files.
Thanks for your input; looking forward to your project writeup...
Wonderful! Talk to you after your holidays! Bjoern
Damion
Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada ________________________________________ From: Björn Grüning [bjoern.gruening@gmail.com] Sent: Saturday, August 23, 2014 12:17 AM To: Dooley, Damion; galaxy-dev@lists.bx.psu.edu Cc: Hsiao, William Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool
Hi Damion,
the idea sounds fantastic! Can we go a step further and use a specific datatype that keeps entire fasta files versioned and the user can choose which version he wants to use, in any tool? Please have a look at my talk at GCC2012. Maybe you are interested in the (old) patches. I would be very interested to restart this old project.
https://wiki.galaxyproject.org/Events/GCC2012/Abstracts#Keeping_Track_of_Lif...
Am 23.08.2014 um 03:24 schrieb Dooley, Damion:
We are about to implement a fasta database (file) versioning system as a Galaxy tool. I wanted to get interested people's feedback first before we roll ahead with the prototype implementation. The versioning system aims to: .... - Simple makeblastdb or bowtie-build commands, or more specific workflows that include dustmasker etc can be implemented.
Does this sound attractive?
I think all of the use cases are covered by the old project mentioned above. But I did not create a new tool I have created a new 'select type' everyone can use in all tools. It was using git underneath (yeah, I have the entire PDB in git and it is working fine :)) but we can probably change git with a database if you like.
To answer your question: Yes, very attractive!
We're hoping such a vision could handle Fasta databases from 12mb to e.g. 200Gb (probably requires makeblastdb in parallel at that scale).
Preliminary work suggests this project is doable via the Galaxy API without galaxy customization - does that sound right?!
Yes, as long as the User has an API key.
Cheers, Bjoern