February 2013 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Best way to work with one directory and many files as 1 input
by Pierre Pericard 01 Feb '13

01 Feb '13

Hi Christos, Yes, I would be very interested by your modified file. I'm forwarding to the mailing-list so this bug can be listed. Thanks, Pierre Pierre Pericard IE CDD - Projet Peptisan Service Informatique et Bio-informatique (SIB) Station Biologique de Roscoff CNRS - UPMC Place Georges Teissier CS 90074 29688 ROSCOFF CEDEX FRANCE Tel : (+33) 2 98 29 56 46 http://abims.sb-roscoff.fr/ Le 01/02/2013 00:34, chriskannas(a)gmail.com a écrit : > Hi, > > I'm working on the same thing, having a composite dataset with many files in many sub-folders. > > Well during the development & testing of the tool that uses this composite datatype, I found out that Galaxy does not handle correctly sub-dirs, it works pretty good with files but if it finds a sub-dir then the copy messes it up. > > To be more specific it uses shutil.copy without checking if it is file or directory, when coping dirs it has to use shutil.copytree. > > I managed to make our local Galaxy installation to handle them quite ok with some modifications in method DiskObjectStor.update_from_file(...) located in lib/galaxy/objectstore/__init__.py > > I can send you a copy of the file I've modified. > > Regards, > Christos > > ------ > > Christos Kannas > Researcher > Ph.D Student > > e-Health Laboratory > Dept. Computer Science, > University of Cyprus > > <quote author='Pierre Pericard'> > Ok, thanks a lot, I'll try and get back to the mailing list if other > problems seem to occur. > > Pierre. > > > Pierre Pericard > IE CDD - Projet Peptisan > > Service Informatique et Bio-informatique (SIB) > Station Biologique de Roscoff > CNRS-UPMC > Place Georges Teissier > CS 90074 > 29688 Roscoff CEDEX > FRANCE > http://abims.sb-roscoff.fr/ > > Le 30/01/2013 11:45, Ross a écrit : >> I'd suggest: >> 1) Make your new datatype a subclass of Html - it's a subclass of >> composite that contains an HTML document as the object's native >> display - so it can inform users what's there. >> >> 2) When constructing these new things, pass the file_path of the Html >> (composite) dataset subclass to your wrapper on the command line >> >> 3) Your wrapper code can construct any arbitrary structure as long as >> it's rooted in that directory - Galaxy stores it without any fuss. The >> wrapper should also populate the Html file itself with nicely >> laid annotation for the user to check out. >> >> 4) The key is that all tools that take this new datatype as input must >> know how to decode this structure - they must be passed the >> $input.extra_files_path which gives them that same path root. >> >> 5) Yes, it's odd and annoying that it's extra_files_path for >> files_path. Go figure. >> >> 6) grep extra_files tools/*.xml to find some examples - I think the >> velvetg one uses a complex subdirectory structure - but it doesn't >> really matter - as long as your tools know how to deal with it, it's >> just a directory to Galaxy! >> I hope all this helps... >> >> >> On Wed, Jan 30, 2013 at 8:22 PM, Pierre Pericard >> <pierre.pericard(a)sb-roscoff.fr <mailto:pierre.pericard@sb-roscoff.fr>> >> wrote: >> >> In that case, could anyone point me to an example of a Composite >> Datatype which could accept as input an unknown number of files in >> an unknown number of directories. I can't seem to understand how >> that would work based on the wiki. >> >> But maybe are we anticipating a near functionality of Galaxy. >> There were talks about changing the way Galaxy handle zip files, >> is it still on the table ? >> >> Thank in advance for any help, >> >> Pierre >> >> >> >> Pierre Pericard >> IE CDD - Projet Peptisan >> >> Service Informatique et Bio-informatique (SIB) >> Station Biologique de Roscoff >> CNRS-UPMC >> Place Georges Teissier >> CS 90074 >> 29688 Roscoff CEDEX >> FRANCE >> http://abims.sb-roscoff.fr/ >> >> Le 29/01/2013 18:04, Peter Cock a écrit : >> >> On Tue, Jan 29, 2013 at 4:58 PM, Pierre Pericard >> <pierre.pericard(a)sb-roscoff.fr >> <mailto:pierre.pericard@sb-roscoff.fr>> wrote: >> >> If I'm not mistaking, Composite Datatypes allow for only >> one directory, >> whereas we need to keep a constant directory structure >> with 2 or more >> sub-directories containing our input files. >> >> I'm not sure if that is true - the example of HTML output with >> images >> comes to mind as a common use-case where subfolder(s) would be >> expected. I've only had limited first hand experience with >> Galaxy's >> composite datatypes myself though. >> >> We have no way to change these tools behavior (obviously >> not Galaxy-friendly >> ;-) ) and therefore need to maintain this structure in the >> job working >> directory. >> >> Perhaps a tool wrapper could create a dummy folder using symlinks >> (faster and less wasted disk than copying files), but that >> isn't ideal. >> >> Peter >> >> >> ___________________________________________________________ >> > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > </quote> > Quoted from: > http://dev.list.galaxyproject.org/Best-way-to-work-with-one-directory-and-m…

1 0

uploading big file on local Galaxy
by stefano cardinale 01 Feb '13

01 Feb '13

Hi, I have installed Galaxy on our local server, which is up and running. I have a very big file (>5Gb) which I want to upload and I don't want to use the upload option but rather sftp or use FileZilla, where should I transfer the file to show up in my History? thanks! stefano Cardinale Stefano Postdoctoral researcher Arkin Lab Department of Bioengineering Energy Biosciences Building 2151 Berkeley Way 94720 - Berkeley

3 2

iterating over files
by mark.rose＠syngenta.com 31 Jan '13

31 Jan '13

Hello I've seen posts from a couple of years ago asking about galaxy's ability to iterate over multiple files. The reply was that the developers were working on it. Have they succeeded and if so how can it now be done. Thanks Mark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.

3 2

how to sftp a file on a local instance of Galaxy
by stefano cardinale 31 Jan '13

31 Jan '13

Hi, I have installed Galaxy on our local server, which is up and running. I have a very big file (>5Gb) which I want to upload and I don't want to use the upload option but rather sftp or use FileZilla, where should I transfer the file to show up in my History? thanks! stefano Cardinale Stefano Postdoctoral researcher Arkin Lab Department of Bioengineering Energy Biosciences Building 2151 Berkeley Way 94720 - Berkeley

2 1