- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch
by Sascha Kastens 03 Nov '12

03 Nov '12

Hi Dev-Team, are you planning or maybe working on an update which enables the possiblilty to run workflows in batch mode with paired end data? Cheers, Sascha

3 3

Re: [galaxy-dev] Dataset upload fail
by Hans-Rudolf Hotz 02 Nov '12

02 Nov '12

On 11/02/2012 02:20 PM, Vladimir Yamshchikov wrote: > Someone has advised also that SQLite, which is the default option, > is not adequate for serious work. Is this something I need to > change too? Indeed, see: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Production%20Server > I mean could this prevent dataset upload? I tried to intall > PostgreSQL, but it is quite messy and instructions are far > from beng adequate or even accurate. Although, using SQLite has many disadvantages, I doubt, it has a direct influence on the actual upload process. Well, it has been a while since I last installed PostgreSQL from scratch, but I never ran into troubles. And since PostgreSQL is an Open Source Project as well, there is a lot of support available, see: http://www.postgresql.org/community/ Regards, Hans-Rudolf > > Vladimir > > ----- Original Message ----- > From: "Hans-Rudolf Hotz" <hrh(a)fmi.ch> > To: "Vladimir Yamshchikov" <yaximik(a)gmail.com> > Cc: <galaxy-dev(a)lists.bx.psu.edu> > Sent: Thursday, November 01, 2012 3:18 AM > Subject: Re: [galaxy-dev] Dataset upload fail > > >> Hi Vladimir >> >> On 11/01/2012 03:46 AM, Vladimir Yamshchikov wrote: >>> Local install of Galaxy on SciLinux55. Fails to upload 5.2 GB fastq file >>> from local HD, while normally loading smaller fastq and fasta datasets >>> (less than 1 GB). Chunks of 1.2 GB size remain in */database/tmp, which >>> all represent beginning of the file that fails to upload. Several >>> attempts to upload made and several chunks of the same size are present. >>> Can I just copy the file dataset to database directory instead of >>> uploading through web interface? >>> >> >> >> Make sure you provide a directory for "library_import_dir" in the >> universe_wsgi.ini file. And copy your files to this location (or a sub >> directory of it). This will give you an extra option in the admin menu: >> 'Upload directory of files' and then use the 'Link files without copying >> to Galaxy' option. >> >> >> Alternatively, set "allow_library_path_paste" to 'True' (see also the >> comments and warnings in the universe_wsgi.ini file ) >> >> >> Regards, Hans-Rudolf >> >> >> >> >> >> >> >> >> >

1 0

Filter by quality error
by Scooter Willis 01 Nov '12

01 Nov '12

I have a new workflow setup and filter by quality on an input fastqsanger file. I get the following error message where the original elements worked correctly. The IP address is a compute node. I have one compute node running with autoscaling on. Any reason why the mnt is working? /opt/sge/default/spool/execd/ip-10-140-4-84/job_scripts/39: line 13: /mnt/galaxyTools/tools/fastx_toolkit/0.0.13/env.sh: No such file or directory /opt/sge/default/spool/execd/ip-10-140-4-84/job_scripts/39: line 13: cd: /mnt/galaxyTools/galaxy-central: No such file or directory /opt/sge/default/spool/execd/ip-10-140-4-84/job_scripts/39: line 13: /mnt/galaxyTools/galaxy-central/set_metadata.sh: No such file or directory

1 0

JBrowse direct export to Galaxy
by Erik Derohanian 01 Nov '12

01 Nov '12

Hello, I am a JBrowse Dev hoping to add the ability to export data directly from JBrowse (JavaScript) to Galaxy (without exporting from JBrowse, saving locally, opening Galaxy, loading file from drive). I have looked through the wiki page on Data Sources<http://wiki.g2.bx.psu.edu/Admin/Internals/Data%20Sources>, but JBrowse is client-side only. I was wondering if anybody knew of a different way to do this, or had any pointers. Thanks, Erik Derohanian

4 6

Where should I put a tool's required binaries?
by Joel Rosenberg 01 Nov '12

01 Nov '12

I installed the bedtools tool from the public toolshed to my galaxy cluster. The jobs were failing because I hadn't installed the required binaries: An error occurred running this job: /opt/sge/default/spool/execd/ip-10-194-50-118/job_scripts/2: line 13: genomeCoverageBed: command not found Taking a closer look at the toolshed's page, I noticed there were a handful of required binaries (genomeCoverageBed, intersectBed, etc). So I downloaded BEDTools and compiled all the binaries. I've put them in well organized directories using symlinks to specific versions, etc. The final executable directory containing symlinks to these binaries is "/mnt/galaxyTools/shed_tool_binaries/bin/". I can make a snapshot of the tools volume so that these compiled binaries are always available to me when I bring up my cluster, but how do I integrate them into the PATH in a way that lets the bedtools galaxy tool see them, but also survive cluster shutdowns? Thanks, -Joel

1 0

Re: [galaxy-dev] Empty TopHat output
by Joachim Jacob 01 Nov '12

01 Nov '12

The same here. Cheers, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

3 5

Stop/Start/Reboot/Terminate
by Scooter Willis 01 Nov '12

01 Nov '12

Last night I used the Amazon console to stop my working instance. Today started up the instance using amazon console. Waited appropriate time, using new assigned public ip address and no response. Also did a reboot and no response. I can ssh to the instance but do not know what to check for errors. Should you be able to stop/start and instance in amazon console and have it work correctly? Trying to avoid the "power down" option using the galaxy web interface since I had the problem with new instances being started in a different availability zone where the EBS volume was located. Looks like I will be leaving the master instance running and contributing to amazon profit margins!

2 2

Unable to fetch the sequence ERROR
by Aarthi Talla 01 Nov '12

01 Nov '12

Hello, I am trying to fetch sequences using an interval file with columns 'chr', start, end and name. The names in the chr column are like chr1,chr2...chrY... and for the chrMT and the HSCHR*. I could fetch sequences for chr1-Y but it does not fetch for MT and HSCHR* and GL000*.1 I get the error "4536 warnings, 1st is: Unable to fetch the sequence from '33529672' to '329' for chrom 'HSCHR6_MHC_MCF'." Skipped 4536 invalid lines, 1st is #775, "HSCHR6_MHC_MCF 33529672 33530001 KIFC1". I locally cached the database as Human Hg19. Now does ur Hg19 contain MT and the other GL000*.1 and haplotype sequences in it ? Or do I change the headers of those lines in the interval file accordingly ? Id be glad if you could get back to me asap. Thanks

1 0

Parallelism tag and job splitter
by Edward Hills 01 Nov '12

01 Nov '12

Hi Galaxy-Team, After reading a message on this mailing list about the job splitter I began to investigate what and how this is used. Unfortunately I have been unable to find any documentation on your website for it. Am I blind and missing it or is it yet to be properly documented? Sorry if this turns to be out a pointless exercise, but it would be extremely useful for my Galaxy development. Cheers, Ed

2 5

Re: [galaxy-dev] Fwd: pass more information on a dataset merge
by Alex.Khassapov＠csiro.au 01 Nov '12

01 Nov '12

Hi John, Do you think it's possible to create a <test> for your 'm:" format? I couldn't find how to specify multi input files for the test. -Alex -----Original Message----- From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of John Chilton Sent: Tuesday, 23 October 2012 7:59 AM To: Jorrit Boekel Cc: Khassapov, Alex (CSIRO IM&T, Clayton) Subject: Re: Fwd: [galaxy-dev] pass more information on a dataset merge Hello again Jorrit, Great, I am glad we are largely on the same page here. I don't know when I will get a chance to look at this particular aspect, if you get there first that will be great, if not I will get there eventually. -John On Mon, Oct 22, 2012 at 2:51 AM, Jorrit Boekel <jorrit.boekel(a)scilifelab.se> wrote: > IIRC, I implemented the task_X suffix (galaxy does so as well but to > the split subdirectories) to ensure jobs that contained multiple split > datasets would be run in sync. Files from two datasets that belong > together then get analysed together in subsequent steps. > > It would however be much nicer to retain original file names through a > pipeline or at least the possibility to retrieve them. Since the > split/merge now run actively look and match for files with identical > 'task_x', it may be an option to do: > > fraction1.raw -> fraction1.raw_dataset_43.dat_task_0 -> > fraction1.raw_dataset_44.dat_task_0 > fraction2.raw -> fraction2.raw_dataset_43.dat_task_1 -> > fraction2.raw_dataset_44.dat_task_1 > > (Note that python starts counting at 0, while most researchers number > their first fraction 1.) > > I wouldn't mind looking more into that as well, since it would be a > big improvement UI-wise. > > cheers, > jorrit > > > > > > > On 10/19/2012 04:40 PM, John Chilton wrote: >> >> Jorrit I meant to cc you on this response to Alex. >> >> ---------- Forwarded message ---------- >> From: John Chilton <chil0060(a)umn.edu> >> Date: Fri, Oct 19, 2012 at 9:40 AM >> Subject: Re: [galaxy-dev] pass more information on a dataset merge >> To: Alex.Khassapov(a)csiro.au >> >> >> Hey Alex, >> >> I think the idea here is that your initially uploaded files would >> have different names, but after Jorrit's tool split/merge step they >> will all just be named after the dataset id (see screenshot) so you >> need the task_X at the end so they don't all just have the same name. >> >> I have not thought a whole lot about the naming thing, in general it >> seems like a tough problem and one that Galaxy itself doesn't do a >> particularly good job at. >> >> Jorrit have you given any thought to this? >> >> I wonder if it would be feasible to use the initial uploaded name as >> a sort of prefix going forward. So if I upload say >> >> fraction1.RAW >> fraction2.RAW >> fraction3.RAW >> >> and run a conversion step, maybe I could get: >> >> fraction1_dataset567.ms2 >> fraction2_dataset567.ms2 >> fraction3_dataset567.ms2 >> >> instead of >> >> dataset567.dat_task_0 >> dataset567.dat_task_1 >> dataset567.dat_task_2 >> >> Jorrit do you mind if I give implementing that a shot? It seems like >> it would be a win to me. Am I am going to hit some problem I don't >> see now (presumable we have to send some data from the split to the >> merge and that might be tricky)? >> >> -John >> >> On Thu, Oct 18, 2012 at 7:00 PM, <Alex.Khassapov(a)csiro.au> wrote: >>> >>> Thanks John, >>> >>> I wonder what's the reason for appending _task_XX to the file names, >>> why can't we just keep original file names? >>> >>> Alex >>> >>> -----Original Message----- >>> From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of >>> John Chilton >>> Sent: Friday, 19 October 2012 6:16 AM >>> To: Khassapov, Alex (CSIRO IM&T, Clayton) >>> Subject: Re: [galaxy-dev] pass more information on a dataset merge >>> >>> On Tue, Oct 16, 2012 at 11:11 PM, <Alex.Khassapov(a)csiro.au> wrote: >>>> >>>> Hi John, >>>> >>>> I am definitely interested in this idea, not only me - we are >>>> currently working on moving a few scientific tools (not related to >>>> genome) into cloud using Galaxy. >>> >>> Great. My interests in Galaxy are mostly outside of genomics as >>> well, it is good to have more people utilizing Galaxy in this way >>> because it will force the platform to become more generic and >>> address more broader use cases. >>> >>>> We will try it further and see if we need any changes. For now one >>>> improvement would be nice, make dataset_id.dat contain list of >>>> paths to the location of the uploaded files, so by displaying html >>>> page the user could just click on the link and download the file. >>>> >>> Code that attempted to do this was in there, but didn't work >>> obviously. I have now fixed it up. >>> >>> Thanks for beta testing. >>> >>> -John >>> >>>> We are pretty new to Galaxy, so our understanding of Galaxy is >>>> pretty limited. >>>> >>>> Thanks again, >>>> >>>> Alex >>>> >>>> >>>> -----Original Message----- >>>> From: jmchilton(a)gmail.com [mailto:jmchilton@gmail.com] On Behalf Of >>>> John Chilton >>>> Sent: Wednesday, 17 October 2012 3:21 AM >>>> To: Khassapov, Alex (CSIRO IM&T, Clayton) >>>> Subject: Re: [galaxy-dev] pass more information on a dataset merge >>>> >>>> Wow, thanks for the rapid feedback! I have made the changes you >>>> have suggested. It seems you must be interested in this >>>> idea/implementation. Let me know if you have specific use >>>> cases/requirements in mind and/or if you would be interested in write access to the repository. >>>> >>>> -John >>>> >>>> On Mon, Oct 15, 2012 at 11:51 PM, <Alex.Khassapov(a)csiro.au> wrote: >>>>> >>>>> Hi John, >>>>> >>>>> I tried your galaxy-central-homogeneous-composite-datatypes >>>>> implementation, works great thank you (and Jorrit). >>>>> >>>>> A couple of fixes: >>>>> >>>>> 1. Add multi_upload.xml to too_conf.xml 2. >>>>> lib/galaxy/tools/parameters/grouping.py line 322 (in >>>>> get_filenames( context )) - >>>>> "if ftp_files is not None:" >>>>> Remove "is not None" as ftp_files is empty [], but not None, >>>>> then line 331 "user_ftp_dir = os.path.join( >>>>> trans.app.config.ftp_upload_dir, trans.user.email )" throws an exeption if ftp_upload_dir isn't set. >>>>> >>>>> Alex >>>>> >>>>> -----Original Message----- >>>>> From: galaxy-dev-bounces(a)lists.bx.psu.edu >>>>> [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of John >>>>> Chilton >>>>> Sent: Tuesday, 16 October 2012 1:07 AM >>>>> To: Jorrit Boekel >>>>> Cc: galaxy-dev(a)lists.bx.psu.edu >>>>> Subject: Re: [galaxy-dev] pass more information on a dataset merge >>>>> >>>>> Here is an implementation of the implicit multi-file composite >>>>> datatypes piece of that idea. I think the implicit parallelism may >>>>> be harder. >>>>> >>>>> https://bitbucket.org/galaxyp/galaxy-central-homogeneous-composite >>>>> -da >>>>> t >>>>> atypes/compare >>>>> >>>>> Jorrit do you have any objection to me trying to get this included >>>>> in galaxy-central (this is 95% code I stole from you)? I made the >>>>> changes against a clean galaxy-central fork and included nothing >>>>> proteomics specific in anticipation of trying to do that. I have >>>>> talked with Jim Johnson about the idea and he believes it would be >>>>> useful his mothur metagenomics tools, so the idea is valuable outside of proteomics. >>>>> >>>>> Galaxy team, would you be okay with including this and if so is >>>>> there anything you would like to see either at a high level or at >>>>> the level of the actual implementation. >>>>> >>>>> -John >>>>> >>>>> ------------------------------------------------ >>>>> John Chilton >>>>> Senior Software Developer >>>>> University of Minnesota Supercomputing Institute >>>>> Office: 612-625-0917 >>>>> Cell: 612-226-9223 >>>>> Bitbucket: https://bitbucket.org/jmchilton >>>>> Github: https://github.com/jmchilton >>>>> Web: http://jmchilton.net >>>>> >>>>> On Mon, Oct 8, 2012 at 9:24 AM, John Chilton <chilton(a)msi.umn.edu> >>>>> wrote: >>>>>> >>>>>> Jim Johnson and I have been discussing that approach to handling >>>>>> fractionated proteomics samples as well (composite datatypes, not >>>>>> the specifics of the interface for parallelizing). >>>>>> >>>>>> My perspective has been that Galaxy should be augmented with >>>>>> better native mechanisms for grouping objects in histories, >>>>>> operating over those groups, building workflows that involve >>>>>> arbitrary numbers of inputs, etc... Composite data types are >>>>>> kindof a kludge, I think they are more useful for grouping HTML >>>>>> files together when you don't care about operating on the >>>>>> constituent parts you just want to view pages a as a report or >>>>>> something. With this proteomic data we are working with, the individual pieces are really interesting right? >>>>>> You want to operate on the individual pieces with the full array >>>>>> of tools (not just these special tools that have the logic for >>>>>> dealing with the composite datatypes), you want to visualize the >>>>>> files, etc... Putting these component pieces in the composite >>>>>> data type extra_files path really limits what you can do with the >>>>>> pieces in Galaxy. >>>>>> >>>>>> I have a vague idea of something that I think could bridge some >>>>>> of the gaps between the approaches (though I have no clue on the >>>>>> feasibility). Looking through your implementation on bitbucket it >>>>>> looks like you are defining your core datatypes (MS2, >>>>>> CruxSequest) as subclasses of this composite data type >>>>>> (CompositeMultifile). My recommendation would be to try to define >>>>>> plain datatypes for these core datatype (MS2, CruxSequest) and >>>>>> then have the separate composite datatype sort of delegate to the plain datatypes. >>>>>> >>>>>> You could then continue to explicitly declare subclasses of the >>>>>> composite datatype (maybe MS2Set, CruxSequestSet), but also maybe >>>>>> augement the tool xml so you can do implicit data type instances >>>>>> the way you can with tabular data for instance (instead of >>>>>> defining columns you would define the datatype to delegate to). >>>>>> >>>>>> The next step would be to make the parallelism implicit (i.e pull >>>>>> it out of the tool wrapper). Your tool wrappers wouldn't >>>>>> reference the composite datatypes, they would reference the >>>>>> simple datatypes, but you could add a little icon next to any >>>>>> input that let you replace a single input with a composite input >>>>>> for that type. It would be kind of like the run workflow page >>>>>> where you can replace an input with a multiple inputs. If a >>>>>> composite input (or inputs) are selected the tool would then produce composite outputs. >>>>>> >>>>>> For the steps that actually combine multiple inputs, I think in >>>>>> your case this is perculator maybe (a tool like interprophet or >>>>>> Scaffold that merges peptide probabilities across runs and groups >>>>>> proteins), then you could have the same sort of implicit >>>>>> replacement but instead of for single inputs it could do that for >>>>>> multi-inputs (assuming the Galaxy powers that be accept my fixes >>>>>> for multi-input tool parameters: >>>>>> >>>>>> https://bitbucket.org/galaxy/galaxy-central/pull-request/76/multi-input-dat…) >>>>>> >>>>>> The upshot of all of that would be that then even if these >>>>>> composites datatypes aren't used widely, other people could still >>>>>> use your proteomics tools (my users are definitely interested in >>>>>> Crux for >>>>>> instance) and you could then use other developers' proteomic >>>>>> tools with your composite datatypes even though they weren't >>>>>> designed with that use case in mind (I have msconvert, myrimatch, >>>>>> idpicker, proteinpilot, Ira Cooke has X! Tandem, OMSSA, TPP, and >>>>>> NBIC has an entire suite of label free quant tools). A third >>>>>> benefit would be that people working in other -omicses could make >>>>>> use of the homogenous composite datatype implementation without >>>>>> needing to rewrite their wrappers and datatypes. >>>>>> >>>>>> There is probably something that I am missing that makes this >>>>>> very difficult, let me know if you think this is a good idea and >>>>>> what its feasibility might be. I forked your repo and set off to >>>>>> try to implement some of this stuff last week and I ended up with >>>>>> my galaxy pull requests to improve batching workflows and >>>>>> multi-input tool parameters instead, but I hope to eventually get around to it. >>>>>> >>>>>> -John >>>>>> >>>>>> ------------------------------------------------ >>>>>> John Chilton >>>>>> Senior Software Developer >>>>>> University of Minnesota Supercomputing Institute >>>>>> Office: 612-625-0917 >>>>>> Cell: 612-226-9223 >>>>>> Bitbucket: https://bitbucket.org/jmchilton >>>>>> Github: https://github.com/jmchilton >>>>>> Web: http://jmchilton.net >>>>>> >>>>>> On Mon, Oct 1, 2012 at 8:24 AM, Jorrit Boekel >>>>>> <jorrit.boekel(a)scilifelab.se> wrote: >>>>>>> >>>>>>> Dear list, >>>>>>> >>>>>>> I thought I was working with fairly large datasets, but they >>>>>>> have recently started to include ~2Gb files in sets of >50. I >>>>>>> have ran these sort of things before as merged data by using tar >>>>>>> to roll them up in one set, but when dealing with >100Gb >>>>>>> tarfiles, Galaxy on EC2 seems to get very slow, although that's >>>>>>> probably because of my implementation of dataset type detection >>>>>>> (untar and read through files). >>>>>>> >>>>>>> Since tarring/untarring isn't very clean, I want to switch from >>>>>>> tarring to creating composite files on merge by putting a tool's >>>>>>> results into the dataset.extra_files_path. This doesn't seem to >>>>>>> be supported yet, because we currently pass in do_merge the >>>>>>> output dataset.filename to the respective datatype's merge method. >>>>>>> >>>>>>> I would like to pass more data to the merge method (let's say >>>>>>> the whole dataset object) to be able to get the composite files >>>>>>> directory and 'merge' >>>>>>> the files in there. Good idea, bad idea? If anyone has views on >>>>>>> this, I'd love to hear them. >>>>>>> >>>>>>> cheers, >>>>>>> jorrit >>>>>>> >>>>>>> ___________________________________________________________ >>>>>>> Please keep all replies on the list by using "reply all" >>>>>>> in your mail client. To manage your subscriptions to this and >>>>>>> other Galaxy lists, please use the interface at: >>>>>>> >>>>>>> http://lists.bx.psu.edu/ >>>>> >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this and >>>>> other Galaxy lists, please use the interface at: >>>>> >>>>> http://lists.bx.psu.edu/ > > > > -- > Scientific programmer > Mass spec analysis support @ BILS > Janne Lehtiö / Lukas Käll labs > SciLifeLab Stockholm >

1 0