October 2009 - galaxy-user - lists.galaxyproject.org

Re: [galaxy-user] Fwd: starting workflows with a pool of input files
by Matthias Dodt 12 Oct '09

12 Oct '09

Okay Thanks;-)! greetings mat Dominic Tolle schrieb: > fyI > > Begin forwarded message: > >> *From: *Greg Von Kuster <ghv2(a)psu.edu <mailto:ghv2@psu.edu>> >> *Date: *October 8, 2009 3:34:35 PM GMT+02:00 >> *To: *Matthias Dodt <matthias.dodt(a)mdc-berlin.de >> <mailto:matthias.dodt@mdc-berlin.de>> >> *Cc: *galaxy <galaxy-user(a)bx.psu.edu <mailto:galaxy-user@bx.psu.edu>> >> *Subject: **Re: [galaxy-user] starting workflows with a pool of input >> files* >> >> Hello Matthias, >> >> I've opened the following ticket for this question - you can "follow" >> the ticket in bitbucket if you want. >> >> http://bitbucket.org/galaxy/galaxy-central/issue/197/starting-workflows-wit… >> >> Greg Von Kuster >> Galaxy Development Team >> >> >> Matthias Dodt wrote: >>> Hi there! >>> >>> I would like to start a workflow several times (with a pool of input >>> files). >>> >>> The reason is: >>> Lets say we process huge FASTA files. To do so, i incorporated a >>> splitting tool into our galaxy, which creates several smaller >>> split-files, which can be more easily processed on a cluster in >>> parallel. Now i would like to start a workflow with these splitfiles as >>> input (without starting it manually by clicking lets say 100 times - >>> for >>> each split-file - the same procedure). >>> >>> Is there an easy way to do that? >>> >>> thanks! >>> >>> greetings >>> >>> mat >>> _______________________________________________ >>> galaxy-user mailing list >>> galaxy-user(a)bx.psu.edu >>> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >>> >>> >> _______________________________________________ >> galaxy-user mailing list >> galaxy-user(a)bx.psu.edu >> http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user >

1 0

Re: [galaxy-user] Experience with Loading NGS data on standalone instance of galaxy
by Abhishek Pratap 12 Oct '09

12 Oct '09

Hi Greg Based on the testing of the new feature I am not sure if it working as expected. I gave galaxy a path to GERALD folder (which is a analysis folder generated by Illumina pipeline) and selected file type to be just "solexafastq". It took about two hours for the process to be completed. Somehow it went through each file and uploaded everything in the library. Ideally it should look up only the relevant fastq files and upload total of 8 files (1 / lane ) present in the folder. This process should be very fast as it would just record the path. Thanks, -Abhi On Mon, Oct 5, 2009 at 11:37 AM, Abhishek Pratap <abhishek.vit(a)gmail.com>wrote: > I made sure I selected just one file format(fastqsolexa). The path I > gave was to a GERLAD folder. Basically a folder with bunch of analysis > files plus 8 fastq files. I would say it is still taking a long time > to put the path of only 8 fastq files in question to galaxy system. > > -Abhi > > On Mon, Oct 5, 2009 at 11:21 AM, Greg Von Kuster <ghv2(a)psu.edu> wrote: > > This logging looks correct. I replied to your previous message: > regarding > > this issue, are you setting the datatype manually or allowing for > > "auto-detect"? Manually setting the datatype would probably be a good > idea > > to speed things up. If not all files being uploaded are of the same > type, > > pick one and then fix the others later using the "Edit this dataset's > > information" feature. > > > > > > Abhishek Pratap wrote: > >> > >> Hi Greg > >> > >> I hope you had a good weekend. Carrying on from where we left it. I am > >> not able to get to the required page from where I believe I should be > >> able to upload datasets without them getting copied to galaxy > >> filepath. > >> > >> This is what I did > >> > >> 1. Admin -> Data -> Manage Data Libraries > >> 2. Create new data library datasets -> Selected (Upload Files from > >> Filesystem Paths) > >> 3. Pasted the path in the text box > >> 4. Checked the checkbox Copy Data into Galaxy ? NO > >> > >> It is taking very long for the dataset pseudo upload to galaxy > >> filesystem. I can see the following incrementing entries in the log > >> file. Is it the way it is supposed to be ? > >> > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:42,189 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:43,892 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:45,598 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:47,380 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:49,022 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:50,446 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:52,029 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:53,491 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:55,134 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:56,788 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:11:58,433 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:00,112 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:01,752 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:03,712 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:05,525 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:07,235 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:09,062 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:10,864 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:12,914 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:14,567 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:16,240 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> galaxy.tools.actions.upload_common DEBUG 2009-10-05 11:12:18,057 > >> DEBUGDEBUG: In Test_GA_Data_Set, found a folder name match: 8:Temp > >> > >> > >> Thanks, > >> -Abhi > >> > >> > >> > >> On Fri, Oct 2, 2009 at 4:26 PM, Abhishek Pratap <abhishek.vit(a)gmail.com > > > >> wrote: > >>> > >>> I found the error. I had multiple test instance running and I was > >>> using a wrong one to see the changes. My bad. > >>> > >>> -A > >>> > >>> On Fri, Oct 2, 2009 at 4:20 PM, Greg Von Kuster <ghv2(a)psu.edu> wrote: > >>>> > >>>> Hmm...ok, can you attach your universe_wsgi.ini file so I can take a > >>>> look? > >>>> > >>>> > >>>> Abhishek Pratap wrote: > >>>>> > >>>>> I see > >>>>> > >>>>> Analyze Data ¬†Workflow Data Libraries Help User > >>>>> > >>>>> -A > >>>>> > >>>>> On Fri, Oct 2, 2009 at 4:13 PM, Greg Von Kuster <ghv2(a)psu.edu> > wrote: > >>>>>> > >>>>>> What so you see in the top menu bar? > >>>>>> > >>>>>> Do you see this? > >>>>>> > >>>>>> Analyze Data ‚Ä† ‚Ä†Workflow ‚Ä† ‚Ä† ‚Ä† ‚Ä†Data Libraries ‚Ä† ‚Ä† > ‚Ä† > >>>>>> ‚Ä† ‚Ä†Admin ‚Ä† ‚Ä†Help > >>>>>> > >>>>> > >>>> > >> > >> > > > > >

2 3

Registering new users/Datalibraries
by Matthias Dodt 08 Oct '09

08 Oct '09

Hi there! Did anybody gain any experience with this: 1. Is it possible to let the administrator manually confirm each user that registers before he can log into galaxy? 2. We would like to have different Administrators: - for managing users and libraries - for managing (creating) only new libraries and assigning users As i can see so far it is not possible to assign any specific permissions to roles? thanks for any hint! greetings mat

2 4

Password as input parameter
by Jelle Scholtalbers 08 Oct '09

08 Oct '09

Hi, is there a possibility to have the type 'password' in the input parameter section? So that the password the user is typing is not shown on screen? Considering users and passwords, can I use user information (from a user that is logged in) in my tool configuration files? Cheers, Jelle

2 1

Directly extracting results from Galaxy
by Jeff Shrager 05 Oct '09

05 Oct '09

We (BioBike) would like to do to Galaxy what Galaxy does to the UCSC browser, that is, be able to UI+programmatically (i.e., without download/upload of tables) get data from Galaxy to BioBike so that a BioBike user can then carry out next-level-up symbolic analysis on the data that has been molded, folded, and joined by Galaxy. Can someone point me to a description of the API that would enable us to do this? Thanks!

3 4

varying number of output files in XML
by Matthias Dodt 05 Oct '09

05 Oct '09

Hi galaxy-users! I wrote a tool that splits a FASTA file into n output files, each one of a predefined maximum size. The program could return the number of files or a list of filenames... Is it possible to define the number of outputs dynamically (nr of output files dependent on input-filesize)? Thanks! till now i experimented with: <tool id="seqan_splitter_1" name="FASTA splitter"> <description>Splits input files into pieces of desired size</description> <command interpreter="python"> ./tools/RNA-seq/fasta-splitter/fasta-splitter.py --maxsize $size 2> $log_report </command> <inputs> <param name="source" type="data" format="fasta" label="input fasta file"/> <param name="size" type="integer" label="Size in Megabyte of each output file" value="500" optional="false"/> <param name="files" type="hidden" value="10"/> </inputs> <outputs> #for $i < $files <data format="fasta" name="\$i" label="Splitted file"/> #end for <data format="text" name="log_report" label="Detailed log report from splitter"/> </outputs> </tool>

3 3

Sharing histories use case (another rant)
by Assaf Gordon 03 Oct '09

03 Oct '09

Another small usability issue... Use case: I want to share an existing history by link. 1. click on "options" button 2. click on "Stored By you" link (not very intuitive - one needs to read the grayed line ABOVE the link for the complete sentence: "List Your Histories Stored By You" ... usually it is recommended that the link itself contains the verb, like "List Your Histories" ). 3. Scroll to the page containing the desired history, no way to find it if I didn't tag it. 4. find the history, click on the little triangle next to it. 5. from the popup menu, select "share" 6. see the share page, realize this is sharing only by email, not by link - oops. 7. go back to history list. 8. again scroll to the right page. 9. Select the check-box next to the history 10. click on "Enable Import Via Link" 11. You are "jumped" back to page one. (arrgg!!! pages!!!!) 12. scroll again to the right page. 13. see the "importable" link next to the desired history. click it. 14. you get a page of ALL your shared histories, not sorted by any understandable way (time of sharing?) 15. manually search for the right history (by name). 16. copy link. 17. profit! I love galaxy, but sometimes it drives me crazy ;) -gordon.

1 0

"Pages" in history list (short rant)
by Assaf Gordon 03 Oct '09

03 Oct '09

(My galaxy's not working, I have lots of time to write emails...) There's a new "pages" feature on the main galaxy site (was it introduced just yesterday?). It almost gave me a heart attack... There's no clear indication of division to pages (like a big shiny "next" button) and the links to the 1/2/3/4 pages are really tiny. For a second I thought all my old histories were gone. That being said, I really don't like this feature. On my internal galaxy (which is not upgraded yet) I have more than 250 histories. That's 25 pages (!). The histories are not tagged (since tagging is a new feature) - so there's no way of filtering them. Without pagination, all my histories are shown on one page, and I can use my browser's "find" function to look for a history by name (or just sort by date and scroll down the list). Why do I need division to pages ? Please note that almost all web sites that do use pages, offer a global "search" button (not just tag search) - so in practice (if the search engine is good) - there's no need to go to the "next" page. This is not the case in Galaxy. Think about how much work I have to do in order to find a history, If I didn't tag it (or if I forgot the tag name) - way too taxing. Also, the links themselves are very very small - it might make me sound like a grumpy bitter old man (which I am), but navigating galaxy with all those small triangle drop-down menus and small links becomes an exercise in hand-eye coordination and fine motor skills. Not all galaxy users are computer-savvy, they don't all swoosh and pinch and swap (or whatever you mac people do with the mouse). Please keep galaxy easy and accessible. Just my two cents, - gordon.

1 0

How to get a list of coding exons in which each exon is represented once.
by Rehman, Atteeq (NIH/NIDCD) [F] 01 Oct '09

01 Oct '09

Hey everyone, I am a new galaxy user. I am trying to upload coding exons of specific region from ucsc genome browser but I end up with loading coding exons of all splice variants/isoforms. As a result some exons are repeated many times (depending upon how many isoforms the crossponding gene has) which I don't want. Can somebody help me how can I have a list of coding exons in which each exon is represented only once irrespective of how many splice variants/isoforms the gene has? Thanks a lot [cid:image001.jpg@01CA41C8.B3A28E40] Atteeq Ur Rehman Visiting Fellow National Institute on Deafness and Other Communication Disorders, National Institutes of Health, Room 2A-19, 5 Research court, Rockville, MD, USA, 20850. Lab Ph. No. 301-402-9059

2 1

Shortening sequences?
by Thomas Haverkamp 01 Oct '09

01 Oct '09

Hi All, I have a little how to do question and was hoping somebody knows the answer? I have a metagenomic data set with reads of lengths between 100 and 1000 bp. Now I want to create a dataset from my original dataset, with sequences of exact 200bp. I know I can use the filter tool to extract all reads longer than 199bp from the original data set. But then I want to cut off all the sequence bit that is longer than 200bp. so I end up with only a dataset of exactly 200bp. Does anybody know how I can do that in Galaxy. I was thinking of some of the EMBOSS tools, but they only see the first sequence and not all the other sequences in my Fasta file? Any ideas are welcome. Cheers Thomas Dr. Thomas H.A. Haverkamp Centre for Ecological and Evolutionary Synthesis (CEES) Dept. of Biology University of Oslo P.O. Box 1066 Blindern 0316 Oslo Norway Phone: +47 22 85 44 00 Mobile: +47 48 09 49 32 E-mail: thhaverk(a)bio.uio.no Skype: Thomieh73

4 5