May 2011 - galaxy-user - lists.galaxyproject.org

suggestion for multithreading
by Louise-Amélie Schmitt 08 Aug '11

08 Aug '11

Hello everyone, I'm using TORQUE with Galaxy, and we noticed that if a tool is multithreaded, the number of needed cores is not communicated to pbs, leading to job crashes if the required resources are not available when the job is submitted. Therefore I modified a little the code as follows in lib/galaxy/jobs/runners/pbs.py 256 # define PBS job options 257 attrs.append( dict( name = pbs.ATTR_N, value = str( "%s_%s_% s" % ( job_wrapper.job_id, job_wrapper.tool.id, job_wrapper.user ) ) ) ) 258 mt_file = open('tool-data/multithreading.csv', 'r') 259 for l in mt_file: 260 l = string.split(l) 261 if ( l[0] == job_wrapper.tool.id ): 262 attrs.append( dict( name = pbs.ATTR_l, resource = 'nodes', value = '1:ppn='+str(l[1]) ) ) 263 attrs.append( dict( name = pbs.ATTR_l, resource = 'mem', value = str(l[2]) ) ) 264 break 265 mt_file.close() 266 job_attrs = pbs.new_attropl( len( attrs ) + len( pbs_options ) ) (sorry it didn't come out very well due to line breaking) The csv file contains a list of the multithreaded tools, each line containing: <tool id>\t<number of threads>\t<memory needed>\n And it works fine, the jobs wait for their turn properly, but information is duplicated. Perhaps there would be a way to include something similar in galaxy's original code (if it is not already the case, I may not be up-to-date) without duplicating data. I hope that helps :) Best regards, L-A

7 15

Mosaik with Paired Reads
by John David Osborne 08 Aug '11

08 Aug '11

Hello, I'm trying to run Mosaik on our galaxy instance on Ilumina paired reads. However when I selected "paired reads" and Ilumina as an input option, I can still only select one of the two fastq files as input. No 2nd file selector appears like with bwa, bowtie, etc... Can anybody tell me what is going on - is this a known issue? -John

2 1

keeping aditional data in the aaChanges tool
by Ximena Bonilla 08 Aug '11

08 Aug '11

Dear Galaxy staff, I have recently started using your tool and it has been really helpful, thank you! When using Human Genome Variation, aaChanges, I would like to keep some extra lines in the output file from either of the input files. In the tool description it says I should be able to keep them: "...chromosome, start, and end position as well as the SNP. The SNP can be given using ambiguous-nucleotide symbols or a list of two to four alleles separated by '/'. *Any other columns in the first input file will not be used but will be kept for the output*. The second input file contains..." However, I haven't found a way of actually have them in the output file. What am I missing/doing incorrectly? What I've been trying to keep by the way is rs IDs or Ensembl gene IDs. Thank you in advance for your answer. Kind regards, Ximena

2 2

Fwd: Workflows with conditional statements
by Florent Angly 25 Jul '11

25 Jul '11

I filed an enhancement report since if the workflow conditional facility does not appear to exist in Galaxy: https://bitbucket.org/galaxy/galaxy-central/issue/547/conditional-workflow-… Best, Florent -------- Original Message -------- Subject: Workflows with conditional statements Date: Wed, 18 May 2011 10:31:21 +1000 From: Florent Angly <florent.angly(a)gmail.com> To: galaxy-user(a)lists.bx.psu.edu <galaxy-user(a)lists.bx.psu.edu> Hi all, I was wondering if there is a way to put conditional statements in a Galaxy workflow. This would be useful, for example, in the case of a workflow that has an optional advanced option that the user can click. This advanced option would add some extra steps to the data processing. Another example of how this could be useful is if inside a workflow, the data needs to be processed differently based on the results of previous workflow steps. Say, you have a worflow that takes some sequences, and calculate their average length. Using a conditional statement, the workflow would put the data through DeBruijn assembler if the reads are small, but through a traditional Overlap-Layout-Consensus assembler if the reads are long. Are conditional statements possible in Galaxy workflows and I just don't know how to use them? Best, Florent

3 2

RNA-seq Galaxy workflow for PE barcoded samples?
by Whyte, Jeffrey 13 Jul '11

13 Jul '11

Hello, I posted to the seqanswers forum, but have not received any feedback. I am working with RNA-seq Illumina data files in Galaxy (http://main.g2.bx.psu.edu/) The two files are 100bp paired-end reads, multiplexed with barcoding to distinguish samples. The barcodes are the first four bases of the sequences in the s_7_1_sequence.txt file. Would the following Galaxy workflow be correct? 1. Upload both s_7_1_sequence.txt and s_7_2_sequence.txt to Galaxy with the reference genome selected 2. Run NGS: QC and manipulation --> FASTQ Groomer on each file to convert to Sanger FASTQ 3. Run NGS: QC and manipulation --> FASTQ joiner to combine the data from the two files 4. Run FASTX-TOOLKIT FOR FASTQ DATA --> Barcode Splitter to generate separate FASTQ files for each barcode group 5. Run NGS: RNA Analysis --> Tophat to map the reads from each group to the reference genome The problem I am having is that if I select paired-end for the library in Tophat, it requests two FASTQ files. Would I have to use FASTQ Splitter to separate the joined FASTQ files? If there is a more standard way to handle these types of barcoded files, I would appreciate hearing about this workflow. Thanks very much in advance, jjw P.S. Galaxy is an incredibly useful resource. Thanks!

2 1

Workflow API
by Simon Lank 12 Jul '11

12 Jul '11

Hi Again Galaxy team. I'm attempting to use the new workflow API, which I understand is still in development. I created a test workflow with a single input, and was able to use 'example_watch_folder.py' to successfully execute it as a history and get the output to a specified location. My question is how I can modify the script to accept multiple inputs (i.e. how do I define which files in the input folder I want to be each input) and if there's a way to specify runtime parameters. For instance, the workflow I want to execute has a filter step on a tabular input item as one of the later steps which needs to be defined at runtime. How would I specify this in the 'watch_folder.py' parameters? or is this not possible yet? FYI, I know perl but not python, so ultimately I want to wrap the python scripts into a larger perl script to execute recursive workflows. Thanks for all your help. Galaxy is an amazing tool and the workflow API is a fantastic improvement. Simon Simon Lank Research Specialist O'Connor Lab, WNPRC 555 Science Dr. Madison WI (608) 265-3389

2 1

Question regarding quality filtering of 454 amplicons
by Jackie Lighten 01 Jul '11

01 Jul '11

Hi, I have a question for you guys regarding quality filtering. I have a data set of double MID tagged 454 amplicons, from which I wish to select high quality sequences above Q20. The 454 quality filtering system seems to work differently from that given for the Illumina sequencing i.e. 454 filtering takes high quality segments, while Illumina (FASTQ) can select high quality full reads based on certain parameters. OK, so I know that the total length of my amplicon, including primers and barcodes is around 260bp. If I then set the 454 quality filtering tool to extract contiguous high quality sequence of >260, it gives me back around 45% of my raw data as hitting this criterion i.e. All 260bp are above Q20. I don¹t necessarily need this high stringency as most bases may not be informative. But if I convert my 454 data to FASTQ format and then run the Illumina filtering system which also allows me to set the number of bases allowed to deviate from the Q20 criteria, I get back over 90% of my data (allowing 10bp to deviate from Q20). I then need to go ahead and convert back to 454 format. Can you tell me if this is OK? Will I loose /confuse information somewhere along these conversions? It seems that if I do this, my barcodes are removed, as amplicons do not sort properly when I parse them through my barcode filtering program. Does anyone know of a program to filter 454 data based on average sequence quality score, which doesn¹t involve Linux and the Roche off instrument program (I have no experience in Linux! ) Thanks! -- Jack Lighten, Ph.D. Candidate, Bentzen Lab, Room 6078, Department of Biology, Dalhousie University, Halifax, NS, B3H 4J1 Canada Office:(902) 494-1398 Email: Jackie.Lighten(a)Dal.Ca Profile: www.marinebiodiversity.ca/CHONe/Members/lightenj/profile/bio

2 1

error occurred when converting the SOLID output to fastq
by Jia-Xing Yue 29 Jun '11

29 Jun '11

Hi, when I converted the solid output to fastq, the galaxy always complained the following error. Does anybody know why>> ------------- An error occurred running this job:Traceback (most recent call last): File "/galaxy/home/g2main/galaxy_main/tools/next_gen_conversion/solid2fastq.py", line 207, in <module> main() File "/galaxy/home/g2main/galaxy_main/tools/next_gen_conversion/solid2fastq.py", line 188, in main Traceback (most recent call last): File "/galaxy/home/g2main/galaxy_main/tools/next_gen_conversion/solid2fastq.py", line 207, in main() File "/galaxy/home/g2main/galaxy_main/tools/next_gen_conversion/solid2fastq.py", line 188, in main merge_reads_qual( fr, fq, con, trim_name=options.trim_name, out='db', double_encode=options.de, trim_first_base=options.trim_first_base, min_qual=options.min_qual, table_name="f3" ) File "/galaxy/home/g2main/galaxy_main/tools/next_gen_conversion/solid2fastq.py", line 83, in merge_reads_qual cursor.execute('insert into %s values("%s","%s","%s")' % (table_name, defline, lines[0], qual ) ) sqlite3.OperationalError: near "?3": syntax error ----------- Thanks! Jia-Xing -- Jia-Xing Yue Graduate Student Ecology & Evolutionary Bio. -MS 170 Rice University 6100 Main Street Houston TX 77005 Phone: 1-832-360-6228 E-mail: yjx(a)rice.edu Blog: http://bestrok.blogspot.com/

5 4

pileup analysis
by Andrea Edwards 29 Jun '11

29 Jun '11

Hello I was wondering if there was anything available within galaxy that would let you do the following with pileup files: 1) filter for homozygous SNVs (i.e. that do not contain the reference genome allele in the genotype) 2) compare the pileup files for 2 (or more) individuals to find SNVs unique to each individual and to further limit this to homozygous SNPs unique to each individual 3) compare the pileup files for 2 (or more) individuals to find shared SNVs and to further limit to this to shared SNVs where the individuals have different alleles (rare as would assume triallelic snv) and then group the individuals according to the allele for each SNV. I could only see the filter pileup tool but nothing for comparing pileup files thanks

2 1

MACS
by Sher, Falak 14 Jun '11

14 Jun '11

while using MSCS tool from Galaxy I get following mesg. with BED file: "Treatment tags and Control tags are uneven! FDR may be wrong" Any suggestion to fix it.. I don't understand its implication, my data is form single illumina chIP-Seq experiment. I use Bowtie from Galaxy for mapping. F

2 1

Re: [galaxy-user] UCSC->EMBOSS/fuzznuc->UCSC workflow?
by Jennifer Jackson 07 Jun '11

07 Jun '11

Hello Curtis, The BED extraction data can be resolved in Galaxy. Pull out the whole gene and then modify the coordinates in Galaxy to be 10k upstream. To be clear - this coordinate data is going to be used to transform the coordinates in your current fuzznuc output that is transcript-based to be genome-based. The coordinates are not input for fuzznuc - the are used after fuzznuc is run on the fasta file, in order to covert the result coordinates only. This page in the UCSC wiki has a good description of how the UCSC coordinates are organized. http://genomewiki.ucsc.edu/index.php/Coordinate_Transforms The output format for fuzznuc is documented in the tool's help - the last line on the tool form has a link. Hopefully this helps to clear up the suggested processing, Thanks, Jen Galaxy team On 5/17/11 2:08 PM, Robert Curtis Hendrickson wrote: > Jennifer, > > I tried getting data from UCSC as .BED - two issues: > > 1. Unlike "get sequence", I can no longer specify how far upstream I > want - it's EITHER "whole gene" (what's the definition of that!!!) OR > #bp_upstream OR exons OR introns -- with get seq those are not mutually > exclusive - I happen to want the genomic region (5'UTR, exons, introns > 3'UTR + 10kbp upstream of 5'UTR) > > 2. fuzznuc does not recognize BED as a valid input format. So, I can't > run fuzznuc because my BED file doesn’t' show up in the pulldown. > > Indeed, BED files are just annotation, they don’t carry any sequence. > > Have I mis-understood your directions? > > Regards, > > Curtis > > -----Original Message----- > From: Jennifer Jackson [mailto:jen@bx.psu.edu] > Sent: Tuesday, May 17, 2011 11:23 AM > To: Robert Curtis Hendrickson > Cc: 'galaxy-user(a)lists.bx.psu.edu' > Subject: Re: [galaxy-user] UCSC->EMBOSS/fuzznuc->UCSC workflow? > > Hello Curtis, > > No need to use the fasta headers from your original fasta file. > > To obtain the coordinates in BED format: using "Get Data -> UCSC main" > > again to link to the UCSC Table browser, set the same selection criteria > > as for the original fasta sequence, only change the output type to be > > "BED" (instead of "sequence"). Once in your Galaxy history, this format > > will be easier to work with. > > Best, > > Jen > > Galaxy team > > On 5/16/11 9:04 PM, Robert Curtis Hendrickson wrote: > > > Jennifer, > > > > > > Thanks for the outline. I'll try that approach. > > > > > > However, it seems rather painful to have to join the fuzznuc output > back to the original fasta to get at the header information that really > should have been passed along. It would see that there must be a way to > get the data out of UCSC without that space in the fasta header, so that > the chromosome& genomic location get correctly preserved in the fuzznuc > output. Failing that, is there an easy text manipulation that would > convert that fasta header space to a "|"? > > > > > > Regards, > > > Curtis > > > > > > > > > -----Original Message----- > > > From: Jennifer Jackson [mailto:jen@bx.psu.edu] > > > Sent: Monday, May 16, 2011 6:50 PM > > > To: Robert Curtis Hendrickson > > > Cc: 'galaxy-user(a)lists.bx.psu.edu' > > > Subject: Re: [galaxy-user] UCSC->EMBOSS/fuzznuc->UCSC workflow? > > > > > > Hello Curtis, > > > > > > The coordinates of your match are with respect to the fasta sequence, > > > not with respect to the reference genome. Only data mapped to the > > > reference genome can be viewed in the UCSC Browser > > > > > > You will need to calculate from the position of the match in the fasta > > > sequence back through to the reference genome. > > > > > > One suggested way to do this: > > > > > > a) Merge together the original genomic coordinates of the 2kb regions > > > with each line of output from fuzznuc. Use the original source fasta > > > sequence name as the common key for the merge. If both data are in BED > > > format, that would be ideal and make the following steps possible. You > > > may need to split the file based on whether the original fasta sequence > > > came from the positive or negative strand to run "b" and "c" below > > > separately. > > > b) Use "Text Manipulation -> Compute an expression on every row" to > > > create new coordinates. For example, if your data is on the positive > > > strand, and base 1 in your fasta file was genomic coordinate 100, and > > > the alignment from fuzznuc started at base 5 (local coordinate == "4" if > > > in BED format with a zero-based start), then the new genomic start > > > coordinate would be [100 + 4)] = 104. Do this for both start and stop. > > > c) Adjust the logic for "b" if any of your original fasta sequences are > > > from the negative strand, on the negative strand portion of your data > > > ("b" would be run on just the positive strand portion of your data). > > > d) arrange/cut the resulting file down into a standard BED format to > > > remove the local coordinates and keep the genomic coordinates, using the > > > original chromosome names. > > > e) once the logic for the calculations is worked out, save the process > > > into a workflow for use again. > > > > > > Hopefully this helps, > > > > > > Best, > > > > > > Jen > > > Galaxy team > > > > > > On 5/13/11 9:32 AM, Robert Curtis Hendrickson wrote: > > >> Folks, > > >> > > >> I wanted to scan the 2kb upstream of a list of human gene isoforms > for TFBS using fuzznuc. I was able to > > >> "Get Data"> "UCSC Main"> "As sequence" and get my sequences > > >> "EMBOSS"> fuzznuc ran fine, and output the hits > > >> > > >> HOWEVER, fuzznuc lost the genomic position information that UCSC has > put after a space in the sequence headers of the FASTA file. It only > provided offsets within the fasta. > > >> > > >> http://main.g2.bx.psu.edu/u/curtish-uab/h/ucsc-fuzznuc-ucsc-broken > > >> > > >> Thus, when I converted the fuzznuc output back to a BED file and > tried to visualize the hits in UCSC browser, it failed with "invalid BED > File". > > >> I tried fuzznuc with output: seqtable, feattable and gff3, but in > all cases the genomic position was missing, and being a bit of Galaxy > novice, I couldn't figure out how to get the output back to UCSC to > visualize the hits. > > >> > > >> Can anyone tell me how to link up these tools correctly, or share a > history with some other tool set that accomplishes this goal? > > >> > > >> Regards, > > >> Curtis > > >> > > >> Research Associate > > >> Center for Clinical and Translational Science > > >> University of Alabama at Birmingham > > >> > > >> ___________________________________________________________ > > >> The Galaxy User list should be used for the discussion of > > >> Galaxy analysis and other features on the public server > > >> at usegalaxy.org. Please keep all replies on the list by > > >> using "reply all" in your mail client. For discussion of > > >> local Galaxy instances and the Galaxy source code, please > > >> use the Galaxy Development list: > > >> > > >> http://lists.bx.psu.edu/listinfo/galaxy-dev > > >> > > >> To manage your subscriptions to this and other Galaxy lists, > > >> please use the interface at: > > >> > > >> http://lists.bx.psu.edu/ > > > > > -- > > Jennifer Jackson > > http://usegalaxy.org > > http://galaxyproject.org > -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org

2 3

Flagstat on BAM files
by Slim Sassi 03 Jun '11

03 Jun '11

Hello, I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats, but I got results like you see below. It doesn't seem to be working. Any suggestions? 26584869 in total 0 QC failure 0 duplicates 26584869 mapped (100.00%) 0 paired in sequencing 0 read1 0 read2 0 properly paired (-nan%) 0 with itself and mate mapped 0 singletons (-nan%) 0 with mate mapped to a different chr 0 with mate mapped to a different chr (mapQ>=5) Thanks Slim The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

6 11

biomart error
by Christophe Klopp 31 May '11

31 May '11

Hi, I've tried a simple query in biomart : retrieve all gene locations for the Danio rerio in Ensembl 62 and I get an error : An error occurred running this job: /failure preparing job / The same happens with the test site? Cheers Christophe/ / -- ==================================================================== = Christophe KLOPP BIA INRA Toulouse 31326 Castanet-Tolosan = = Tel: 33 5 61 28 50 36 Email: christophe.klopp(a)toulouse.inra.fr = = http://www.sigenae.org/ = = http://bioinfo.genotoul.fr/ = ====================================================================

5 10

Mycoplasma pneumoniae M129 and FH reference genome
by David K Crossman 31 May '11

31 May '11

Hello! I noticed that Mycoplasma pneumonia M129 and FH are not found in the reference genome in Galaxy. Would it be possible to have both of those in there? Thanks, David

2 2

Concatenate fasta tool problem
by Paul McAdam 31 May '11

31 May '11

Hi, I'm trying to use the concatenate fasta tool at http://main.g2.bx.psu.edu/root?tool_id=fasta_concatenate0 to concatenate an xmfa file output from mauve (headers formatted to following standard: >1:517-24215(+)|0402981.fa but am receiving the following error. Traceback (most recent call last): File "/galaxy/home/g2main/galaxy_main/tools/fasta_tools/fasta_concatenate_by_species.py", line 40, in if __name__ == "__main__" : __main__() File "/galaxy/home/g2main/galaxy_main/tools/fasta_tools/fasta_concatenate_by_species.py", line 23, in __main__ species[component.species] = tempfile.TemporaryFile() File "/usr/lib/python2.6/tempfile.py", line 475, in TemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) File "/usr/lib/python2.6/tempfile.py", line 228, in _mkstemp_inner fd = _os.open(file, flags, 0600) OSError: [Errno 24] Too many open files: '/space/g2main/tmpotY_hX' Does anyone have any ideas what the issue may be? Regards Paul -- Laboratory for Bacterial Evolution and Pathogenesis Centre for Infectious Diseases and The Roslin Institute University of Edinburgh Easter Bush Campus Midlothian EH25 9RG Tel: +44 (0)131 6519231 email: p.mcadam(a)sms.ed.ac.uk

2 1

Re: [galaxy-user] (no subject)
by Bossers, Alex 24 May '11

24 May '11

Aaron, Please do check for contaminants.... our experience with service providers and QC....I can write a book probably :(. The FastQC suite is a good place to start (also a galaxy wrapper is available for that). Even for 454 (not having fixed base positions and fixed lengths) it's quite informative (kmer overrepresentation and such). In addition...check for contaminating sequences (ie Coli or Mycoplasma sequences not expected when sequencing human cells.... but you better check ....experience). In the MIRA documentation you will find some info on this prior to assembly filtering as well if I remember correctly. Please keep us posted on your progress. @Peter; hope you manage to take a flight and join the conference. A pity I won't be there but it looks very promising... Alex Van: Aaron Jex [mailto:ajex@unimelb.edu.au] Verzonden: dinsdag 24 mei 2011 8:52 Aan: Bossers, Alex Onderwerp: RE: [galaxy-user] (no subject) Hi Alex, Thanks for the email. I will have to have a closer read of the MIRA documentation I think. I know that it definitely makes use of the quality data to some extent, but I hadn't considered whether it ignores low quality data or not (perhaps there's a threshold setting I could use - I'll check that). I'm not too worried about adaptor sequence at the moment as these "should" be trimmed by our sequencing service, and I clip the ends on the reads when I extract the qual and fasta files from the original sff files anyways. Best regards, Aaron Aaron Jex, BSc, PhD Senior Research Officer, Department of Veterinary Science, The University of Melbourne, 250 Princes Highway, Werribee, Victoria, 3030 tel: +61 3 9731 2294 From: Bossers, Alex [mailto:Alex.Bossers@wur.nl] Sent: Tuesday, 24 May 2011 4:44 PM To: Aaron Jex; galaxy-user(a)bx.psu.edu Subject: RE: [galaxy-user] (no subject) Aaron, As far as I remember MIRA....isn't MIRA taking into account the low/high quality bases anyway? So no need to filter there right? Only filtering needed is for contaminating sequences.....(incl adapters and such). You can/have to check the MIRA website to be sure though. The high qual segments I have used as in the metagenomics example but indeed you loose the exact qual info....but that is already above the provided threshold (default above 20 in Sanger quality score range). Alex Van: galaxy-user-bounces(a)lists.bx.psu.edu [mailto:galaxy-user-bounces@lists.bx.psu.edu] Namens Aaron Jex Verzonden: dinsdag 24 mei 2011 1:40 Aan: galaxy-user(a)bx.psu.edu Onderwerp: [galaxy-user] (no subject) Hi, Can't seem to find an answer to this on your wiki site and it's not in the tutorial. I would like to filter my 454 reads for high quality regions, rename the resulting sequence fragments AND relink the new reads (fragments) to the original quality data so that I can take these filtered reads and assembly them using MIRA. Is there a way to do this with Galaxy? So basically all I want to do is take the new read fragments I get from converting the tabular file to the fasta file as shown in your metagenomics tutorial, and generate a corresponding qual file for these 'new' reads. Best regards, Aaron Aaron Jex, BSc, PhD Senior Research Officer, Department of Veterinary Science, The University of Melbourne, 250 Princes Highway, Werribee, Victoria, 3030 tel: +61 3 9731 2294

1 0

Follow the 2011 Galaxy Community Conference on Twitter
by Dave Clements 24 May '11

24 May '11

Hello all, The 2011 Galaxy Community Conference starts in about 10 hours with the sold out Introduction to Galaxy session, and then gets fully rolling Wednesday morning with the main meeting. See the schedule ( http://galaxy.psu.edu/gcc2011/Programme.html) for full details. If you want to follow the meeting on Twitter, we are encouraging copious tweeting by participants using the #usegalaxy hashtag. You can follow at https://twitter.com/#!/search/%23usegalaxy or http://galaxy.psu.edu/gcc2011/Home.html In addition, NBIC, the conference host, is also sponsoring the videotaping of all 23 talks. These will be placed online after the conference and will be open to anyone. We have 148 registered attendees (about twice as many as last year) form 19 countries (6 continents!) Well over 100 different organizations will have representatives at the meeting. I'm expecting great things. Thanks, Dave C -- http://galaxy.psu.edu/gcc2011/ http://getgalaxy.org http://usegalaxy.org/

1 0

(no subject)
by Aaron Jex 24 May '11

24 May '11

Hi, Can't seem to find an answer to this on your wiki site and it's not in the tutorial. I would like to filter my 454 reads for high quality regions, rename the resulting sequence fragments AND relink the new reads (fragments) to the original quality data so that I can take these filtered reads and assembly them using MIRA. Is there a way to do this with Galaxy? So basically all I want to do is take the new read fragments I get from converting the tabular file to the fasta file as shown in your metagenomics tutorial, and generate a corresponding qual file for these 'new' reads. Best regards, Aaron Aaron Jex, BSc, PhD Senior Research Officer, Department of Veterinary Science, The University of Melbourne, 250 Princes Highway, Werribee, Victoria, 3030 tel: +61 3 9731 2294

3 2

May 20, 2011 Galaxy Development News Brief
by Jennifer Jackson 20 May '11

20 May '11

May 20, 2011 Galaxy Development News Brief http://bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011_… Prior releases: https:bitbucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBriefs ------------ 2011_04_08 Highlights Picard FastQC NBIC Galaxy Hackathon Results Multiple-Input Workflows ------------ How to get this distribution https:bitbucket.org/galaxy/galaxy-dist/ new: % hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist upgrade: % hg pull -u -r 8c11dd28a3cf ------------ Key Upcoming Galaxy Event: GCC 2011 May 24-26, 2011 Galaxy Community Conference Lunteren, The Netherlands http://galaxy.psu.edu/gcc2011 ------------ What's New = NBIC Galaxy Hackathon Results = Two new features (so far) were created and added to Galaxy from the May conference. Hackaton: tool_conf.xml Autogeneration Work from the NBIC Galaxy Hackathon by Rob Hooft, Henk van den Toorn, and Wil Koetsier which adds new optional tags to tool configuration files, and a script which uses these tags to automatically generate tool_conf.xml. The scripts can be found at: * scripts/build_toolbox.py * scripts/extract_toolbox_sections.py Documentation on the new XML tags and how to use the scripts has not yet been prepared. Hackathon: Tool Tags Work from the NBIC Galaxy Hackathon by Freek de Bruijn, Alex Bossers, and Nate Coraor which enables associating tags with tools. This feature requires 'enable_tool_tags = True' since there are some inefficient database operations performed during tool loading to support this feature. Tool authors can specify tags directly in tool configuration files with the new optional <tags> tagset: <tags> <tag>ngs</tag> <tag>mapping</tag> </tags> If used with the tool_conf.xml autogeneration scripts, this will create <tool> tags containing a new "tags" attribute: <tool file="example/example.xml" tags="ngs,mapping"/> Upon startup, Galaxy associates these tags with the tools and presents them in a cloud at the top of the tool menu when unhidden via the tool panel's "Options" menu. = Picard = http://picard.sourceforge.net|http://picard.sourceforge.net New Galaxy tools wrapping the most commonly used **Picard** functions related to metrics and repair of mapped short read sequencing. Metrics include * BAM index statistics (count of mapped reads by reference chromosome) * Alignment summary * Hybrid selection (for targeted data) * Insert size (for paired reads) * Library complexity Repair tools include * Fix mate pair, mark optical/pcr duplicates * Add or replace read groups * SAM/BAM: replace headers and/or reorder based on a different reference = FastQC = http://www.bioinformatics.bbsrc.ac.uk/projects/fastqc/|http://www.bioinform… New tool wrapper generates a comprehensive and useful QC report. Inputs and Outputs * This wrapper will accept any FASTQ, SAM, or BAM file as primary input. It will also take an optional file containing a list of contaminants information, in the form of a tab-delimited file with 2 columns, name and sequence. * The tool produces a single HTML output file that contains all of the results, including the following basic statistics: ** Per base sequence quality ** Per sequence quality scores ** Per base sequence content ** Per base GC content ** Per sequence GC content ** Per base N content ** Sequence Length Distribution ** Sequence Duplication Levels ** Overrepresented sequences http://bytebucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011… = Workflows & Multiple datasets = Workflows can now be run on multiple datasets at the same time. The run workflow page will now show a new stacked dataset icon. http://bytebucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011… Upon clicking that, the selection box changes to a multi-select, and an independent workflow execution will occur for each of these input dataset steps. The rest of the parameters of the workflow will be identical. Combining this functionality with the existing "Send results to a new history" option will send the results of *each* workflow execution to a separate history, numbered sequentially "<name> 1", "<name> 2", etc., where <name> is whatever text you put in the new history name box. http://bytebucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011… Please note that this new type of "multiple-input dataset" step can currently be used only once in any individual workflow. ------------ Updated & Improved = Current Tools = * BAM to SAM tool can now optionally output headers. * GFF,GFF3,GTF related ** Gracefully handle parsing errors in GFFReader and accurately compute raw size of GFF features. ** Enable GFF and GFF3 attributes to be written in GTF format. ** Make Operate on Genomic Intervals (GOPS) intersect and subtract tools compatible with GFF features rather than GFF lines. ** Enable GFF filter attributes tool to accept arbitrary conditions. * Datasource tools: Remove hard-coded special-case handling of UCSC Table Browser and GBrowse datasource tools; functionality remains, but is now a part of the individual tool's XML configuration files. Auto-detect is now available by providing data_type=auto parameters. * Make Cufflinks, Cuffcompare, and Cuffdiff wrappers compatible with v1.0.1 (new option implementation pending). * BWA wrapper enhancement ** The Galaxy BWA wrappers (for Illumina and for SOLiD) were updated for version 0.5.9 of BWA. Three new options have been added to them: Maximum number of alignments to output in the XA tag for reads paired properly (samse/sampe -n); Maximum number of alignments to output in the XA tag for disconcordant read pairs (excluding singletons) (sampe -N); and Specify the read group (samse/sampe -r). ** If read groups are to be specified, the following aspects MUST be set: *** Read group identifier (ID) *** Library name *** Platform/technology used to produce the reads sample ** And the following can be set: *** Sequencing center that produced the read *** Description *** Date that run was produced *** Flow order *** Array of nucleotide bases that correspond to the key sequence of each read *** Programs used for processing the read group *** Predicted median insert size *** Platform unit ** Formerly, when sampse/sampe -n was specified, it would cause BWA to output a format other than SAM. This is no longer the case. The BWA manual can be found at http://bio-bwa.sourceforge.net/bwa.shtml|http://bio-bwa.sourceforge.net/bwa… * SAM header ** For several wrappers where SAM header suppression was optional (BWA, BFAST, Bowtie, SRMA), the default was changed to NOT suppress, however it is still optional. ** Bam-to-sam now keeps the header in the BAM file. * Setting of output dbkey ** Outputs for the following now are correctly set to the relevant dbkey (for reference dbkey whether using built-in or one from history): Freebayes, SRMA, Mosaik, BFAST, Bowtie, BWA, sam-to-bam, and bam-to-sam. = New Tools = * GATK ** note: These tool integrations should be considered alpha. Changes are not necessarily backwards-compatible with workflows or re-run functionality. ** Realigner Target Creator ** Indel Realigner ** Count Covariates ** Table Recalibration ** Analyze Covariates ** Unified Genotyper * Add tool Filter GTF by attribute values list. Tool filters a GTF based on a list of attribute values. The tool is especially useful as a downstream analysis tool for filtering GTF files based on Cuffdiff outputs. = Trackster = * Greatly improve LineTrack performance to fetch optimal amount of data for display. * Add support for Operate on Genomic Intervals (GOPS) intersect and subtract tool. * Enable users not logged in to use tools in shared visualizations. * Add support for static tool select parameters. * Enable datasets that cannot be indexed to be used as tool inputs. * Ensure that reads are drawn in squish and pack modes even when view area is large by setting a minimum width of 1px. * Add histogram mode to feature tracks so that user can generate coverage histogram at any level of data. * Use user preferences when drawing summary tree. * Enable a tool to be run on complete dataset or a visible region. = User Interface (UI) = * Show rerun and info buttons in dataset previews for additional states (e.g. running, queued). * Show details button functional for all tools run within a history, even if currently retired. * Show inheritance chain for datasets expanded to note if source was another history or a library. = CloudMan = * Cloud instance sharing: now share your entire cloud instance deployment (including data, analyses, and/or customizations) with the world or specific users with a click of a button. http://bytebucket.org/galaxy/galaxy-central/wiki/Features/DevNewsBrief/2011… = Source = * Reserved/predefined tool template values ** Tool command line templates may make use of certain variables pre-defined by the Galaxy framework. Some of these already existed but were undocumented. All have been changed to use a common (pythonic) naming scheme, but the old names are retained for backwards compatibility: *** new name = old name (if any) = value description *** __new_file_path__ = universe_wsgi.ini new_file_path value *** __tool_data_path__ = GALAXY_DATA_INDEX_DIR = universe_wsgi.ini tool_data_path value *** __root_dir__ = GALAXY_ROOT_DIR = Top-level Galaxy source directory made absolute via os.path.abspath() *** __datatypes_config__ = GALAXY_DATATYPES_CONF_FILE = universe_wsgi.ini datatypes_config value *** __user_id__ = userId = Email's numeric ID (id column of galaxy_user table in the database) *** __user_email__ = userEmail = User's email address *** __app__ = The galaxy.app.UniverseApplication instance, gives access to *** __app__.config and much more. Should be used as a last resort, may go away in future releases. = Tool Framework = * When label text for a static option in a SelectToolParameter is not provided, default to using the 'value'. * Fix for dynamic options when referencing a DataToolParameter that has already been wrapped. * Only allow a user to rerun if they have access permissions on the dataset. = Test Framework = * Add a "contains" compare type to functional tests. Enables simple checking for substrings in a test output file, on a line-by-line basis. * Fix for expand grouping to allow toolbox tests to use the default parameter value. = Bug Fixes = * Have MACS peak caller wrapper use return code to set error state. Fixes issues seen when MACS was be green, despite encountering fatal errors. * Fix hyperlinks in Cuffcompare and Cuffdiff documentation. * Add support for comment handling to gff_to_interval_index tool. * Workflow Parameter bugfix for the improperly handled case when a parameter isn't used in any workflow step, but should still be available for PJAs. * Bugfix for workflow run not reloading history. * SGE/DRMAA runners did not respect the value set in cluster_files_directory. * Galaxy did not set a public username when 'use_remote_user = True' and did not provide an interface to set it. Upon account creation, Galaxy will now automatically create a public username matching the username portion of the user's email address, with any non-alphanumeric characters replaced with a '-'. If the username is not unique, a '1' is appended, and then incremented until the username is unique. Users may modify their public username via the User menu in the masthead. ------------ = About Galaxy = The Galaxy team is a part of http://www.bx.psu.edu/|BX at http://www.psu.edu/|Penn State, and the http://www.biology.emory.edu/|Biology and http://www.mathcs.emory.edu/|Mathematics and Computer Science departments at http://www.emory.edu/home/index.html|Emory University. Galaxy is supported in part by http://www.nsf.gov/|NSF, http://www.genome.gov/|NHGRI, the http://www.huck.psu.edu/|Huck Institutes of the Life Sciences, and http://www.ics.psu.edu/|The Institute for CyberScience at Penn State, and http://www.emory.edu/home/index.html|Emory University. http://bitbucket.org/galaxy/galaxy-central/wiki/GalaxyTeam|GalaxyTeam http://usegalaxy.org|Use Galaxy! http://galaxyproject.org|GalaxyProject.org Join us at Twitter #usegalaxy http://twitter.com/#!/search/galaxyproject|http://twitter.com/#!/search/gal… ------------ Galaxy Team

1 0

Workflows with conditional statements
by Florent Angly 17 May '11

17 May '11

Hi all, I was wondering if there is a way to put conditional statements in a Galaxy workflow. This would be useful, for example, in the case of a workflow that has an optional advanced option that the user can click. This advanced option would add some extra steps to the data processing. Another example of how this could be useful is if inside a workflow, the data needs to be processed differently based on the results of previous workflow steps. Say, you have a worflow that takes some sequences, and calculate their average length. Using a conditional statement, the workflow would put the data through DeBruijn assembler if the reads are small, but through a traditional Overlap-Layout-Consensus assembler if the reads are long. Are conditional statements possible in Galaxy workflows and I just don't know how to use them? Best, Florent

1 0

Calling SNP's
by sukeerthi teja Rallapalli 17 May '11

17 May '11

Hi, Is it possible to convert a fasta file to sam/bam format using galaxy. ? OR Is is possible to call SNP's using Blast 2.2.25 (Stand Alone Blast) Thank you Teja

4 6

Reload ".loc" files without restarting the Galaxy server?
by Luobin Yang 17 May '11

17 May '11

Hi all, I am wondering if it's possible to to reload a ".loc" file without restarting the Galaxy server? Thanks, Luobin

6 8

problem with filter RNA-seq
by shamsher jagat 17 May '11

17 May '11

I have tried to follow the steps: File 33 in my history is generated by using filter GTF data by attributes. Two files used were file 29 which is a splicing diff file filtered for yes and file 2 is combined GTF. I used TSS-id for filtering. the out put file file 32 is empty. Any suggestion? Jagat, It depends what filter tool you're using and what dataset you're filtering. There is a generic filter tool that can be used to filter Cuffdiff tabular files for either FPKM values and differential expression tests. There is also a tool for filtering GTF files based on a Cuffdiff expr dataset. It sounds like you may be confusing either the tools or the inputs. If after double-checking you're still having problems with filtering, please put together a short list of your analysis steps and share your history with me, and I can take a look. Thanks, J. Further to my question, It appear that there is some problem with the filter option: When I use the isoform/gene exp file as such it work fine but when I filter these files with either parameter such as status if test was successful or on p value it return me empty file. The way am saving the file is - expr file filter save as txt file and upload back in Galaxy. Any suggestion? Jagat

1 1

Re: [galaxy-user] Calling SNP's
by Jennifer Jackson 16 May '11

16 May '11

Hello Teja, Exploring the "Pages" tutorials is recommended as many have SNP-specific analysis examples. I am not sure if your data is RNA or DNA or if you have quality scores available, but there are tutorials that cover most common-use cases. http://main.g2.bx.psu.edu/page/list_published It is recommended to start with these: http://main.g2.bx.psu.edu/u/aun1/p/screencasts http://main.g2.bx.psu.edu/u/james/p/exercises http://main.g2.bx.psu.edu/u/aun1/p/galaxy101 Good luck with your research! Best, Jen Galaxy team ps: For next time, please keep the galaxy-user cc so that our entire team can contribute to replies. This would be especially helpful when you have a new question. On 5/16/11 11:53 AM, sukeerthi teja Rallapalli wrote: > Jen, > > Thank you so much. In the NGS toolbox what option should i choose. > > Regards > Teja -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org

1 0

finding Conserved non coding region
by De Kumar, Bony 16 May '11

16 May '11

Hi all , I have a set of 1500 sequence obtained from my Chip_Seq experiment. I am interested to Know how many of them are within Conserved non coding Region. My definition of conserved Non coding region is as follows 1. Atleast a 200bp region with 80% conservation between Human , Mice and Chicken 2. Atleast a 200bp region with 70% conservation between Human , Mice , Chicken and Fish If someone can suggest me how to use galaxy for this purpose. Thanks in advance. Bony

2 1