Initial QC and grooming for Illumina HiSeq2000 paired end RNAseq data
I am trying to do RNAseq analysis on Paired end data from the Hiseq2000. I have about 50 files for each sample (25 forward and 25 reverse - although each sample has a different number of files). I think that I need to: -convert them into FASTQ sanger format using the FASTSQ groomer tool -check the quality using the FASTQqc tool I don't know how to handle this many files. Do I have to groom and run the QC for each file? Should I join the paired files and run both tools on each pair, or should I combine all of the data for each sample (which I don't know how to do) and then groom and run the QC for all of the reads for the sample. Thanks in advance for advice Lindsey
Hello Lindsey, Yes, you have this correct. The general path would be to: - join forward and reverse data per run - run FASTQ Groomer & FastQC (note: if your data is already in Sanger FASTQ format with Phred+33 quality scaled values, the datatype '.fastqsanger' can be directly assigned and the FASTQ Groomer step skipped. This is likely true if your data is a from the latest CASAVA pipeline, but please double check.) - discard data as needed based on quality - split forward and reverse data that passes QC - concatenate all forward reads from a sample into one FASTQ file - concatenate all reverse reads from a sample into one FASTQ file. - for each sample, run TopHat using the two concatenated FASTQ files To manipulate paired end data, please see the tools -> NGS: QC and manipulation: FASTQ splitter & FASTQ joiner. To combined data files head-to-tail from multiple runs into a single FASTQ file please see the tool -> Text Manipulation: Concatenate datasets. I am not sure of the actual volume of data, but if these start to get large or TopHat errors with a memory problem, a local or cluster instance would be the recommendation: http://getgalaxy.org For reference: http://tophat.cbcb.umd.edu/manual.html http://www.nature.com/nprot/journal/v7/n3/full/nprot.2012.016.html Hopefully this helps. Others are welcome to post comments/suggestions. Jen Galaxy team On 7/2/12 11:17 AM, Lindsey Kelly wrote:
I am trying to do RNAseq analysis on Paired end data from the Hiseq2000. I have about 50 files for each sample (25 forward and 25 reverse - although each sample has a different number of files).
I think that I need to:
-convert them into FASTQ sanger format using the FASTSQ groomer tool
-check the quality using the FASTQqc tool
I don't know how to handle this many files. Do I have to groom and run the QC for each file? Should I join the paired files and run both tools on each pair, or should I combine all of the data for each sample (which I don't know how to do) and then groom and run the QC for all of the reads for the sample.
Thanks in advance for advice
Lindsey
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
Good Morning, I confirmed with Illumina that my reads are in Sanger FASTQ format and used edit attributes to change them to fastqsanger. Then I joined the forward and reverse and ran the FastQ Joiner and the FastQC on the joined data file. I'm getting the following error: ## odpath=None: No output found in None. Output for the run was: Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/space/g2main Started analysis of FASTQ Analysis complete for FASTQ Failed to process file FASTQ java.lang.IllegalArgumentException: No known encodings with chars < 33 (Yours was ) at uk.ac.bbsrc.babraham.FastQC.Sequence.QualityEncoding.PhredEncoding.getFastQEncodingOffset(PhredEncoding.java:33) at uk.ac.bbsrc.babraham.FastQC.Modules.PerBaseQualityScores.getPercentages(PerBaseQualityScores.java:70) at uk.ac.bbsrc.babraham.FastQC.Modules.PerBaseQualityScores.raisesError(PerBaseQualityScores.java:164) at uk.ac.bbsrc.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:194) at uk.ac.bbsrc.babraham.FastQC.Report.HTMLReportArchive.(HTMLReportArchive.java:59) at uk.ac.bbsrc.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:157) at uk.ac.bbsrc.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:108) at java.lang.Thread.run(Thread.java:662) I thought that perhaps this was because I just changed the file format and maybe they really aren't in Fastqsanger format, so I: changed the format back to fastq, ran the groomer, joined them and then ran the FastQC and received the same error. I also tried to join them first and then run the groomer, but the files don't show up if I haven't changed the attributes to Fastqsanger (which makes sense if Galaxy only uses fastqsanger files). I'm not sure what to try next. Thank you Lindsey On Wed, Jul 4, 2012 at 3:53 PM, Jennifer Jackson <jen@bx.psu.edu> wrote:
Hello Lindsey,
Yes, you have this correct. The general path would be to:
- join forward and reverse data per run - run FASTQ Groomer & FastQC (note: if your data is already in Sanger FASTQ format with Phred+33 quality scaled values, the datatype '.fastqsanger' can be directly assigned and the FASTQ Groomer step skipped. This is likely true if your data is a from the latest CASAVA pipeline, but please double check.) - discard data as needed based on quality - split forward and reverse data that passes QC - concatenate all forward reads from a sample into one FASTQ file - concatenate all reverse reads from a sample into one FASTQ file. - for each sample, run TopHat using the two concatenated FASTQ files
To manipulate paired end data, please see the tools -> NGS: QC and manipulation: FASTQ splitter & FASTQ joiner.
To combined data files head-to-tail from multiple runs into a single FASTQ file please see the tool -> Text Manipulation: Concatenate datasets.
I am not sure of the actual volume of data, but if these start to get large or TopHat errors with a memory problem, a local or cluster instance would be the recommendation: http://getgalaxy.org
For reference: http://tophat.cbcb.umd.edu/manual.html http://www.nature.com/nprot/journal/v7/n3/full/nprot.2012.016.html
Hopefully this helps. Others are welcome to post comments/suggestions.
Jen Galaxy team
On 7/2/12 11:17 AM, Lindsey Kelly wrote:
I am trying to do RNAseq analysis on Paired end data from the Hiseq2000. I have about 50 files for each sample (25 forward and 25 reverse - although each sample has a different number of files).
I think that I need to:
-convert them into FASTQ sanger format using the FASTSQ groomer tool
-check the quality using the FASTQqc tool
I don't know how to handle this many files. Do I have to groom and run the QC for each file? Should I join the paired files and run both tools on each pair, or should I combine all of the data for each sample (which I don't know how to do) and then groom and run the QC for all of the reads for the sample.
Thanks in advance for advice
Lindsey
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jacksonhttp://galaxyproject.org
-- Lindsey M Kelly University of Pittsburgh Cellular and Molecular Pathology Program Yuri Nikiforov Laboratory 412-578-9208
Hello Lindsey, Would you please send in a bug report from the FastQC error dataset that resulted from the run where you included running the Groomer tool first? This will help us to track down the problem. Please be sure to leave all datasets in your history for this run undeleted and in the original history that they were executed in. If you need to undelete, use "Option (gear icon) -> View Deleted Datasets", the click on the links to undelete. We will need the original inputs and full path through tools run in Galaxy. http://wiki.g2.bx.psu.edu/Support#Reporting_tool_errors Please include this email address in the bug report notes if your Galaxy account uses a different one. Thanks and I will watch for your bug report, Jen Galaxy team On 7/11/12 5:32 AM, Lindsey Kelly wrote:
Good Morning,
I confirmed with Illumina that my reads are in Sanger FASTQ format and used edit attributes to change them to fastqsanger. Then I joined the forward and reverse and ran the FastQ Joiner and the FastQC on the joined data file. I'm getting the following error:
## odpath=None: No output found in None. Output for the run was:
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/space/g2main Started analysis of FASTQ Analysis complete for FASTQ Failed to process file FASTQ java.lang.IllegalArgumentException: No known encodings with chars < 33 (Yours was ) at uk.ac.bbsrc.babraham.FastQC.Sequence.QualityEncoding.PhredEncoding.getFastQEncodingOffset(PhredEncoding.java:33) at uk.ac.bbsrc.babraham.FastQC.Modules.PerBaseQualityScores.getPercentages(PerBaseQualityScores.java:70) at uk.ac.bbsrc.babraham.FastQC.Modules.PerBaseQualityScores.raisesError(PerBaseQualityScores.java:164) at uk.ac.bbsrc.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:194) at uk.ac.bbsrc.babraham.FastQC.Report.HTMLReportArchive.(HTMLReportArchive.java:59) at uk.ac.bbsrc.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:157) at uk.ac.bbsrc.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:108) at java.lang.Thread.run(Thread.java:662)
I thought that perhaps this was because I just changed the file format and maybe they really aren't in Fastqsanger format, so I: changed the format back to fastq, ran the groomer, joined them and then ran the FastQC and received the same error.
I also tried to join them first and then run the groomer, but the files don't show up if I haven't changed the attributes to Fastqsanger (which makes sense if Galaxy only uses fastqsanger files).
I'm not sure what to try next.
Thank you Lindsey
On Wed, Jul 4, 2012 at 3:53 PM, Jennifer Jackson <jen@bx.psu.edu <mailto:jen@bx.psu.edu>> wrote:
Hello Lindsey,
Yes, you have this correct. The general path would be to:
- join forward and reverse data per run - run FASTQ Groomer & FastQC (note: if your data is already in Sanger FASTQ format with Phred+33 quality scaled values, the datatype '.fastqsanger' can be directly assigned and the FASTQ Groomer step skipped. This is likely true if your data is a from the latest CASAVA pipeline, but please double check.) - discard data as needed based on quality - split forward and reverse data that passes QC - concatenate all forward reads from a sample into one FASTQ file - concatenate all reverse reads from a sample into one FASTQ file. - for each sample, run TopHat using the two concatenated FASTQ files
To manipulate paired end data, please see the tools -> NGS: QC and manipulation: FASTQ splitter & FASTQ joiner.
To combined data files head-to-tail from multiple runs into a single FASTQ file please see the tool -> Text Manipulation: Concatenate datasets.
I am not sure of the actual volume of data, but if these start to get large or TopHat errors with a memory problem, a local or cluster instance would be the recommendation: http://getgalaxy.org
For reference: http://tophat.cbcb.umd.edu/manual.html http://www.nature.com/nprot/journal/v7/n3/full/nprot.2012.016.html
Hopefully this helps. Others are welcome to post comments/suggestions.
Jen Galaxy team
On 7/2/12 11:17 AM, Lindsey Kelly wrote:
I am trying to do RNAseq analysis on Paired end data from the Hiseq2000. I have about 50 files for each sample (25 forward and 25 reverse - although each sample has a different number of files).
I think that I need to:
-convert them into FASTQ sanger format using the FASTSQ groomer tool
-check the quality using the FASTQqc tool
I don't know how to handle this many files. Do I have to groom and run the QC for each file? Should I join the paired files and run both tools on each pair, or should I combine all of the data for each sample (which I don't know how to do) and then groom and run the QC for all of the reads for the sample.
Thanks in advance for advice
Lindsey
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server atusegalaxy.org <http://usegalaxy.org>. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Jennifer Jackson http://galaxyproject.org
--
Lindsey M Kelly University of Pittsburgh Cellular and Molecular Pathology Program Yuri Nikiforov Laboratory 412-578-9208
-- Jennifer Jackson http://galaxyproject.org
participants (2)
-
Jennifer Jackson -
Lindsey Kelly