Hi Wei, 3Gb is not that large of a file and should upload without issue. How are you uploading the file? - May I suggest that you try placing the file in a web-accessable location and provide the url to the upload tool in the paste box. Galaxy does accept the upload of individual gzipped files which can greatly reduce the time required for the transfer. Thanks, Dan On Apr 6, 2010, at 6:04 PM, Yanwei Tan wrote:
Hi Dan,
Here I met a problem regarding the speed of upload file into Galaxy. Since my data is around 3G fastq file, I uploaded two days ago, the uploading still did not finish yet. I was wondering if I should try the zip compressed file.
Many thanks! Wei
On 4/6/10 5:00 PM, Daniel Blankenberg wrote:
Hi Florent,
Thanks very much for the comments. A sliding window sounds like an excellent approach: allow users to specify the window size, step size, an aggregation action to perform on the window (min, max, sum, mean, etc ), a comparison method (<,<=, ==, etc) and a threshold quality value; allowing users to specify the ends (both or only one or the other) to trim would also likely be useful. Would it also be desirable to allow specifying a number of quality scores that can be excluded from the aggregation action (the zero low quality base pairs in your example)? A window size of 1 would handle the simple case of only trimming the very ends while allowing the user to configure more complex windowing schemes. Thoughts?
Thanks,
Dan
On Apr 6, 2010, at 4:00 AM, Florent Angly wrote:
Thanks for your reply Daniel.
You are correct that there is not currently a tool to trim directly by quality in Galaxy; currently the the Summary statistics and boxplot tools are used to determine good cut off for use in the trim by column tool; percentage of read length can be more useful on variable length reads. However, adding a tool that can directly trim reads based upon a threshold quality score seems like a natural fit for Galaxy, when uniform read length is not present at the start and/or not a requirement at the end and the percentage-of-read-length method is not sufficient
That's right... I did not even think about using the boxplot tool to find how much to trim the ends. My reads all have the same length, but still, is seems more natural to only trim as much as needed and no more. For example, I have some reads that are completely low quality and should entirely trimmed/removed, whereas some might of good quality over almost all their length.
Lets verify that you are looking for something like this, where 'x' is a low quality base and 'o' is a high quality base: Start with: xxxooooxxooooxxx after trimming ends for 'x': ooooxxoooo So that trimming happens only from the ends and stops as soon as a base above the threshold is found and internal low quality bases are not considered.
It's probaby better to use a short sliding window (of, say, 5 bp) and trim the ends until the window has no more than, say zero low quality base pairs. So, the following sequence would be converted from: xxxoxooooooxxooooooxoxxx to: ooooooxxoooooo
Florent
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
-- Yanwei Tan Institute of Neurobiology 1.OG, AG Bading Im Neuenheimer Feld 364 University of Heidelberg 69120 Heidelberg Germany
Tel:+49-6221-548319 Fax:+49-6221-546700
Hi Dan, Many thanks for your reply. I just upload the file from my computer and browse the location. If I upload the zip compressed file, should I choose txtseq.zip format? My data is fastq file in txt file. And how could I put the file in a web-accessible location? You mean the FTP server? Thanks for your help! Wei On 4/7/10 3:53 PM, Daniel Blankenberg wrote:
Hi Wei,
3Gb is not that large of a file and should upload without issue. How are you uploading the file? - May I suggest that you try placing the file in a web-accessable location and provide the url to the upload tool in the paste box.
Galaxy does accept the upload of individual gzipped files which can greatly reduce the time required for the transfer.
Thanks,
Dan
On Apr 6, 2010, at 6:04 PM, Yanwei Tan wrote:
Hi Dan,
Here I met a problem regarding the speed of upload file into Galaxy. Since my data is around 3G fastq file, I uploaded two days ago, the uploading still did not finish yet. I was wondering if I should try the zip compressed file.
Many thanks! Wei
On 4/6/10 5:00 PM, Daniel Blankenberg wrote:
Hi Florent,
Thanks very much for the comments. A sliding window sounds like an excellent approach: allow users to specify the window size, step size, an aggregation action to perform on the window (min, max, sum, mean, etc ), a comparison method (<,<=, ==, etc) and a threshold quality value; allowing users to specify the ends (both or only one or the other) to trim would also likely be useful. Would it also be desirable to allow specifying a number of quality scores that can be excluded from the aggregation action (the zero low quality base pairs in your example)? A window size of 1 would handle the simple case of only trimming the very ends while allowing the user to configure more complex windowing schemes. Thoughts?
Thanks,
Dan
On Apr 6, 2010, at 4:00 AM, Florent Angly wrote:
Thanks for your reply Daniel.
You are correct that there is not currently a tool to trim directly by quality in Galaxy; currently the the Summary statistics and boxplot tools are used to determine good cut off for use in the trim by column tool; percentage of read length can be more useful on variable length reads. However, adding a tool that can directly trim reads based upon a threshold quality score seems like a natural fit for Galaxy, when uniform read length is not present at the start and/or not a requirement at the end and the percentage-of-read-length method is not sufficient
That's right... I did not even think about using the boxplot tool to find how much to trim the ends. My reads all have the same length, but still, is seems more natural to only trim as much as needed and no more. For example, I have some reads that are completely low quality and should entirely trimmed/removed, whereas some might of good quality over almost all their length.
Lets verify that you are looking for something like this, where 'x' is a low quality base and 'o' is a high quality base: Start with: xxxooooxxooooxxx after trimming ends for 'x': ooooxxoooo So that trimming happens only from the ends and stops as soon as a base above the threshold is found and internal low quality bases are not considered.
It's probaby better to use a short sliding window (of, say, 5 bp) and trim the ends until the window has no more than, say zero low quality base pairs. So, the following sequence would be converted from: xxxoxooooooxxooooooxoxxx to: ooooooxxoooooo
Florent
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Hi Wei, To upload a gzipped file (e.g. somefile.fastq.gz), just upload as you normally would (don't set to txtseq.zip - this is a special format), the file will be ungzipped automatically; zip files are not yet supported. As far as a web-accessible location, you can specify http(s) and ftp locations by entering e.g. http://someserver.org/somefile.fastq or ftp://someserver.org/somefile.fastq into the 'paste' box in the upload tool, if you have access this type of resource. Thanks, Dan On Apr 7, 2010, at 11:21 AM, Yanwei Tan wrote:
Hi Dan,
Many thanks for your reply. I just upload the file from my computer and browse the location. If I upload the zip compressed file, should I choose txtseq.zip format? My data is fastq file in txt file. And how could I put the file in a web-accessible location? You mean the FTP server?
Thanks for your help! Wei
On 4/7/10 3:53 PM, Daniel Blankenberg wrote:
Hi Wei,
3Gb is not that large of a file and should upload without issue. How are you uploading the file? - May I suggest that you try placing the file in a web-accessable location and provide the url to the upload tool in the paste box.
Galaxy does accept the upload of individual gzipped files which can greatly reduce the time required for the transfer.
Thanks,
Dan
On Apr 6, 2010, at 6:04 PM, Yanwei Tan wrote:
Hi Dan,
Here I met a problem regarding the speed of upload file into Galaxy. Since my data is around 3G fastq file, I uploaded two days ago, the uploading still did not finish yet. I was wondering if I should try the zip compressed file.
Many thanks! Wei
On 4/6/10 5:00 PM, Daniel Blankenberg wrote:
Hi Florent,
Thanks very much for the comments. A sliding window sounds like an excellent approach: allow users to specify the window size, step size, an aggregation action to perform on the window (min, max, sum, mean, etc ), a comparison method (<,<=, ==, etc) and a threshold quality value; allowing users to specify the ends (both or only one or the other) to trim would also likely be useful. Would it also be desirable to allow specifying a number of quality scores that can be excluded from the aggregation action (the zero low quality base pairs in your example)? A window size of 1 would handle the simple case of only trimming the very ends while allowing the user to configure more complex windowing schemes. Thoughts?
Thanks,
Dan
On Apr 6, 2010, at 4:00 AM, Florent Angly wrote:
Thanks for your reply Daniel.
You are correct that there is not currently a tool to trim directly by quality in Galaxy; currently the the Summary statistics and boxplot tools are used to determine good cut off for use in the trim by column tool; percentage of read length can be more useful on variable length reads. However, adding a tool that can directly trim reads based upon a threshold quality score seems like a natural fit for Galaxy, when uniform read length is not present at the start and/or not a requirement at the end and the percentage-of-read-length method is not sufficient
That's right... I did not even think about using the boxplot tool to find how much to trim the ends. My reads all have the same length, but still, is seems more natural to only trim as much as needed and no more. For example, I have some reads that are completely low quality and should entirely trimmed/removed, whereas some might of good quality over almost all their length.
Lets verify that you are looking for something like this, where 'x' is a low quality base and 'o' is a high quality base: Start with: xxxooooxxooooxxx after trimming ends for 'x': ooooxxoooo So that trimming happens only from the ends and stops as soon as a base above the threshold is found and internal low quality bases are not considered.
It's probaby better to use a short sliding window (of, say, 5 bp) and trim the ends until the window has no more than, say zero low quality base pairs. So, the following sequence would be converted from: xxxoxooooooxxooooooxoxxx to: ooooooxxoooooo
Florent
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
participants (2)
-
Daniel Blankenberg -
Yanwei Tan