Dear Officer,
I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA.
Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan.
I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours.
in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes.
For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser.
Cheers Kevin
On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan asifullah111@gmail.comwrote:
Dear Officer,
I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA.
Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan.
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Dear Asifullah
As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally,
try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over
also you can alternatively use a perl or python script locally
http://seqanswers.com/forums/showthread.php?t=269
best Abhay Krishna
On 18 August 2010 08:47, Kevin Lam aboulia@gmail.com wrote:
I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours.
in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes.
For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser.
Cheers Kevin
On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan asifullah111@gmail.comwrote:
Dear Officer,
I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA.
Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan.
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main.
-Farhat
Hi Farhat and others,
Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome.
--nate
Farhat Habib wrote:
I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main.
-Farhat
-- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhat
Office: +91 20 2590 8015 Cell: +91 973 0996 233
On Wed, Aug 18, 2010 at 12:25 PM, Abhay Krishna <abhay.krishna@gmail.com mailto:abhay.krishna@gmail.com> wrote:
Dear Asifullah As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally, try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over also you can alternatively use a perl or python script locally http://seqanswers.com/forums/showthread.php?t=269 best Abhay Krishna On 18 August 2010 08:47, Kevin Lam <aboulia@gmail.com <mailto:aboulia@gmail.com>> wrote: I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan <asifullah111@gmail.com <mailto:asifullah111@gmail.com>> wrote: Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan. _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files.
It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some.
with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver?
Cheers Kevin
On Thu, Aug 19, 2010 at 3:42 AM, Nate Coraor nate@bx.psu.edu wrote:
Hi Farhat and others,
Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome.
--nate
Farhat Habib wrote:
I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main.
-Farhat
-- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhat http://www.iiserpune.ac.in/%7Efarhat
Office: +91 20 2590 8015 Cell: +91 973 0996 233
On Wed, Aug 18, 2010 at 12:25 PM, Abhay Krishna <abhay.krishna@gmail.commailto: abhay.krishna@gmail.com> wrote:
Dear Asifullah
As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally,
try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over
also you can alternatively use a perl or python script locally
http://seqanswers.com/forums/showthread.php?t=269
best Abhay Krishna
On 18 August 2010 08:47, Kevin Lam <aboulia@gmail.com mailto:aboulia@gmail.com> wrote:
I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan <asifullah111@gmail.com <mailto:asifullah111@gmail.com>> wrote: Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan. _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
The 2 GB is not a Galaxy limitation. In fact, web browsers are in general uncapable of uploading very large files. See these links: http://kb.globalscape.com/Print10600.aspx http://www.motobit.com/help/ScptUtl/pa98.htm HTML is just not the best way to move big files around... Florent
On 19/08/10 15:51, Kevin Lam wrote:
Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files.
It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some.
with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver?
Cheers Kevin
On Thu, Aug 19, 2010 at 3:42 AM, Nate Coraor <nate@bx.psu.edu mailto:nate@bx.psu.edu> wrote:
Hi Farhat and others, Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome. --nate Farhat Habib wrote: I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main. -Farhat -- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhat <http://www.iiserpune.ac.in/%7Efarhat> Office: +91 20 2590 8015 Cell: +91 973 0996 233 On Wed, Aug 18, 2010 at 12:25 PM, Abhay Krishna <abhay.krishna@gmail.com <mailto:abhay.krishna@gmail.com> <mailto:abhay.krishna@gmail.com <mailto:abhay.krishna@gmail.com>>> wrote: Dear Asifullah As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally, try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over also you can alternatively use a perl or python script locally http://seqanswers.com/forums/showthread.php?t=269 best Abhay Krishna On 18 August 2010 08:47, Kevin Lam <aboulia@gmail.com <mailto:aboulia@gmail.com> <mailto:aboulia@gmail.com <mailto:aboulia@gmail.com>>> wrote: I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan <asifullah111@gmail.com <mailto:asifullah111@gmail.com> <mailto:asifullah111@gmail.com <mailto:asifullah111@gmail.com>>> wrote: Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan. _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> <mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu>> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> <mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu>> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> <mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu>> http://lists.bx.psu.edu/listinfo/galaxy-user ------------------------------------------------------------------------ _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Hi Kevin,
step into your directory and execute: python -m SimpleHTTPServer 8082 Your files will be accesible from: http://ipaddress:8082/
Source: http://sysadmingeek.com/articles/serve-up-files-easily-with-the-python-simpl...
Cheers, Jelle
On Thu, Aug 19, 2010 at 7:51 AM, Kevin Lam aboulia@gmail.com wrote:
Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files.
It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some.
with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver?
Cheers Kevin
On Thu, Aug 19, 2010 at 3:42 AM, Nate Coraor nate@bx.psu.edu wrote:
Hi Farhat and others,
Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome.
--nate
Farhat Habib wrote:
I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main.
-Farhat
-- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhathttp://www.iiserpune.ac.in/%7Efarhat
Office: +91 20 2590 8015 Cell: +91 973 0996 233
On Wed, Aug 18, 2010 at 12:25 PM, Abhay Krishna <abhay.krishna@gmail.commailto: abhay.krishna@gmail.com> wrote:
Dear Asifullah
As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally,
try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over
also you can alternatively use a perl or python script locally
http://seqanswers.com/forums/showthread.php?t=269
best Abhay Krishna
On 18 August 2010 08:47, Kevin Lam <aboulia@gmail.com mailto:aboulia@gmail.com> wrote:
I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan <asifullah111@gmail.com <mailto:asifullah111@gmail.com>> wrote: Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan. _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.eduhttp://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Kevin Lam wrote:
Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files.
It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some.
Hi Kevin,
Thanks for the suggestion. The donwside I see here is that it'd be a two step process to get it to S3 and then from there to Galaxy (and in addition, it's not free). Ideally we'd need a method to upload to Galaxy directly, but without the browser flakiness.
Of course we can do this now with something like FTP, but a browser based solution would be more ideal.
with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver?
Not for history uploads, but you can with Library uploads (and then import the library item into your history):
http://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFiles
--nate
Cheers Kevin
On Thu, Aug 19, 2010 at 3:42 AM, Nate Coraor <nate@bx.psu.edu mailto:nate@bx.psu.edu> wrote:
Hi Farhat and others, Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome. --nate Farhat Habib wrote: I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main. -Farhat -- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhat <http://www.iiserpune.ac.in/%7Efarhat> Office: +91 20 2590 8015 Cell: +91 973 0996 233 On Wed, Aug 18, 2010 at 12:25 PM, Abhay Krishna <abhay.krishna@gmail.com <mailto:abhay.krishna@gmail.com> <mailto:abhay.krishna@gmail.com <mailto:abhay.krishna@gmail.com>>> wrote: Dear Asifullah As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally, try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over also you can alternatively use a perl or python script locally http://seqanswers.com/forums/showthread.php?t=269 best Abhay Krishna On 18 August 2010 08:47, Kevin Lam <aboulia@gmail.com <mailto:aboulia@gmail.com> <mailto:aboulia@gmail.com <mailto:aboulia@gmail.com>>> wrote: I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan <asifullah111@gmail.com <mailto:asifullah111@gmail.com> <mailto:asifullah111@gmail.com <mailto:asifullah111@gmail.com>>> wrote: Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan. _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> <mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu>> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> <mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu>> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> <mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu>> http://lists.bx.psu.edu/listinfo/galaxy-user ------------------------------------------------------------------------ _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user
Quick supplement to what Nate said, if you did want to ship data from dropbox -> galaxy, you could (temporarily) throw the file to be shipped in your dropbox public folder and then put that public download link into galaxy's upload box and it would work fine.
-Dannon
On Aug 19, 2010, at 12:37 PM, Nate Coraor wrote:
Kevin Lam wrote:
Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files. It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some.
Hi Kevin,
Thanks for the suggestion. The donwside I see here is that it'd be a two step process to get it to S3 and then from there to Galaxy (and in addition, it's not free). Ideally we'd need a method to upload to Galaxy directly, but without the browser flakiness.
Of course we can do this now with something like FTP, but a browser based solution would be more ideal.
with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver?
Not for history uploads, but you can with Library uploads (and then import the library item into your history):
http://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFiles
--nate
Cheers Kevin On Thu, Aug 19, 2010 at 3:42 AM, Nate Coraor <nate@bx.psu.edu mailto:nate@bx.psu.edu> wrote: Hi Farhat and others, Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome. --nate Farhat Habib wrote: I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main. -Farhat -- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhat http://www.iiserpune.ac.in/%7Efarhat Office: +91 20 2590 8015 Cell: +91 973 0996 233 On Wed, Aug 18, 2010 at 12:25 PM, Abhay Krishna <abhay.krishna@gmail.com mailto:abhay.krishna@gmail.com <mailto:abhay.krishna@gmail.com mailto:abhay.krishna@gmail.com>> wrote: Dear Asifullah As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally, try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over also you can alternatively use a perl or python script locally http://seqanswers.com/forums/showthread.php?t=269 best Abhay Krishna On 18 August 2010 08:47, Kevin Lam <aboulia@gmail.com mailto:aboulia@gmail.com <mailto:aboulia@gmail.com mailto:aboulia@gmail.com>> wrote: I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin On Wed, Aug 18, 2010 at 2:17 PM, asifullah khan <asifullah111@gmail.com mailto:asifullah111@gmail.com <mailto:asifullah111@gmail.com mailto:asifullah111@gmail.com>> wrote: Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan. _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu <mailto:galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu> http://lists.bx.psu.edu/listinfo/galaxy-user ------------------------------------------------------------------------ _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu mailto:galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Yes, that would be fine for data that is not sensitive. I believe dropbox's API is publicly avail? perhaps there can be a dropbox user acct for each galaxy user acct?
I acknowlegde Nate's concern that its an additional waste of bandwidth hence I mentioned it's a convoluted way of getting data across. I imagine it might be helpful for those that already use such services to store their data on the cloud.
That said, has anyone considered the use of java uploaders that allow drag and drop like those for flickr and dropbox?
Just tossing ideas about really. sorry if i sound like I don't know what I am saying.
Cheers Kevin
On Fri, Aug 20, 2010 at 1:08 AM, Dannon Baker dannonbaker@me.com wrote:
Quick supplement to what Nate said, if you did want to ship data from dropbox -> galaxy, you could (temporarily) throw the file to be shipped in your dropbox public folder and then put that public download link into galaxy's upload box and it would work fine.
-Dannon
Kevin Lam wrote:
Yes, that would be fine for data that is not sensitive. I believe dropbox's API is publicly avail? perhaps there can be a dropbox user acct for each galaxy user acct?
I acknowlegde Nate's concern that its an additional waste of bandwidth hence I mentioned it's a convoluted way of getting data across. I imagine it might be helpful for those that already use such services to store their data on the cloud.
That said, has anyone considered the use of java uploaders that allow drag and drop like those for flickr and dropbox?
I haven't used dropbox, but flickr's browser uploader uses Flash, which loads the entire file into memory before sending (unfeasible for large files). Java is certainly something we've considered as an alternate in-browser uploader (which doesn't replace the regular form). We'd like to stay away from any sort of desktop uploader application.
Just tossing ideas about really. sorry if i sound like I don't know what I am saying.
Much appreciated, thanks!
--nate
Cheers Kevin
On Fri, Aug 20, 2010 at 1:08 AM, Dannon Baker <dannonbaker@me.com mailto:dannonbaker@me.com> wrote:
Quick supplement to what Nate said, if you did want to ship data from dropbox -> galaxy, you could (temporarily) throw the file to be shipped in your dropbox public folder and then put that public download link into galaxy's upload box and it would work fine. -Dannon
asifullah khan wrote:
Dear Officer,
I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA.
Hi Asifullah,
Have you retried the upload? Occasionally, uploads may be interrupted. We do try to handle this and mark the dataset in an error state, but sometimes Galaxy is not able to detect the failure, and it never changes out of the uploading state.
As others have said, if you have an FTP or web site on which you can place the file, and then put the URL to that file in the URL/text box on the upload form, this method is more reliable.
Unfortunately, browsers are not well suited for sending large files, especially if over a slow or flaky connection.
--nate
Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan.
galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
galaxy-user@lists.galaxyproject.org
-
Abhay Krishna -
asifullah khan -
Dannon Baker -
Farhat Habib -
Florent Angly -
Jelle Scholtalbers -
Kevin Lam -
Nate Coraor