Re: [galaxy-user] Loading files to local Galaxy
From: Tilahun Abebe <Tilahun.Abebe@uni.edu>
We are trying to load Illumina data to our local Galaxy instance. The files are between 700 MB and 2.2 GB. Files below 2 GB load in less than 5 minutes. Files larger than 2 GB don't upload at all. We installed Galaxy locally because we thought loading files will be faster than the server version. Any suggestions to solve this problem is highly appreciated.
While (from others experience) Galaxy _should_ be able to upload files that large, we've had some problems with our local installation too. Investigation didn't reveal any cause, so we put it down to the quality of our network. You might what to look at the webserver or proxy that you have in front of Galaxy - from memory, both Apache and nginx can be configured to impose file size limits, so that _may_ be the problem. In any event, you might want to configure your server to handle uploads and downloads directly as per <https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServe r>. Finally you can pass an url to the upload dialog to get Galaxy to pull the file from an ftp server - for example - and that may prove more robust. ---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency ----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
Paul-Michael Agapow wrote:
From: Tilahun Abebe <Tilahun.Abebe@uni.edu>
We are trying to load Illumina data to our local Galaxy instance. The files are between 700 MB and 2.2 GB. Files below 2 GB load in less than 5 minutes. Files larger than 2 GB don't upload at all. We installed Galaxy locally because we thought loading files will be faster than the server version. Any suggestions to solve this problem is highly appreciated.
While (from others experience) Galaxy _should_ be able to upload files that large, we've had some problems with our local installation too. Investigation didn't reveal any cause, so we put it down to the quality of our network. You might what to look at the webserver or proxy that you have in front of Galaxy - from memory, both Apache and nginx can be configured to impose file size limits, so that _may_ be the problem. In any event, you might want to configure your server to handle uploads and downloads directly as per <https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServe r>. Finally you can pass an url to the upload dialog to get Galaxy to pull the file from an ftp server - for example - and that may prove more robust.
Hi Paul, These failures may be due to the fact that many browsers will simply fail to upload files > 2GB (although I know there are people out there have successfully done it). Tilahun, I'll echo Paul's suggestion to use the Production setup. There are also alternative options for getting data into Galaxy. For users, you can have an FTP server (or just a local directory on the server where they can place files for upload): https://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP Or you can use data libraries and load directly off filesystems accessible to the server: https://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFile... --nate
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Thank you Paul and Nate. We will try the options you suggested. One thing we found to work much faster (less than 5 minutes for most files) is to upload zipped data. Galaxy could upload and unzip the files without any problem. It doesn't seem any sequence data is lost. Has anyone tried this before? Thanks. Tilahun --------------- Nate Coraor wrote:
Paul-Michael Agapow wrote:
From: Tilahun Abebe<Tilahun.Abebe@uni.edu>
We are trying to load Illumina data to our local Galaxy instance. The files are between 700 MB and 2.2 GB. Files below 2 GB load in less than 5 minutes. Files larger than 2 GB don't upload at all. We installed Galaxy locally because we thought loading files will be faster than the server version. Any suggestions to solve this problem is highly appreciated.
While (from others experience) Galaxy _should_ be able to upload files that large, we've had some problems with our local installation too. Investigation didn't reveal any cause, so we put it down to the quality of our network. You might what to look at the webserver or proxy that you have in front of Galaxy - from memory, both Apache and nginx can be configured to impose file size limits, so that _may_ be the problem. In any event, you might want to configure your server to handle uploads and downloads directly as per <https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServe r>. Finally you can pass an url to the upload dialog to get Galaxy to pull the file from an ftp server - for example - and that may prove more robust. Hi Paul,
These failures may be due to the fact that many browsers will simply fail to upload files> 2GB (although I know there are people out there have successfully done it).
Tilahun, I'll echo Paul's suggestion to use the Production setup. There are also alternative options for getting data into Galaxy. For users, you can have an FTP server (or just a local directory on the server where they can place files for upload):
https://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP
Or you can use data libraries and load directly off filesystems accessible to the server:
https://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFile...
--nate
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Tilahun Abebe wrote:
Thank you Paul and Nate. We will try the options you suggested.
One thing we found to work much faster (less than 5 minutes for most files) is to upload zipped data. Galaxy could upload and unzip the files without any problem. It doesn't seem any sequence data is lost. Has anyone tried this before?
Yes, this is a standard feature. zip, gzip, and bzip2 are all supported. Only one file per archive at this time, however. --nate
Thanks.
Tilahun ---------------
Nate Coraor wrote:
Paul-Michael Agapow wrote:
From: Tilahun Abebe<Tilahun.Abebe@uni.edu>
We are trying to load Illumina data to our local Galaxy instance. The files are between 700 MB and 2.2 GB. Files below 2 GB load in less than 5 minutes. Files larger than 2 GB don't upload at all. We installed Galaxy locally because we thought loading files will be faster than the server version. Any suggestions to solve this problem is highly appreciated.
While (from others experience) Galaxy _should_ be able to upload files that large, we've had some problems with our local installation too. Investigation didn't reveal any cause, so we put it down to the quality of our network. You might what to look at the webserver or proxy that you have in front of Galaxy - from memory, both Apache and nginx can be configured to impose file size limits, so that _may_ be the problem. In any event, you might want to configure your server to handle uploads and downloads directly as per <https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServe r>. Finally you can pass an url to the upload dialog to get Galaxy to pull the file from an ftp server - for example - and that may prove more robust. Hi Paul,
These failures may be due to the fact that many browsers will simply fail to upload files> 2GB (although I know there are people out there have successfully done it).
Tilahun, I'll echo Paul's suggestion to use the Production setup. There are also alternative options for getting data into Galaxy. For users, you can have an FTP server (or just a local directory on the server where they can place files for upload):
https://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP
Or you can use data libraries and load directly off filesystems accessible to the server:
https://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFile...
--nate
---- Paul Agapow (paul-michael.agapow@hpa.org.uk) Bioinformatics, Centre for Infections, Health Protection Agency
----------------------------------------- ************************************************************************** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk **************************************************************************
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (3)
-
Nate Coraor -
Paul-Michael Agapow -
Tilahun Abebe