Hello, With the continued advancement of sequencing technology, we've seen the size of files uploaded to Galaxy grow quite large. Although our current file upload methods have worked fine for years, they are not well suited for the extremely large files in common use today. Uploading directly from the browser can be unreliable and browsers don't provide feedback on upload progress and state like they do for downloads. Because of this, we have implemented file uploads to Galaxy via FTP. FTP will allow you to monitor as mentioned above, as well as resume interrupted transfers. To get started using FTP, you'll need to have registered a regular Galaxy account on our public server at: http://main.g2.bx.psu.edu/ Once registered, you can initiate an FTP connection in your preferred FTP client to the same host (main.g2.bx.psu.edu) using your registered email address and password for the login details. Files uploaded to this server won't automatically be imported to Galaxy - rather, you will be presented with a list of the contents of your FTP directory on the standard "Upload File" tool interface. Files not imported within 3 days will be cleaned up from the FTP site. We hope that this service solves a crucial problem in data transfer speed and reliability. Please report any problems or suggestions with the service to us at: galaxy-bugs@bx.psu.edu Please note that it may not always be practical to use the public Galaxy servers. If you're routinely working with very large data and having to wait for it to upload, a local Galaxy server could be a more practical solution. Instructions on installing your own server can be found at: http://getgalaxy.org/ http://usegalaxy.org/production In addition, with the recent announcement of Amazon's parallel upload capability for the Simple Storage Service (S3), we are investigating incorporating easy access to S3 buckets for Galaxy instances on the Amazon Elastic Compute Cloud (EC2). But you don't need to wait for the pretty interface, you can already access contents of S3 buckets by pasting links to their contents in the "URL/Text:" field of the "Upload File" tool. For an example of how to do this, see the "Watch how the complete analysis can be performed on the Amazon Cloud" screencast at: http://usegalaxy.org/heteroplasmy Thanks, --nate Galaxy Team
On Wed, Nov 17, 2010 at 4:23 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hello,
With the continued advancement of sequencing technology, we've seen the size of files uploaded to Galaxy grow quite large. Although our current file upload methods have worked fine for years, they are not well suited for the extremely large files in common use today. Uploading directly from the browser can be unreliable and browsers don't provide feedback on upload progress and state like they do for downloads.
Because of this, we have implemented file uploads to Galaxy via FTP. ...
We hope that this service solves a crucial problem in data transfer speed and reliability. Please report any problems or suggestions with the service to us at:
Hi Nate, Just to clarify - the new functionality is to allow FTP from the user's computer (running an FTP client) to a Galaxy server (running an FTP server), right? I'm already using the "Upload File from your computer" tool to import a file from a remote server into a Galaxy server. Note this functionality would be more discoverable if the tool name or description was changed... how about: "Import File from your computer or another server"? Peter
Peter wrote:
On Wed, Nov 17, 2010 at 4:23 PM, Nate Coraor <nate@bx.psu.edu> wrote:
Hello,
With the continued advancement of sequencing technology, we've seen the size of files uploaded to Galaxy grow quite large. Although our current file upload methods have worked fine for years, they are not well suited for the extremely large files in common use today. Uploading directly from the browser can be unreliable and browsers don't provide feedback on upload progress and state like they do for downloads.
Because of this, we have implemented file uploads to Galaxy via FTP. ...
We hope that this service solves a crucial problem in data transfer speed and reliability. Please report any problems or suggestions with the service to us at:
Hi Nate,
Just to clarify - the new functionality is to allow FTP from the user's computer (running an FTP client) to a Galaxy server (running an FTP server), right?
That is correct.
I'm already using the "Upload File from your computer" tool to import a file from a remote server into a Galaxy server. Note this functionality would be more discoverable if the tool name or description was changed... how about: "Import File from your computer or another server"?
So many things already refer to "Upload File" that it might be a bit difficult to change. The upside is that this long establishment means that it will hopefully be the first place people look for such functionality. Thanks, --nate
Peter
Great, works well. Will there be documentation on how to implement this on a local install? Shaun Quoting Nate Coraor <nate@bx.psu.edu>:
Hello,
With the continued advancement of sequencing technology, we've seen the size of files uploaded to Galaxy grow quite large. Although our current file upload methods have worked fine for years, they are not well suited for the extremely large files in common use today. Uploading directly from the browser can be unreliable and browsers don't provide feedback on upload progress and state like they do for downloads.
Because of this, we have implemented file uploads to Galaxy via FTP. FTP will allow you to monitor as mentioned above, as well as resume interrupted transfers. To get started using FTP, you'll need to have registered a regular Galaxy account on our public server at:
Once registered, you can initiate an FTP connection in your preferred FTP client to the same host (main.g2.bx.psu.edu) using your registered email address and password for the login details.
Files uploaded to this server won't automatically be imported to Galaxy - rather, you will be presented with a list of the contents of your FTP directory on the standard "Upload File" tool interface. Files not imported within 3 days will be cleaned up from the FTP site.
We hope that this service solves a crucial problem in data transfer speed and reliability. Please report any problems or suggestions with the service to us at:
galaxy-bugs@bx.psu.edu
Please note that it may not always be practical to use the public Galaxy servers. If you're routinely working with very large data and having to wait for it to upload, a local Galaxy server could be a more practical solution. Instructions on installing your own server can be found at:
http://getgalaxy.org/ http://usegalaxy.org/production
In addition, with the recent announcement of Amazon's parallel upload capability for the Simple Storage Service (S3), we are investigating incorporating easy access to S3 buckets for Galaxy instances on the Amazon Elastic Compute Cloud (EC2). But you don't need to wait for the pretty interface, you can already access contents of S3 buckets by pasting links to their contents in the "URL/Text:" field of the "Upload File" tool. For an example of how to do this, see the "Watch how the complete analysis can be performed on the Amazon Cloud" screencast at:
http://usegalaxy.org/heteroplasmy
Thanks, --nate Galaxy Team _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
SHAUN WEBB wrote:
Great, works well.
Will there be documentation on how to implement this on a local install?
Yes! The short version is to set the two new 'ftp_*' options in the config. I'll add a sample FTP server configuration to the Wiki as well. --nate
Shaun
Quoting Nate Coraor <nate@bx.psu.edu>:
Hello,
With the continued advancement of sequencing technology, we've seen the size of files uploaded to Galaxy grow quite large. Although our current file upload methods have worked fine for years, they are not well suited for the extremely large files in common use today. Uploading directly from the browser can be unreliable and browsers don't provide feedback on upload progress and state like they do for downloads.
Because of this, we have implemented file uploads to Galaxy via FTP. FTP will allow you to monitor as mentioned above, as well as resume interrupted transfers. To get started using FTP, you'll need to have registered a regular Galaxy account on our public server at:
Once registered, you can initiate an FTP connection in your preferred FTP client to the same host (main.g2.bx.psu.edu) using your registered email address and password for the login details.
Files uploaded to this server won't automatically be imported to Galaxy - rather, you will be presented with a list of the contents of your FTP directory on the standard "Upload File" tool interface. Files not imported within 3 days will be cleaned up from the FTP site.
We hope that this service solves a crucial problem in data transfer speed and reliability. Please report any problems or suggestions with the service to us at:
galaxy-bugs@bx.psu.edu
Please note that it may not always be practical to use the public Galaxy servers. If you're routinely working with very large data and having to wait for it to upload, a local Galaxy server could be a more practical solution. Instructions on installing your own server can be found at:
http://getgalaxy.org/ http://usegalaxy.org/production
In addition, with the recent announcement of Amazon's parallel upload capability for the Simple Storage Service (S3), we are investigating incorporating easy access to S3 buckets for Galaxy instances on the Amazon Elastic Compute Cloud (EC2). But you don't need to wait for the pretty interface, you can already access contents of S3 buckets by pasting links to their contents in the "URL/Text:" field of the "Upload File" tool. For an example of how to do this, see the "Watch how the complete analysis can be performed on the Amazon Cloud" screencast at:
http://usegalaxy.org/heteroplasmy
Thanks, --nate Galaxy Team _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
Nate Coraor wrote:
SHAUN WEBB wrote:
Great, works well.
Will there be documentation on how to implement this on a local install?
Yes! The short version is to set the two new 'ftp_*' options in the config. I'll add a sample FTP server configuration to the Wiki as well.
Added here: http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP And linked from the production server page. --nate
--nate
Shaun
Quoting Nate Coraor <nate@bx.psu.edu>:
Hello,
With the continued advancement of sequencing technology, we've seen the size of files uploaded to Galaxy grow quite large. Although our current file upload methods have worked fine for years, they are not well suited for the extremely large files in common use today. Uploading directly from the browser can be unreliable and browsers don't provide feedback on upload progress and state like they do for downloads.
Because of this, we have implemented file uploads to Galaxy via FTP. FTP will allow you to monitor as mentioned above, as well as resume interrupted transfers. To get started using FTP, you'll need to have registered a regular Galaxy account on our public server at:
Once registered, you can initiate an FTP connection in your preferred FTP client to the same host (main.g2.bx.psu.edu) using your registered email address and password for the login details.
Files uploaded to this server won't automatically be imported to Galaxy - rather, you will be presented with a list of the contents of your FTP directory on the standard "Upload File" tool interface. Files not imported within 3 days will be cleaned up from the FTP site.
We hope that this service solves a crucial problem in data transfer speed and reliability. Please report any problems or suggestions with the service to us at:
galaxy-bugs@bx.psu.edu
Please note that it may not always be practical to use the public Galaxy servers. If you're routinely working with very large data and having to wait for it to upload, a local Galaxy server could be a more practical solution. Instructions on installing your own server can be found at:
http://getgalaxy.org/ http://usegalaxy.org/production
In addition, with the recent announcement of Amazon's parallel upload capability for the Simple Storage Service (S3), we are investigating incorporating easy access to S3 buckets for Galaxy instances on the Amazon Elastic Compute Cloud (EC2). But you don't need to wait for the pretty interface, you can already access contents of S3 buckets by pasting links to their contents in the "URL/Text:" field of the "Upload File" tool. For an example of how to do this, see the "Watch how the complete analysis can be performed on the Amazon Cloud" screencast at:
http://usegalaxy.org/heteroplasmy
Thanks, --nate Galaxy Team _______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
_______________________________________________ galaxy-user mailing list galaxy-user@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-user
participants (3)
-
Nate Coraor -
Peter -
SHAUN WEBB