Moving data about the EC2/EBS/S3 spaces on the Amazon cloud.
Hello, I am new to cloud computing and am trying to use the Galaxy cloudman service through Amazon to analyse NGS data. I have a couple of questions regarding data transfer: If I am running an EC2 instance, with an EBS volume attached, how can I retrieve the data from the EBS volume? In addition, how do I view what files are in the EBS volume? Can I send the data to an S3 bucket? And how would I move data from the S3 bucket to a fresh EC2 instance? Assuming I can store my files on EBS (or S3), what is the best way to download them? Can I set up a ftp transfer to (quickly) download my potentially very large files? Many thanks for any insight and direction! Cheers, Mo Heydarian
Hi Mo, On Wed, Apr 18, 2012 at 4:53 AM, Mohammad Heydarian <mheydar1@jhmi.edu>wrote:
Hello, I am new to cloud computing and am trying to use the Galaxy cloudman service through Amazon to analyse NGS data. I have a couple of questions regarding data transfer:
If I am running an EC2 instance, with an EBS volume attached, how can I retrieve the data from the EBS volume?
If the data is not accessible via Galaxy (in which case you can use the Save icon for a given dataset in your History), you will need to scp the files over to your local instance. A command like the following should work: [local] $ scp -i <path to private key> ubuntu@ec2-<rest of instance DNS>:<path to files (see below)> <path on your local machine>
In addition, how do I view what files are in the EBS volume?
If using CloudMan to setup a data volume, all the user data will be stored under /mnt/galaxyData/. There will be more directories there with the Galaxy ones under the 'files' subdir.
Can I send the data to an S3 bucket? And how would I move data from the S3 bucket to a fresh EC2 instance?
This needs to be done manually via ec2 tools, a combnation of boto and pyhton or a combination of scp and a GUI tools such as cyberduck.
Assuming I can store my files on EBS (or S3), what is the best way to download them?
From EBS, you must use scp; from S3 you can use the AWS console or a 3rd
party tool (cyberduck)
Can I set up a ftp transfer to (quickly) download my potentially very large files?
CloudMan sets up an FTP server on the instance for use with Galaxy ( http://wiki.g2.bx.psu.edu/FTPUpload). For downloading files, you could probably leverage that server but it's likely to take some playing around with (here are instructions mirrored on the cloud instance that may be useful http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP). Hope this clears up some of your questions, Enis
Many thanks for any insight and direction!
Cheers, Mo Heydarian
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (2)
-
Enis Afgan -
Mohammad Heydarian