Hi, I wish to remove most of the mnt/galaxyIndices/genomes files so I can have a smaller volume that contains those files and save money. I am new to this system. I have set up a Galaxy Cloudman account and used the tools on the web interface, and I can ssh the server using Ubuntu. So far, so good. I tried the first approach here twice https://wiki.galaxyproject.org/CloudMan/CustomizeGalaxyCloud but the system hangs (>24h). I can't follow the "manual" instructions on this page and I'm not sure it would result in a smaller volume anyway.
So, I was trying to follow Enis' steps from this thread to reduce the size of the galaxyIndices volume: http://dev.list.galaxyproject.org/Reducing-costs-in-Cloud-Galaxy-td4483448.h...
His instructions are: "1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount & detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate & delete the cluster you created in step 1"
Here are my questions to help me to get through step 4:
1) Step 1: Is a "cluster" the same thing as an "instance"?
2) Step 2: I deleted the directories for individual genomes using rm -rf . Is that the correct approach?
3) Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep?
4) Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?)
5) Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)?
Any help would be gratefully received! Denise
Hi Denise, Although the instructions you found can still be used, a couple of steps can be updated now so I'll to that now and hopefully answer your questions in the process.
Updated instructions: 1. Start a brand new Galaxy CloudMan cluster/instance (this will be only a temporary cluster needed for the duration of the following steps) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices; using '*rm -rf <dir name>*' is the command to use) 3. From CloudMan's Admin page, add a new file system of type 'new volume' and desired size (ie, size of data on /mnt/galaxyIndices after you've deleted what you didn't want). Give this file system a name 'indices'. 4. ssh into the master instance and copy over the data from /mnt/galaxyIndices to /mnt/indices using command: rsync -avz /mnt/galaxyIndices/ /mnt/indices/. (do this as 'galaxy' user) 5. Unmount the new 'indices' file system using command: sudo umount /mnt/indices 6. From the AWS console, create a snapshot of the volume for the 'indices' file system. You can retrieve the volume ID from the file system 'details' on the CloudMan Admin page 7. For the cluster you want to keep around (while it is terminated), edit *persistent_data.yaml* in it's bucket on S3 (you can get the bucket name from the CloudMan Admin page - it will look like so *cm-<hash>*) and replace the existing snap ID for the *galaxyIndices* with the snapshot ID you got in the previous step 8. Start that cluster and you should have a file system from the new snapshot mounted 9. Terminate & delete the cluster you created in step 1
Here are my questions to help me to get through step 4:
Step 1: Is a “cluster” the same thing as an “instance”?
Yes
Step 2: I deleted the directories for individual genomes usingrm –rf . Is that the correct approach?
Yes
Step 3: Do I add the newly created EBS volume to the sameinstance where I deleted the genomes? Or is it added the instance I want to keep?
You add the new file system (ie, EBS volume) to the newly created cluster (and you also delete the extra genomes on that same cluster).
Step 3: I can see how to attach this newly created volume usingthe AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?)
With the updated instructions above, mounting the file system by hand is no longer necessary.
Step 4: what is the syntax for the rsync (or cp) command to copydirectories/files from one volume to another volume (within the same instance, or in if they are in different instances)?
I included in the command in the instructions now.
Hope this helps. Let us know if you have any trouble or if you have any more questions, Enis
galaxy-dev@lists.galaxyproject.org
-
Denise Clark -
Enis Afgan