Hello Marcel, In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions. Thanks! On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote:
Hi Greg,
I'm currently trying to create a GalaxyCloudman version that includes CADDSuite. Thus, I launched GalaxyCloudman as described in your wiki and tried to modify it afterwards.
Well, starting cloudman worked without any problems... so far, so good :-) As described on http://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group 'galaxy'.
Thus my question: How should users be able to customize cloudman? Is there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-)
Sorry Greg, if you are not the correct contact in this case, but I found not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ...
Cheers, Marcel
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
Marcel, After logging in as the ubuntu user, simply 'sudo su galaxy', and then make any modifications you want. If you do change the tools volume, remember to use the cloudman admin panel and select "persist changes to the file system: galaxyTools". -Dannon On Nov 30, 2011, at 9:42 AM, Greg Von Kuster wrote:
Hello Marcel,
In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions.
Thanks!
On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote:
Hi Greg,
I'm currently trying to create a GalaxyCloudman version that includes CADDSuite. Thus, I launched GalaxyCloudman as described in your wiki and tried to modify it afterwards.
Well, starting cloudman worked without any problems... so far, so good :-) As described on http://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group 'galaxy'.
Thus my question: How should users be able to customize cloudman? Is there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-)
Sorry Greg, if you are not the correct contact in this case, but I found not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ...
Cheers, Marcel
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications. Good luck, Enis On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:
Hello Marcel,
In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions.
Thanks!
On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote:
Hi Greg,
I'm currently trying to create a GalaxyCloudman version that includes CADDSuite. Thus, I launched GalaxyCloudman as described in your wiki and tried to modify it afterwards.
Well, starting cloudman worked without any problems... so far, so good :-) As described on http://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group 'galaxy'.
Thus my question: How should users be able to customize cloudman? Is there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-)
Sorry Greg, if you are not the correct contact in this case, but I found not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ...
Cheers, Marcel
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
Hi Enis and Dannon, thanks for the reply. Yes, I should have tried 'sudo su' ... this works. Ok, integrating CADDSuite was no problem at all after this and everything seems to run well. However, when I create an AMI, terminate the cluster and create a new cluster using the new AMI, both /mnt/galaxyData and /mnt/galaxyTools do not exist anymore, i.e. /dev/sdg3 and /dev/sdg4 are not mounted automatically. If I mount those two devices manually, everything runs smoothly again. So, is there anything that I might have forgotten to do while creating the AMI? Is there a way to make sure that those devices will be mounted automatically? Regards, Marcel On 11/30/11 3:57 PM, Enis Afgan wrote:
Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications.
Good luck, Enis
On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kuster<greg@bx.psu.edu> wrote:
Hello Marcel,
In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions.
Thanks!
On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote:
Hi Greg,
I'm currently trying to create a GalaxyCloudman version that includes CADDSuite. Thus, I launched GalaxyCloudman as described in your wiki and tried to modify it afterwards.
Well, starting cloudman worked without any problems... so far, so good :-) As described on http://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group 'galaxy'.
Thus my question: How should users be able to customize cloudman? Is there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-)
Sorry Greg, if you are not the correct contact in this case, but I found not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ...
Cheers, Marcel
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
-- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Hi Marcel, However, when I create an AMI, terminate the cluster and create a new
cluster using the new AMI, both /mnt/galaxyData and /mnt/galaxyTools do not exist anymore, i.e. /dev/sdg3 and /dev/sdg4 are not mounted automatically. If I mount those two devices manually, everything runs smoothly again.
So, is there anything that I might have forgotten to do while creating the AMI? Is there a way to make sure that those devices will be mounted automatically?
It is not necessary to create a new AMI when wanting to customize your cluster. Instead, on the admin interface - after you have modified the file systems, there is an option to persist static file systems (galaxyTools & galaxyIndices). Once the process is completed and you restart the cluster, just continue to use the same AMI. CloudMan will use the new, customized, data snapshots at runtime.
Let us know how it goes, Enis
Regards, Marcel
On 11/30/11 3:57 PM, Enis Afgan wrote:
Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications.
Good luck, Enis
On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kuster<greg@bx.psu.edu> wrote:
Hello Marcel,
In the future, please send all questions like this to the galaxy-dev mailing list, as doing so will streamline the process of getting a timely answer. I believe Enis is best able to answer your questions.
Thanks!
On Nov 30, 2011, at 9:29 AM, Marcel Schumann wrote:
Hi Greg,
I'm currently trying to create a GalaxyCloudman version that includes
CADDSuite.
Thus, I launched GalaxyCloudman as described in your wiki and tried to
modify it afterwards.
Well, starting cloudman worked without any problems... so far, so good
:-)
As described on http://wiki.g2.bx.psu.edu/**Admin/Cloud/Customize%** 20Galaxy%20Cloud<http://wiki.g2.bx.psu.edu/Admin/Cloud/Customize%20Galaxy%20Cloud> I could then log-in via ssh as user 'ubuntu' (not as user 'galaxy'). However, all files of the galaxy installation belong to user and group
'galaxy'.
Thus my question: How should users be able to customize cloudman? Is
there some trick by which I can log-in as 'galaxy' or do you have any other idea how to make this work ? ;-)
Sorry Greg, if you are not the correct contact in this case, but I found
not specific contact or mailing list for cloudman. Perhaps, you could just forward this mail in that case ...
Cheers, Marcel
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-**tuebingen.de<schumann@informatik.uni-tuebingen.de>
Greg Von Kuster Galaxy Development Team greg@bx.psu.edu
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-**tuebingen.de<schumann@informatik.uni-tuebingen.de>
Hi Enis, well, I know I do not have to create a new AMI if I want to reuse an instance myself. However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI. Ok, I will try to make this work somehow ... but I guess there are no immediate clues as to what could have gone wrong? Or do you have any ideas what I should try? Cheers, Marcel On 12/1/11 10:45 AM, Enis Afgan wrote:
Hi Marcel,
However, when I create an AMI, terminate the cluster and create a new
cluster using the new AMI, both /mnt/galaxyData and /mnt/galaxyTools do not exist anymore, i.e. /dev/sdg3 and /dev/sdg4 are not mounted automatically. If I mount those two devices manually, everything runs smoothly again.
So, is there anything that I might have forgotten to do while creating the AMI? Is there a way to make sure that those devices will be mounted automatically?
It is not necessary to create a new AMI when wanting to customize your cluster. Instead, on the admin interface - after you have modified the file systems, there is an option to persist static file systems (galaxyTools& galaxyIndices). Once the process is completed and you restart the cluster, just continue to use the same AMI. CloudMan will use the new, customized, data snapshots at runtime.
Let us know how it goes, Enis
Regards, Marcel
On 11/30/11 3:57 PM, Enis Afgan wrote:
Hi Marcel, It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications.
Good luck, Enis
On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kuster<greg@bx.psu.edu> wrote:
Hello Marcel,
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-**tuebingen.de<schumann@informatik.uni-tuebingen.de>
-- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Hi Marcel, On Thu, Dec 1, 2011 at 11:28 AM, Marcel Schumann < schumann@informatik.uni-tuebingen.de> wrote:
Hi Enis,
well, I know I do not have to create a new AMI if I want to reuse an instance myself.
However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI.
Unless you modify the system packages (i.e., your customizations are not self contained), you still don't have to create a new AMI to share a cluster. There is the share-a-cluster option (icon next to the cluster name). Just wanted to make sure you were aware of the functionality...
Ok, I will try to make this work somehow ... but I guess there are no immediate clues as to what could have gone wrong? Or do you have any ideas what I should try?
CloudMan sets up the system at runtime so it performs changes that then get persisted when you create the AMI. So, it is necessary to reverse those changes before creating the AMI so that next time a cluster is started, the startup procedure proceeds as before. Did you see what's in the cloudman log (/mnt/cm/paster.log) on your customized AMI? That's probably the easiest place to start and we can work from there. Enis
Cheers, Marcel
On 12/1/11 10:45 AM, Enis Afgan wrote:
Hi Marcel,
However, when I create an AMI, terminate the cluster and create a new
cluster using the new AMI, both /mnt/galaxyData and /mnt/galaxyTools do not exist anymore, i.e. /dev/sdg3 and /dev/sdg4 are not mounted automatically. If I mount those two devices manually, everything runs smoothly again.
So, is there anything that I might have forgotten to do while creating the AMI? Is there a way to make sure that those devices will be mounted automatically?
It is not necessary to create a new AMI when wanting to customize your
cluster. Instead, on the admin interface - after you have modified the file systems, there is an option to persist static file systems (galaxyTools& galaxyIndices). Once the process is completed and you restart the cluster, just continue to use the same AMI. CloudMan will use the new, customized, data snapshots at runtime.
Let us know how it goes, Enis
Regards,
Marcel
On 11/30/11 3:57 PM, Enis Afgan wrote:
Hi Marcel,
It would be best to use 'galaxy' user to add any tools. To do so, after you've logged in as ubuntu user, simply execute: sudo su galaxy and you will become galaxy user. You can then make the desired modifications.
Good luck, Enis
On Wed, Nov 30, 2011 at 3:42 PM, Greg Von Kuster<greg@bx.psu.edu> wrote:
Hello Marcel,
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-**tueb**ingen.de <http://tuebingen.de><; schumann@informatik.**uni-tuebingen.de<schumann@informatik.uni-tuebingen.de>
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-**tuebingen.de<schumann@informatik.uni-tuebingen.de>
Marcel;
well, I know I do not have to create a new AMI if I want to reuse an instance myself.
However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI.
What Enis was suggesting is using the share-a-cluster functionality built into CloudMan. This bundles your data volumes as snapshots and prepares a sharable cluster than anyone can initiate. In the CloudMan interface, there is a little green button next to the cluster name that enables this. That is definitely the easiest way to share and distribute modified CloudMan versions.
Ok, I will try to make this work somehow ... but I guess there are no immediate clues as to what could have gone wrong? Or do you have any ideas what I should try?
After CloudMan boots once, you need to clean up some files before preparing an AMI. This is the automated code we use to clean up for prepping CloudMan compatible AMIs: https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/cloudman.py#L... Be careful if you run that directly. It runs immediately before bundling and removes the ssh keys (so you don't have a backdoor to the AMI you are distributing) so you want to do it as the last thing. It also assumes you have unmounted all of the associated Galaxy data libraries. Hope this helps, Brad
Hi Brad, hi Enis, ok, so one possible way should thus be to use the _cleanup_ec2() function before creating an AMI via the amazon console and the second option perhaps is this share-a-cluster functionality. About the latter: sorry, I tried to find out what it really does, but did not find any documentation of it. So, how does this share-a-cluster functionality work in principle? If it does not create AMI, does it just create snapshots of the individual disks that users have to mount (manually) or ... ? My point here is: I need to make a modified version of GalaxyCloudman available (i.e. one that contains the CADDSuite tools). I do not just want to share an _instance_ (a running cluster) but a cloudman image (as an AMI or by some other means) that includes all my tools, so that other users can easily start their own server. If you do indeed have a description of this somewhere that I just did not find, then I am sorry but would be grateful for a link ;-) Cheers, Marcel On 12/1/11 10:04 PM, Brad Chapman wrote:
Marcel;
well, I know I do not have to create a new AMI if I want to reuse an instance myself.
However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI.
What Enis was suggesting is using the share-a-cluster functionality built into CloudMan. This bundles your data volumes as snapshots and prepares a sharable cluster than anyone can initiate.
In the CloudMan interface, there is a little green button next to the cluster name that enables this. That is definitely the easiest way to share and distribute modified CloudMan versions.
Ok, I will try to make this work somehow ... but I guess there are no immediate clues as to what could have gone wrong? Or do you have any ideas what I should try?
After CloudMan boots once, you need to clean up some files before preparing an AMI. This is the automated code we use to clean up for prepping CloudMan compatible AMIs:
https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/cloudman.py#L...
Be careful if you run that directly. It runs immediately before bundling and removes the ssh keys (so you don't have a backdoor to the AMI you are distributing) so you want to do it as the last thing. It also assumes you have unmounted all of the associated Galaxy data libraries.
Hope this helps, Brad
-- Marcel Schumann University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-tuebingen.de
Hi Marcel, On Thu, Dec 1, 2011 at 10:35 PM, Marcel Schumann < schumann@informatik.uni-tuebingen.de> wrote:
Hi Brad, hi Enis,
ok, so one possible way should thus be to use the _cleanup_ec2() function before creating an AMI via the amazon console and the second option perhaps is this share-a-cluster functionality.
About the latter: sorry, I tried to find out what it really does, but did not find any documentation of it. So, how does this share-a-cluster functionality work in principle? If it does not create AMI, does it just create snapshots of the individual disks that users have to mount (manually) or ... ? My point here is: I need to make a modified version of GalaxyCloudman available (i.e. one that contains the CADDSuite tools). I do not just want to share an _instance_ (a running cluster) but a cloudman image (as an AMI or by some other means) that includes all my tools, so that other users can easily start their own server.
The share-an-instance functionality creates snapshot the data volume (/mnt/galaxyData) so any tool that is installed on that file system will get bundled into the shared instance. Also, any modifications that you make to you galaxy configuration (e.g., galaxy_tools.xml) become part of the shared instance. When a user then starts a new cluster and provides the cluster share string (this is available on the shared cluster after you share it), they will get access to an exact replica of your cluster. Like, I mentioned earlier, unless you are modifing system libraries there is absolutely no reason to create a new AMI. Simply contain your customizations to the file system and use cluster sharing functionality. Try it and see if it does what you need it to do.
If you do indeed have a description of this somewhere that I just did not find, then I am sorry but would be grateful for a link ;-)
Sorry but there's no functionality about this feature on the wiki.
However, there is a pretty detailed description of the functionality and the process on the cloudman console after you click the green share a cluster button. I'll work on preparing some additional documentation on this topic. Enis
Cheers, Marcel
On 12/1/11 10:04 PM, Brad Chapman wrote:
Marcel;
well, I know I do not have to create a new AMI if I want to reuse an
instance myself.
However, I would like to share the modified GalaxyCloudman version with other people and therefore I do have to create an AMI.
What Enis was suggesting is using the share-a-cluster functionality built into CloudMan. This bundles your data volumes as snapshots and prepares a sharable cluster than anyone can initiate.
In the CloudMan interface, there is a little green button next to the cluster name that enables this. That is definitely the easiest way to share and distribute modified CloudMan versions.
Ok, I will try to make this work somehow ... but I guess there are no
immediate clues as to what could have gone wrong? Or do you have any ideas what I should try?
After CloudMan boots once, you need to clean up some files before preparing an AMI. This is the automated code we use to clean up for prepping CloudMan compatible AMIs:
https://github.com/chapmanb/**cloudbiolinux/blob/master/** cloudbio/cloudman.py#L87<https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/cloudman.py#L87>
Be careful if you run that directly. It runs immediately before bundling and removes the ssh keys (so you don't have a backdoor to the AMI you are distributing) so you want to do it as the last thing. It also assumes you have unmounted all of the associated Galaxy data libraries.
Hope this helps, Brad
-- Marcel Schumann
University of Tuebingen Wilhelm Schickard Institute for Computer Science Division for Applied Bioinformatics Room C313, Sand 14, D-72076 Tuebingen
phone: +49 (0)7071-29 70437 fax: +49 (0)7071-29 5152 email: schumann@informatik.uni-**tuebingen.de<schumann@informatik.uni-tuebingen.de>
participants (5)
-
Brad Chapman -
Dannon Baker -
Enis Afgan -
Greg Von Kuster -
Marcel Schumann