Re: [galaxy-dev] [galaxy-user] Why SGE needed for galaxy ?
The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image. Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs. Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :) Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts. -Dannon On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudma... On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker <dannonbaker@me.com> wrote:
The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image.
Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs.
Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :)
Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts.
-Dannon
On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hi, Thanks for answers , Actually i tried to understand cloudman role in galaxy , do you have an architecture paper which i should read ? for e.g. what i unserstood si that cloud man runs a python server and manage SGE through it via some python script . (may be i am wrong) Our proposed installation is like this: Users launch cloud man from Open nebula cloud, when cloudman is running they can add more nodes which are endup in same private cloud. As you suggested in this particular case I should have same images both for (master cloudman) and workers , am i right ? actually i built one image via https://bitbucket.org/galaxy/cloudman . DO you suggest that I should built image from CBL tree you mentioned below ? which wd have both cloudman and galaxy together. BR Zeeshan On W6-Feb 8, 2013, at 10:21 PM, Enis Afgan wrote:
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudma...
On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker <dannonbaker@me.com> wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image.
Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs.
Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :)
Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts.
-Dannon
On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Here's a link to the architecture paper: http://onlinelibrary.wiley.com/doi/10.1002/cpe.1836/full Building CloudMan image from the repo you mention will not work - that repo is for cloudman itself while you need an image capable of running cloudman. For that, you should use CBL. Also, here are some instructions about setting up cloudman and galaxy on OpenNebula cloud: https://www.cloud.sara.nl/projects/mattiasdehollander-project/wiki (note that this mentions use of mi-deployment set of scripts; since that, mi-deployment has been merged into CBL). On Mon, Feb 11, 2013 at 8:21 PM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Hi,
Thanks for answers ,
Actually i tried to understand cloudman role in galaxy , do you have an architecture paper which i should read ? for e.g. what i unserstood si that cloud man runs a python server and manage SGE through it via some python script . (may be i am wrong)
Our proposed installation is like this: Users launch cloud man from Open nebula cloud, when cloudman is running they can add more nodes which are endup in same private cloud. As you suggested in this particular case I should have same images both for (master cloudman) and workers , am i right ?
actually i built one image via https://bitbucket.org/galaxy/cloudman . DO you suggest that I should built image from CBL tree you mentioned below ? which wd have both cloudman and galaxy together.
BR
Zeeshan
On W6-Feb 8, 2013, at 10:21 PM, Enis Afgan wrote:
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudma...
On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker <dannonbaker@me.com> wrote:
The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image.
Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs.
Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :)
Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts.
-Dannon
On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Thanks Enis, The paper is good , i was looking something like it. Ok, I will use CBL , just a small query : we use CBL for cloud man purpose, i think yes . Should we use the same Image for workers and enable cloudman script to use that one for launching nodes ? BR Zeeshan On W7-Feb 11, 2013, at 9:27 PM, Enis Afgan wrote:
Here's a link to the architecture paper: http://onlinelibrary.wiley.com/doi/10.1002/cpe.1836/full
Building CloudMan image from the repo you mention will not work - that repo is for cloudman itself while you need an image capable of running cloudman. For that, you should use CBL.
Also, here are some instructions about setting up cloudman and galaxy on OpenNebula cloud: https://www.cloud.sara.nl/projects/mattiasdehollander-project/wiki (note that this mentions use of mi-deployment set of scripts; since that, mi-deployment has been merged into CBL).
On Mon, Feb 11, 2013 at 8:21 PM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
Thanks for answers ,
Actually i tried to understand cloudman role in galaxy , do you have an architecture paper which i should read ? for e.g. what i unserstood si that cloud man runs a python server and manage SGE through it via some python script . (may be i am wrong)
Our proposed installation is like this: Users launch cloud man from Open nebula cloud, when cloudman is running they can add more nodes which are endup in same private cloud. As you suggested in this particular case I should have same images both for (master cloudman) and workers , am i right ?
actually i built one image via https://bitbucket.org/galaxy/cloudman . DO you suggest that I should built image from CBL tree you mentioned below ? which wd have both cloudman and galaxy together.
BR
Zeeshan
On W6-Feb 8, 2013, at 10:21 PM, Enis Afgan wrote:
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudma...
On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker <dannonbaker@me.com> wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image.
Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs.
Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :)
Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts.
-Dannon
On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Yes, you use the same image for both masters and the workers. -Dannon On Feb 12, 2013, at 5:46 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Thanks Enis, The paper is good , i was looking something like it.
Ok, I will use CBL , just a small query :
we use CBL for cloud man purpose, i think yes . Should we use the same Image for workers and enable cloudman script to use that one for launching nodes ?
BR
Zeeshan
On W7-Feb 11, 2013, at 9:27 PM, Enis Afgan wrote:
Here's a link to the architecture paper: http://onlinelibrary.wiley.com/doi/10.1002/cpe.1836/full
Building CloudMan image from the repo you mention will not work - that repo is for cloudman itself while you need an image capable of running cloudman. For that, you should use CBL.
Also, here are some instructions about setting up cloudman and galaxy on OpenNebula cloud: https://www.cloud.sara.nl/projects/mattiasdehollander-project/wiki (note that this mentions use of mi-deployment set of scripts; since that, mi-deployment has been merged into CBL).
On Mon, Feb 11, 2013 at 8:21 PM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
Thanks for answers ,
Actually i tried to understand cloudman role in galaxy , do you have an architecture paper which i should read ? for e.g. what i unserstood si that cloud man runs a python server and manage SGE through it via some python script . (may be i am wrong)
Our proposed installation is like this: Users launch cloud man from Open nebula cloud, when cloudman is running they can add more nodes which are endup in same private cloud. As you suggested in this particular case I should have same images both for (master cloudman) and workers , am i right ?
actually i built one image via https://bitbucket.org/galaxy/cloudman . DO you suggest that I should built image from CBL tree you mentioned below ? which wd have both cloudman and galaxy together.
BR
Zeeshan
On W6-Feb 8, 2013, at 10:21 PM, Enis Afgan wrote:
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudma...
On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker <dannonbaker@me.com> wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image.
Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs.
Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :)
Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts.
-Dannon
On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Thanks Dannon. BR Zeeshan On W7-Feb 12, 2013, at 12:22 PM, Dannon Baker wrote:
Yes, you use the same image for both masters and the workers.
-Dannon
On Feb 12, 2013, at 5:46 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Thanks Enis, The paper is good , i was looking something like it.
Ok, I will use CBL , just a small query :
we use CBL for cloud man purpose, i think yes . Should we use the same Image for workers and enable cloudman script to use that one for launching nodes ?
BR
Zeeshan
On W7-Feb 11, 2013, at 9:27 PM, Enis Afgan wrote:
Here's a link to the architecture paper: http://onlinelibrary.wiley.com/doi/10.1002/cpe.1836/full
Building CloudMan image from the repo you mention will not work - that repo is for cloudman itself while you need an image capable of running cloudman. For that, you should use CBL.
Also, here are some instructions about setting up cloudman and galaxy on OpenNebula cloud: https://www.cloud.sara.nl/projects/mattiasdehollander-project/wiki (note that this mentions use of mi-deployment set of scripts; since that, mi-deployment has been merged into CBL).
On Mon, Feb 11, 2013 at 8:21 PM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
Thanks for answers ,
Actually i tried to understand cloudman role in galaxy , do you have an architecture paper which i should read ? for e.g. what i unserstood si that cloud man runs a python server and manage SGE through it via some python script . (may be i am wrong)
Our proposed installation is like this: Users launch cloud man from Open nebula cloud, when cloudman is running they can add more nodes which are endup in same private cloud. As you suggested in this particular case I should have same images both for (master cloudman) and workers , am i right ?
actually i built one image via https://bitbucket.org/galaxy/cloudman . DO you suggest that I should built image from CBL tree you mentioned below ? which wd have both cloudman and galaxy together.
BR
Zeeshan
On W6-Feb 8, 2013, at 10:21 PM, Enis Afgan wrote:
As far as actually building the image, the recommended method is to use CloudBioLinux build scripts: https://github.com/chapmanb/cloudbiolinux There is a CloudMan flavor of CBL that allows you to build only CloudMan- and Galaxy-required parts: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudma...
On Sat, Feb 9, 2013 at 12:24 AM, Dannon Baker <dannonbaker@me.com> wrote: The workers don't need their own copy of galaxy installed, but a shared filesystem is a requirement for galaxy (in any cluster environment -- see the galaxy wiki for more http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster). Cloudman handles managing NFS for you and sharing the galaxy/tools/index/data volumes. In order for workers to communicate with the master instance, they'll need the cloudman installation as well, so you should use the same image.
Now that I've answered that, I'm not sure I totally understand your proposed installation yet, but if you're suggesting bypassing cloudman for installation on a private cloud it should be possible. You'd want the master instance up full time running as the galaxy front end, dispatching jobs to a separate cluster managed by SGE/PBS/whatever. Basically the standard cluster configuration outlined in the wiki above, but you'd want your worker nodes automatically configured to mount the shared directories and join the PBS/SGE queue so they could handle jobs.
Depending on what type of private cloud you're working with, it might be easier to just see if you can get cloudman to work :)
Lastly, I swapped this message to galaxy-dev since it's about installation nuts and bolts.
-Dannon
On Feb 8, 2013, at 3:02 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote:
Dear Enis, thanks for reply and being you as cloudman developer it is good to see you in the list .
Q2: On Workers node we need galaxy installed with its shared directories ? like galaxyindices , galaxydata Q3: For a private cloud setup do you prefare to have a master image with cloudman and galaxy and use the same image for workers as well ? or worker images can be vanilla OS ?
BR
Zeeshan
On W6-Feb 7, 2013, at 11:50 PM, Enis Afgan wrote:
Hi Zeeshan, In order to gain from the scalability of the cloud, SGE does need to run. However, CloudMan sets all that up and manages it going forward.
Enis
On Fri, Feb 8, 2013 at 8:59 AM, Zeeshan Ali Shah <zashah@pdc.kth.se> wrote: Hi,
It seems that cloud man need SGE for scaling . Does SGE need also when run cloud on private cloud ?
Zeeshan ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (4)
-
Dannon Baker -
Dannon Baker -
Enis Afgan -
Zeeshan Ali Shah