Hi,
I am struggling at integrating galaxy with our cluster. I am currently running galaxy on a separate server machine from our LSF cluster. The DRMAA library has been installed on the cluster and I've re-installed galaxy into a shared file system visible and accessible from both the server and the cluster.
I've read the wiki page: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster and tried to follow the "unified method" without much success so far. After spending some time digging/reading the mailing list, I would like to be sure that I understand the problem well enough before changing our plans.
Am I right to think that there is no way to make the system works because I cannot currently configure galaxy to ssh into the cluster head node for submitting jobs to the scheduler?
Is the only way forward to have a dedicated cluster node where I could run galaxy on it?
Thanks in advance for your help. Kind regards, Anne. -- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)1223 404 334
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
On Wed, May 2, 2012 at 7:18 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi,
I am struggling at integrating galaxy with our cluster. I am currently running galaxy on a separate server machine from our LSF cluster. The DRMAA library has been installed on the cluster and I've re-installed galaxy into a shared file system visible and accessible from both the server and the cluster.
That's good - the shared file system is highly recommended.
...
Am I right to think that there is no way to make the system works because I cannot currently configure galaxy to ssh into the cluster head node for submitting jobs to the scheduler?
...
Is the Galaxy machine considered a submit node of the cluster? i.e. Can you submit/query/kill cluster jobs at the command line from the Galaxy server?
Peter
Hi Peter,
Thanks for your answer.
No the galaxy machine is not a submitting node of the cluster. So I suppose I do have those two options:
(1) making the galaxy server a submitting node to the cluster or (2) install and run galaxy on a cluster node
What would be best? Any suggestions? If (1), any ideas on what needs to be installed?
Thanks. Anne.
On 2 May 2012, at 21:26, Peter Cock wrote:
On Wed, May 2, 2012 at 7:18 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi,
I am struggling at integrating galaxy with our cluster. I am currently running galaxy on a separate server machine from our LSF cluster. The DRMAA library has been installed on the cluster and I've re-installed galaxy into a shared file system visible and accessible from both the server and the cluster.
That's good - the shared file system is highly recommended.
...
Am I right to think that there is no way to make the system works because I cannot currently configure galaxy to ssh into the cluster head node for submitting jobs to the scheduler?
...
Is the Galaxy machine considered a submit node of the cluster? i.e. Can you submit/query/kill cluster jobs at the command line from the Galaxy server?
Peter
-- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)7958 511 353
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
On Wed, May 2, 2012 at 9:40 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi Peter,
Thanks for your answer.
No the galaxy machine is not a submitting node of the cluster.
That's probably the problem then :)
So I suppose I do have those two options:
(1) making the galaxy server a submitting node to the cluster or (2) install and run galaxy on a cluster node
What would be best? Any suggestions? If (1), any ideas on what needs to be installed?
We went with (1), partly for historical admin reasons of machine ownership - but also this avoided having a single critical point with one machine running both Galaxy and being our cluster head node (since the cluster isn't just used for Galaxy jobs).
Regarding (2), I don't think you want Galaxy running on a cluster compute node - Galaxy isn't that computationally demanding but I wouldn't want the same machine to also be running general cluster jobs. Consider a rogue job submitted to the cluster which consumes too much RAM and brings the node down - that can be annoying, but it would be painful if this also killed your Galaxy server.
It might make sense to put Galaxy on your cluster head node - which might make sharing the data drive simpler too, depending on how your cluster is setup.
Peter
Hi Peter,
Thanks for the clarification, it is really helpful.
Would you mind sharing what needs to be done in term of configuration and instalation to transform the galaxy server into a submitting job to the cluster? Thanks.
Anne.
On 2 May 2012, at 21:51, Peter Cock wrote:
On Wed, May 2, 2012 at 9:40 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi Peter,
Thanks for your answer.
No the galaxy machine is not a submitting node of the cluster.
That's probably the problem then :)
So I suppose I do have those two options:
(1) making the galaxy server a submitting node to the cluster or (2) install and run galaxy on a cluster node
What would be best? Any suggestions? If (1), any ideas on what needs to be installed?
We went with (1), partly for historical admin reasons of machine ownership - but also this avoided having a single critical point with one machine running both Galaxy and being our cluster head node (since the cluster isn't just used for Galaxy jobs).
Regarding (2), I don't think you want Galaxy running on a cluster compute node - Galaxy isn't that computationally demanding but I wouldn't want the same machine to also be running general cluster jobs. Consider a rogue job submitted to the cluster which consumes too much RAM and brings the node down - that can be annoying, but it would be painful if this also killed your Galaxy server.
It might make sense to put Galaxy on your cluster head node - which might make sharing the data drive simpler too, depending on how your cluster is setup.
Peter
-- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)7958 511 353
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
On Wed, May 2, 2012 at 9:58 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi Peter,
Thanks for the clarification, it is really helpful.
Would you mind sharing what needs to be done in term of configuration and instalation to transform the galaxy server into a submitting job to the cluster? Thanks.
You said you have an LSF cluster - so I have no idea, sorry :(
We're using SGE here. I'm presuming similar concepts apply - in our case a key step was getting qsub/qstat/qdel to work from the Galaxy server as well as the cluster head node - which required our cluster administrator to setup our Galaxy server as an SGE submit node.
Regards,
Peter
Yes we do have an LSF cluster. Thanks indeed for all the information, Peter. I'll contact our cluster administrator tomorrow and keep you posted on my progress.
Kind regards, Anne.
On 2 May 2012, at 22:07, Peter Cock wrote:
On Wed, May 2, 2012 at 9:58 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi Peter,
Thanks for the clarification, it is really helpful.
Would you mind sharing what needs to be done in term of configuration and instalation to transform the galaxy server into a submitting job to the cluster? Thanks.
You said you have an LSF cluster - so I have no idea, sorry :(
We're using SGE here. I'm presuming similar concepts apply - in our case a key step was getting qsub/qstat/qdel to work from the Galaxy server as well as the cluster head node - which required our cluster administrator to setup our Galaxy server as an SGE submit node.
Regards,
Peter
-- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)7958 511 353
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
Hi,
Just to keep you updated of my progress. I've just been talking with our cluster administrator, and they are going to install LSF client on the machine which currently host our Galaxy server making it a submitting node to the cluster. I am now waiting to test how it is going to work when all ready. I'll let you know.
Cheers, Anne.
On 2 May 2012, at 22:16, Anne Pajon wrote:
Yes we do have an LSF cluster. Thanks indeed for all the information, Peter. I'll contact our cluster administrator tomorrow and keep you posted on my progress.
Kind regards, Anne.
On 2 May 2012, at 22:07, Peter Cock wrote:
On Wed, May 2, 2012 at 9:58 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi Peter,
Thanks for the clarification, it is really helpful.
Would you mind sharing what needs to be done in term of configuration and instalation to transform the galaxy server into a submitting job to the cluster? Thanks.
You said you have an LSF cluster - so I have no idea, sorry :(
We're using SGE here. I'm presuming similar concepts apply - in our case a key step was getting qsub/qstat/qdel to work from the Galaxy server as well as the cluster head node - which required our cluster administrator to setup our Galaxy server as an SGE submit node.
Regards,
Peter
-- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)7958 511 353
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)1223 404 334
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
We are using PBS locally and had our cluster admin set this up, but I think the same principle applies across most job runners/mgrs. As Peter mentioned, there should be a way for cluster admin to add your server as an allowed submitting host, the PBS variant is here:
http://www.clusterresources.com/torquedocs/2.1jobsubmission.shtml#submitacce...
There also may be additional tweaking required due to firewalls, etc.
After that, run some simple job submission tests (we ran a simple 'sleep 60') from your server to check whether you can submit jobs, see submitted job status, etc.
I found it helped when debugging to log in on both Galaxy server and cluster sides, and if the cluster admin allows it monitor the job manager logs on the cluster side if you run into problems (we found this very handy when debugging local LDAP issues).
chris
On May 2, 2012, at 4:16 PM, Anne Pajon wrote:
Yes we do have an LSF cluster. Thanks indeed for all the information, Peter. I'll contact our cluster administrator tomorrow and keep you posted on my progress.
Kind regards, Anne.
On 2 May 2012, at 22:07, Peter Cock wrote:
On Wed, May 2, 2012 at 9:58 PM, Anne Pajon Anne.Pajon@cancer.org.uk wrote:
Hi Peter,
Thanks for the clarification, it is really helpful.
Would you mind sharing what needs to be done in term of configuration and instalation to transform the galaxy server into a submitting job to the cluster? Thanks.
You said you have an LSF cluster - so I have no idea, sorry :(
We're using SGE here. I'm presuming similar concepts apply - in our case a key step was getting qsub/qstat/qdel to work from the Galaxy server as well as the cluster head node - which required our cluster administrator to setup our Galaxy server as an SGE submit node.
Regards,
Peter
-- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon@cancer.org.uk | +44 (0)7958 511 353
NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose.
We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
galaxy-dev@lists.galaxyproject.org
-
Anne Pajon -
Fields, Christopher J -
Peter Cock