Galaxy on a Cluster -- Active Directory LDAP configuration
Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos.
Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote:
Everyone,
We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP.
Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise.
Thank you all !
Carlos.
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Carlos, We aren’t using Active Directory here – but we do use an OpenLDAP directory which our cluster authenticates against, as does Galaxy. In order to track cluster usage for users running Galaxy jobs we use galaxy-pulsar between Galaxy itself and our SLURM cluster. Pulsar is configured to submit all jobs as a real user on the cluster, via SLURM-DRMAA. This means that all of the Galaxy usage on the cluster appears in our SLURM accounting database just like any other job would. The complexity here is file ownership. Pulsar has to copy all input files into a staging directory, and change ownership to the real user for the job to run on the cluster. It is/was(?) a little complex to setup, as there are more parts involved then a typical Galaxy install, but it works great for us here. When I was getting this setup John Chilton mentioned adding a PulsarEmbedded runner into Galaxy at some point. Not sure whether this has/is happening, but it’d make these situations easier: https://trello.com/c/4YwVZBtq/1865-embedded-pulsar-job-runner DT -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On Behalf Of Nicola Soranzo Sent: Tuesday, September 22, 2015 5:59 AM To: Carlos Lijeron <clijeron@hunter.cuny.edu>; galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote: Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ________________________________ UT Southwestern Medical Center The future of medicine, today.
David, Thank you for the valuable feedback. I will look into the galaxy-pulsar server app and determine how we can implement it to connect our galaxy instance to the working nodes. Perhaps I’ll direct a few relevant questions while we get on board with the implementation, if you don’t mind. If you have any notes or lessons learned that you could share that would be very helpful to us. However, this idea seems already like a good solution for us. I’ll keep all my notes and setup a step by step instructions after we are done. Carlos. From: David Trudgian <David.Trudgian@UTSouthwestern.edu<mailto:David.Trudgian@UTSouthwestern.edu>> Date: Tuesday, September 22, 2015 at 11:53 AM To: Nicola Soranzo <nsoranzo@tiscali.it<mailto:nsoranzo@tiscali.it>>, Carlos Lijeron <clijeron@hunter.cuny.edu<mailto:clijeron@hunter.cuny.edu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" <galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>> Subject: RE: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, We aren’t using Active Directory here – but we do use an OpenLDAP directory which our cluster authenticates against, as does Galaxy. In order to track cluster usage for users running Galaxy jobs we use galaxy-pulsar between Galaxy itself and our SLURM cluster. Pulsar is configured to submit all jobs as a real user on the cluster, via SLURM-DRMAA. This means that all of the Galaxy usage on the cluster appears in our SLURM accounting database just like any other job would. The complexity here is file ownership. Pulsar has to copy all input files into a staging directory, and change ownership to the real user for the job to run on the cluster. It is/was(?) a little complex to setup, as there are more parts involved then a typical Galaxy install, but it works great for us here. When I was getting this setup John Chilton mentioned adding a PulsarEmbedded runner into Galaxy at some point. Not sure whether this has/is happening, but it’d make these situations easier: https://trello.com/c/4YwVZBtq/1865-embedded-pulsar-job-runner DT -- David Trudgian Ph.D. Computational Scientist, BioHPC UT Southwestern Medical Center Dallas, TX 75390-9039 Tel: (214) 648-4833 From: galaxy-dev [mailto:galaxy-dev-bounces@lists.galaxyproject.org] On Behalf Of Nicola Soranzo Sent: Tuesday, September 22, 2015 5:59 AM To: Carlos Lijeron <clijeron@hunter.cuny.edu<mailto:clijeron@hunter.cuny.edu>>; galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote: Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ________________________________ UT Southwestern Medical Center The future of medicine, today.
Good day Nicola, Thank you for the advise on LDAP authentication. Did you need to run Galaxy on Apache to enable this, or did you simply change some configuration files to enable this. I think we’ll be able to write some scripts using SLURM commands to pull data about resource utilization, but my main concern is whether we need Apache to enable LDAP authentication of galaxy users. Thanks again ! Carlos From: Nicola Soranzo <nicola.soranzo@gmail.com<mailto:nicola.soranzo@gmail.com>> on behalf of Nicola Soranzo <nsoranzo@tiscali.it<mailto:nsoranzo@tiscali.it>> Date: Tuesday, September 22, 2015 at 6:59 AM To: Carlos Lijeron <clijeron@hunter.cuny.edu<mailto:clijeron@hunter.cuny.edu>>, "galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" <galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>> Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking. Cheers, Nicola On 21/09/15 20:19, Carlos Lijeron wrote: Everyone, We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP. Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise. Thank you all ! Carlos. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Carlos! No, you don't need Apache. You can also use multiple authentication mechanisms at the same time, e.g. LDAP for users of a particular domain and the usual Galaxy database auth for the others. The configuration file you need to modify is config/auth_conf.xml, the options are documented on the .sample file (look at https://github.com/galaxyproject/galaxy/blob/dev/config/auth_conf.xml.sample for the most updated version). Cheers, Nicola On 23/09/15 20:10, Carlos Lijeron wrote:
Good day Nicola,
Thank you for the advise on LDAP authentication. Did you need to run Galaxy on Apache to enable this, or did you simply change some configuration files to enable this. I think we’ll be able to write some scripts using SLURM commands to pull data about resource utilization, but my main concern is whether we need Apache to enable LDAP authentication of galaxy users.
Thanks again !
Carlos
From: Nicola Soranzo <nicola.soranzo@gmail.com <mailto:nicola.soranzo@gmail.com>> on behalf of Nicola Soranzo <nsoranzo@tiscali.it <mailto:nsoranzo@tiscali.it>> Date: Tuesday, September 22, 2015 at 6:59 AM To: Carlos Lijeron <clijeron@hunter.cuny.edu <mailto:clijeron@hunter.cuny.edu>>, "galaxy-dev@lists.galaxyproject.org <mailto:galaxy-dev@lists.galaxyproject.org>" <galaxy-dev@lists.galaxyproject.org <mailto:galaxy-dev@lists.galaxyproject.org>> Subject: Re: [galaxy-dev] Galaxy on a Cluster -- Active Directory LDAP configuration
Hi Carlos, we are using Active Directory LDAP for user authentication, which works pretty well, but only the Galaxy user to submit jobs to the LSF cluster queue, so I can't help with the resource tracking.
Cheers, Nicola
On 21/09/15 20:19, Carlos Lijeron wrote:
Everyone,
We are setting up Galaxy to work with our cluster and SLURM as the work manager. The cluster itself authenticates to our local Active Directory, so I’m wondering if the best way to track resource utilization of Galaxy users on the cluster is to also have Galaxy authenticate to the same Active Directory LDAP.
Is anyone on this list using the same configuration and tracking resource utilization from Galaxy users submitting jobs to the cluster nodes? Please advise.
Thank you all !
Carlos.
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Carlos Lijeron -
David Trudgian -
Nicola Soranzo