Nginx+Shibboleth+UCSC
by Martin Demko
Dear list,
I have been trying to manage an access to UCSC genome browser from our Galaxy
instance which uses Nginx as a proxy with external authentication via
Shibboleth and IdP service.
Not even the configuration of nginx+shibboleth was quite complicated to
achieve (but I succeeded after a week or so) but now during a testing phase, we
have found out that a cooperation with external UCSC servers is forbidden due
to our auth. configuration.
I have found a way how to do that but only for Apache (https://docs.galaxyproject.org/en/master/admin/special_topics/apache.html...) so I tried to do something similar in Nginx but only with IP addresses as Nginx doesn't support using of named domains and defined a location:
location /display_as {
satisfy any;
deny all;
allow 128.114.119.131;
allow 128.114.119.132;
allow 128.114.119.133;
allow 128.114.119.134;
allow 128.114.119.135;
allow 128.114.119.136;
}
Used IP addresses should be the right ones for UCSC servers according to few
sources such as (https://genome.soe.ucsc.narkive.com/sll2JSk2/ucsc-ip-address):
hgw1.cse.ucsc.edu
hgw2.cse.ucsc.edu
hgw3.cse.ucsc.edu
hgw4.cse.ucsc.edu
hgw5.cse.ucsc.edu
hgw6.cse.ucsc.edu
I tried to change the location also to "/display_application" but didn't help
(actually, I don't know why in the documentation there is used as the location
that "/display_as" as I have never encountered such location in Galaxy which is
my first question). In galaxy.ini I set:
display_servers = hgw1.cse.ucsc.edu,hgw2.cse.ucsc.edu,hgw3.cse.ucsc.edu,hgw4.cse.ucsc.edu,hgw5.cse.ucsc.edu,hgw6.cse.ucsc.edu,hgw7.cse.ucsc.edu,hgw8.cse.ucsc.edu,lowepub.cse.ucsc.edu,128.114.119.131,128.114.119.132,128.114.119.133,128.114.119.134,128.114.119.135,128.114.119.136
Just for case I used both, the domains and also IP addresses. Also I have
tried to use only the domains or only the addresses but nothing helped.
Then I found out after reading of the following (https://www.switch.ch/aai/guides/sp/access-rules/) that the main obstacle should be actually shibboleth so I defined in /etc/shibboleth/shibboleth2.xml an unrestricted path:
<RequestMapper type="XML">
<RequestMap>
<Host name="our.galaxy.something"
authType="shibboleth"
requireSession="true"
redirectToSSL="443" >
<Path name="display_application" requireSession="false" redirectToSSL="443" />
<!--Path name="display_as" authType="shibboleth" requireSession="false"
redirectToSSL="443" /-->
</Host>
</RequestMap>
</RequestMapper>
Again, I have tried to use both locations "display_as" and "display_application", but it
haven't solved my problem entirely. I moved forward I suppose because now the
requests
should be going around the authentication but UCSC stil warns me that:
Expected 200 https://our.galaxy.something/display_application/e1304269a2f56a52/ucsc_bi...: 403 Forbidden
Before that it was a different message containing the address of our idp
server so I believe I'am on the right way here. The presence of attribute
'authType="shibboleth"' seams to be useless as 'requireSession="false"' is
used.
Then I tried to teach Nginx to use also the domain names with the using of 3rd
party nginx-http-rdns module from (https://github.com/flant/nginx-http-rdns),
so my nginx.conf looked like:
....only the part of nginx.conf....
############## Shibboleth authentication conf #################
# FastCGI authorizer for Shibboleth Auth Request module
location = /shibauthorizer {
internal;
include fastcgi_params;
fastcgi_pass unix:/var/run/shibboleth/shibauthorizer.sock;
}
# FastCGI responder for SSO
location /Shibboleth.sso {
include fastcgi_params;
fastcgi_pass unix:/var/run/shibboleth/shibresponder.sock;
}
############### Entry point for Galaxy ###############
# Location secured by Shibboleth
location / {
shib_request /shibauthorizer;
more_clear_input_headers 'Variable-*' 'Shib-*' 'Remote-User' 'REMOTE_USER'
'Auth-Type' 'AUTH_TYPE' 'GX_SECRET';
include uwsgi_params;
shib_request_set $shib_id $upstream_http_variable_eppn;
uwsgi_param HTTP_REMOTE_USER $shib_id;
uwsgi_param HTTP_GX_SECRET $our_secret;
uwsgi_pass 127.0.0.1:4001;
# resolver $correct_resolver_acc_to_resolv.conf;
# rdns on;
# satisfy any;
# rdns_allow genome\.ucsc\.edu;
# rdns_allow hgw1\.cse\.ucsc\.edu;
# rdns_allow hgw2\.cse\.ucsc\.edu;
# rdns_allow hgw3\.cse\.ucsc\.edu;
# rdns_allow hgw4\.cse\.ucsc\.edu;
# rdns_allow hgw5\.cse\.ucsc\.edu;
# rdns_allow hgw6\.cse\.ucsc\.edu;
# rdns_allow hgw7\.cse\.ucsc\.edu;
# rdns_allow hgw8\.cse\.ucsc\.edu;
}
# location /display_as {
# resolver $correct_resolver_acc_to_resolv.conf;
# rdns on;
# satisfy any;
# rdns_allow genome\.ucsc\.edu;
# rdns_allow hgw1\.cse\.ucsc\.edu;
# rdns_allow hgw2\.cse\.ucsc\.edu;
# rdns_allow hgw3\.cse\.ucsc\.edu;
# rdns_allow hgw4\.cse\.ucsc\.edu;
# rdns_allow hgw5\.cse\.ucsc\.edu;
# rdns_allow hgw6\.cse\.ucsc\.edu;
# rdns_allow hgw7\.cse\.ucsc\.edu;
# rdns_allow hgw8\.cse\.ucsc\.edu;
# }
# location /display_application {
# resolver $correct_resolver_acc_to_resolv.conf;
# rdns on;
# satisfy any;
# rdns_allow genome\.ucsc\.edu;
# rdns_allow hgw1\.cse\.ucsc\.edu;
# rdns_allow hgw2\.cse\.ucsc\.edu;
# rdns_allow hgw3\.cse\.ucsc\.edu;
# rdns_allow hgw4\.cse\.ucsc\.edu;
# rdns_allow hgw5\.cse\.ucsc\.edu;
# rdns_allow hgw6\.cse\.ucsc\.edu;
# rdns_allow hgw7\.cse\.ucsc\.edu;
# rdns_allow hgw8\.cse\.ucsc\.edu;
# }
....END of the part of nginx.conf....
Everything commented out, I tried to use together or one at the time but
nothing helped. But there are differences:
1) when uncommented only location /display_application - I will get just a
blank page with: 404 Not Found.
2) when uncommented only location /display_as - nothing changes.
3) when uncommented part inside the location "/" - I will get a new message:
Error unexpected end of input reading http header on https://our.galaxy.something/display_application/e1304269a2f56a52/ucsc_bi...
And now I'm out of ideas. I would really appreciate any help.
PS: I stil don't get why in the documentation for Apache configuration
(https://docs.galaxyproject.org/en/master/admin/special_topics/apache.html...)
there is used the /display_as location. Can somebody explain, please?
Thank you in advance,
Martin Demko
3 months
Galaxy Administrator Training, Jan 28 - Feb 1, Penn State University
by Dave Clements
Hello all,
We are pleased to announce that* the 2019 Galaxy Admin Training
<https://galaxyproject.org/events/2019-admin-training/> will be offered
January 28 through February 1 at Penn State University* (where the Galaxy
Project got started over ten years ago). The workshop offers a 2 day
introductory session
<https://github.com/galaxyproject/dagobah-training#basic-sessions> followed
by a 3 day advanced topics session
<https://github.com/galaxyproject/dagobah-training#advanced-sessions>.
Participants can register for one or both sessions.
This workshop will cover what you need to know to set up your own
high-performance and multi-user production Galaxy instance. Sessions will
be *intensive and hands-on*, and taught by experienced instructors from the
Galaxy Community. Participants will learn how to install, configure,
customize, and extend their own Galaxy servers. Topics include tool
configuration, authentication and user management, using heterogeneous
storage and compute services, and many other topics that will enable you to
get your own Galaxy server up and running, performing well, and used by
your community.
Registration
<https://galaxyproject.org/events/2019-admin-training/#registration> is now
open and starts at $40 / day for participants from non-profits and
academia. Early registration ends October 31. However, the 2016 and 2018
admin training were both full, so you are strongly encouraged to *register
now*.
*The workshop does have prerequisites
<https://galaxyproject.org/events/2019-admin-training/#prerequisites>.
Please read them before you register.*
Cheers,
Dave C.
*About Galaxy *
Galaxy is an open web based platform for biomedical data integration and
analysis. It is deployed at large and small organizations around the world
and used in a broad range of biomedical research domains, and across the
tree of life.
--
https://galaxyproject.org/
https://usegalaxy.org/
7 months
XMLs not reloading in all processes (UWSGI)?
by D K
I'm running version 18.05 and have noticed that when updating the
tool_conf.xml that changes don't always get loaded in all processes (e.g.
if I hit reload sometimes my changes show up and sometimes they do not).
Also, if I try to run a new tool I've added sometimes this will fail
presumably because that process doesn't have the update. The only fix for
this seems to be a galaxy restart.
I've turned watchdog on and am using the UWSGI + mules strategy.
Any suggestions?
7 months, 2 weeks
installation is not finishing
by Jochen Bick
Hi,
I wanted to install two tools but my installation is stuck in Installing
tool dependencies. So I found out that the xmls are cloned but the
dependencies are not getting installed.
It looks like anything related to python is not getting installed?
anyone needs some log file, please tell me.
Cheers Jochen
--
ETH Zurich
*Jochen Bick*
Animal Physiology
Institute of Agricultural Sciences
Postal address: Universitätstrasse 2 / LFW B 58.1
8092 Zurich, Switzerland
Office: Eschikon 27
8315 Lindau, Switzerland
Phone +41 52 354 92 06
jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch>
www.ap.ethz.ch
7 months, 2 weeks
number of elements in collections and multiple-true-params
by Matthias Bernt
Dear list,
in a tool I want to process a set of (bam) files. For the convenience of
the user I thought that it would be nice to allow collections as well as
manually selected files (in a param with multiple=true). Like so:
```
<conditional name="input_type">
<param name="input_type_selector" type="select" label="BAM files">
<option value="list">data set list</option>
<option value="manual">data sets</option>
</param>
<when value="manual">
<param name="input_bam" format="bam" type="data"
multiple="true" optional="false" label="BAM files" />
</when>
<when value="list">
<param name="input_bam" format="bam" type="data_collection"
collection_type="list" label="BAM files" />
</when>
</conditional>
```
Is there a way to get the number of data sets in both cases (in the
collection or selection) in a consistent way?
Currently the only way I found is:
```
#if $input_type.input_type_selector == 'manual':
#set count = len($input_type.input_bam)
#else
#set count = len($input_type.input_bam.keys())
#end if
```
Appart from this looping works like a charm:
```
#for $bam in $input_type.input_bam:
OPERATIONS...
#end for
```
Cheers,
Matthias
--
-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
m.bernt(a)ufz.de, www.ufz.de
Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
-------------------------------------------
7 months, 3 weeks
Multifactor RNA-Seq Analysis and interaction terms
by Lance Parsons
One of the labs I work with regularly uses Galaxy for their RNA-Seq
analysis. They have been doing very simple analyses up to this point,
using DESeq2. However, they are moving toward more complicated
experimental designs that will include additional factors, some of which
(genotype for example) will be interaction terms
<http://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/D...>.
I have a few questions:
1.
I know Marius van den Beek <https://github.com/mvdbeek> added group
tags <https://github.com/galaxyproject/galaxy/pull/5457> in 18.09,
which seem like they are at least one way to use multiple factors
along with collections. Is there any primer/tutorial on using these?
It’s not quite clear to me just yet.
2.
Does anyone have any thoughts or suggestions as to how one could add
interaction terms (/e.g./ |condition:genotype|) to the DESeq2
wrapper? Does this functionality exist in a different Galaxy tool?
Thanks in advance.
--
Lance Parsons - Scientific Programmer
Carl C. Icahn Laboratory - Room 136
Lewis-Sigler Institute for Integrative Genomics
Princeton University
8 months, 1 week
Re: [galaxy-dev] Running Cloudlaunch against a new OpenStack cloud
by Peter van Heusden
Hi there
I've got a bit further, but now I've got a problem - how do you access the
admin panel when you have the complete site set up?
1) If Debug = False (as it is for a production setup), Django doesn't load
static files (e.g. CSS).
2) When going via the nginx proxy, I can't figure out what URL will direct
me to the admin page.
Peter
P.S. I'm doing some work on the ansible to get this all set up - I'll do a
PR once everything seems to be working.
On Fri, 19 Oct 2018 at 01:25, Alexandru Mahmoud <amahmou4(a)jhu.edu> wrote:
> Hello!
>
> I am new to the team, so this will be a brief response to only one of your
> questions while waiting for someone else to get to the email (and I am not
> replying all so that people don’t dismiss it as answered):
>
> For 3 specifically, you can access the admin panel at http://127.0.0.1(or
> localhost):8000/admin, where you should be able to login with the user
> created with the ‘createsuperuser’ command. Under “Appliances” or
> “Applications” (depending on the version of cloudlaunch) you’ll be able to
> see and change/add configurations for apps.
> If you want to see in more detail how the appliances are being set up and
> how the launch config is used in code, look for the "Backend component
> name", then find the corresponding backend plugin in the cloudlaunch repo.
> I hope that is a helpful start, and feel free to email me if nobody
> responds in a reasonable amount of time to your other questions, as I can
> get more answers in person and relay them to you.
>
> Best,
> Alex Mahmoud
>
>
> On Oct 18, 2018, at 3:45 PM, Peter van Heusden <pvh(a)sanbi.ac.za> wrote:
>
> Hi there
>
> I'd like to set up cloudlaunch to be able to set up on-demand Galaxy
> servers and associated clusters on the new Ilifu OpenStack cloud. From
> looking around the repositories it looks like Cloudlaunch is at least 2
> components:
>
> 1. The cloudlaunch server
> 2. The UI - either cloudlaunch-ui or cloudlaunch-cli
>
> I'm trying to get cloudlaunch server installed on a Ubuntu 18.04 VM. The
> only requirement seems to be python3-pip. For postgresql support I
> installed the python3-psycopg2 module, the postgresql-10 server and create
> a user and db (both named cloud). Then I installed the server with `pip3
> install cloudlaunch-server` and ran commands:
>
> cloudlaunch-server django migrate
> cloudlaunch-server django createsuperuser
> cloudlaunch-server django runserver
>
> The next step (https://github.com/galaxyproject/cloudlaunch) refers to
> redis-server. Installing this package seemed to satisfy that requirement,
> running it manually didn't work so I just ran the celery command as
> illustrated. And finally relaunched the django server.
>
> Then nodejs and npm were installed, the cloudlaunch-ui cloned from github,
> the various npm package specified installed and finally npm start brings up
> the cloudlaunch user interface.
>
> So far so good but:
>
> 1. Is there more extensive documentation anywhere? Specifically
> documentation about running this in production.
>
> 2. Is there a docker container that can be used instead of all this manual
> installation?
>
> 3. How are appliances configured and added to the Appliance Catalogue?
>
> Thanks!
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/
>
>
>
8 months, 1 week
Re: [galaxy-dev] Search box in tools menu / uncomplete wrappers list
by Matthias Bernt
Hi,
I observed similar Problems on my Galaxy. But I think not all cases can
be explained by Martins answer. For instance, if I search for "hisat" I
don't get any result, but HISAT2 is definitely installed.
Cheers,
Matthias
> Message: 1
> Date: Fri, 9 Nov 2018 15:54:16 +0000
> From: Sarah Maman <sarah.maman(a)inra.fr>
> To: "galaxy-dev(a)lists.galaxyproject.org"
> <galaxy-dev(a)lists.galaxyproject.org>
> Subject: [galaxy-dev] Search box in tools menu / uncomplete wrappers
> list
> Message-ID: <1541778855881.43440(a)inra.fr>
> Content-Type: text/plain; charset="iso-8859-1"
>
>
> Hello,
>
>
> When I'm searching a keyword in the « search » box in left « tools » menu, results are sometimes false because the wrappers' list generated by the query thanks to a keyword is not full, some tools are missing. In fact, some wrappers are not listed even if the keyword is in the tool name and/or the tool description.
>
> Could you please help us to understand why ?
>
>
> Thanks,
>
> Sarah Maman
>
>
>
>
> _____________________________
> Sarah Maman
>
> Bioinformatique/Bioinformatics
>
> INRA | SIGENAE | GenPhySE
>
> Centre Inra Occitanie-Toulouse
> Tél. : +33(0)5.61.28.57.08
>
> _____________________________
>
> Chemin de Borde Rouge CS 52627
> 31326 Castanet Tolosan cedex
>
> www.toulouse.inra.fr<http://www.toulouse.inra.fr/>
>
> Suivez-nous sur Twitter !
>
8 months, 1 week
Bug in data manager for the STAR aligner indices
by Previti
Dear all,
I've been wanting to point this out for a while, but the latest update
to the STAR data manager didn't fix it either (or maybe I messed up again).
The .loc file indicating which genomes have been installed/indices have
been made for STAR uses the last column to indicate if there
is a gene model/annotation that it should use (0 - for no, 1 - for yes).
This seems to be empty...there is a tab at the end of the row, but no
number.
It works if you manually but in 1 or 0, but not that's not very convenient.
Please let me know if I'm missing something...
Cheers,
Christopher
--
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician
German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661
christopher.previti(a)dkfz.de <http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de/>
Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537
Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.
8 months, 1 week
