Re: [galaxy-dev] External user auth and API

8 Aug 2016


      ...
Hi Eric,
thanks for the hint regarding uWSGI. What doesn't work is importing
files from the local disk (of the galaxy server) to the data library
(see attached screenshot). Everything else seems to be fine, I haven't
encountered any other issues. Maybe it's a bug just in that function?
Here is the error:
10.184.132.10 - - [30/Jul/2016:18:09:27 +0200] "POST
/api/libraries/datasets?encoded_folder_id=F7b46bd6d01de922f&source=userdir_file&path=160308_WTCHG_254732_201.bam&file_type=auto&dbkey=?
HTTP/1.1" 500 - "https://galaxy-server.uni.lu/library/list"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0"
Error - <type 'exceptions.AssertionError'>: use_remote_user is set but
HTTP_REMOTE_USER header was not provided
URL:
https://galaxy-server.uni.lu/api/libraries/datasets?encoded_folder_id=F7b46bd6d01de922f&source=userdir_file&path=160308_WTCHG_254732_201.bam&file_type=auto&dbkey=?
File
'/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py',
line 151 in __call__
  app_iter = self.application(environ, sr_checker)
File
'/mnt/gaiagpfs/projects/galaxy/internal/.venv/local/lib/python2.7/site-packages/paste/recursive.py',
line 85 in __call__
  return self.application(environ, start_response)
File
'/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py',
line 76 in __call__
  return self.app( environ, start_response )
File
'/mnt/gaiagpfs/projects/galaxy/internal/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py',
line 640 in __call__
  return self.application(environ, start_response)
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line
131 in __call__
  return self.handle_request( environ, start_response )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line
158 in handle_request
  trans = self.transaction_factory( environ )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py',
line 68 in <lambda>
  self.set_transaction_factory( lambda e: self.transaction_chooser( e,
galaxy_app, session_cookie ) )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py',
line 99 in transaction_chooser
  return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py',
line 198 in __init__
  self.error_message = self._authenticate_api( session_cookie )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py',
line 380 in _authenticate_api
  self._ensure_valid_session( session_cookie )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py',
line 432 in _ensure_valid_session
  "use_remote_user is set but %s header was not provided" %
self.app.config.remote_user_header
AssertionError: use_remote_user is set but HTTP_REMOTE_USER header was
not provided
I'm afraid I do not have any help for this. Hopefully another galaxy
Hi Sarah,

On 08. aug. 2016 13:58, Sarah DIEHL wrote:
person can chime in here and help debug this.
...
Best regards,
Sarah
----
Sarah Diehl
HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
sarah.diehl@uni.lu <mailto:sarah.diehl@uni.lu>     http://lcsb.uni.lu
<https://urldefense.proofpoint.com/v2/url?u=http-3A__lcsb.uni.lu_&d=CwMF-g&c=ODFT-G5SujMiGrKuoJJjVg&r=p9uZby14OqW9zcjBSjiDKw&m=pwYDgdauhpWnUSnnqVAB6Xyn2OzpGsIMkWdC8fyNRu0&s=Ll-0NRWHk-0Kb159GLi8cgxakoFwK4BQyp39IRY0S48&e=>
-----
This message is confidential and may contain privileged information.
It is intended for the named recipient only. If you receive it in
error please notify me and permanently delete the original message and
any copies.
-----
From: Eric Rasche <esr@tamu.edu <mailto:esr@tamu.edu>>
Organization: TAMU
Date: Monday 8 August 2016 15:43
To: Sarah DIEHL <sarah.diehl@uni.lu <mailto:sarah.diehl@uni.lu>>,
"galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>"
<galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: Re: [galaxy-dev] External user auth and API
...
Hi Eric,
thanks a lot for the info and help! I'm running version 16.04 and my
apache conf is a bit different because I balance over multiple galaxy
web servers:
Just as an aside, if you are still using this configuration method,
you might consider switching to uWSGI
<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.galaxyproject.org_Admin_Config_Performance_Scaling&d=CwMF-g&c=ODFT-G5SujMiGrKuoJJjVg&r=p9uZby14OqW9zcjBSjiDKw&m=pwYDgdauhpWnUSnnqVAB6Xyn2OzpGsIMkWdC8fyNRu0&s=FcalyPc9I-0TF6rCq1c3yxRT6-Rrzmf7mNE3GVfCNBI&e=>,
it will drastically simplify your configuration and should be just as
On 08. aug. 2016 13:25, Sarah DIEHL wrote:
performant (if not more).
...
# API requests get routed through balancer with no authentication
    RewriteCond %{QUERY_STRING} key=
    RewriteRule ^/api/(.*) balancer://galaxy-noauth/api/$1 [P]
# Regular requests get routed through balancer with LDAP
authentication
    RewriteRule ^(.*) balancer://galaxy$1 [P]
<Proxy balancer://galaxy/*>
        BalancerMember  http://localhost:8080
        BalancerMember  http://localhost:8081
        BalancerMember  http://localhost:8082
        BalancerMember  http://localhost:8083
RequestHeader set X-URL-SCHEME https
AuthType Basic
        AuthBasicProvider ldap
        AuthName               "UL HPC Platform Authorized Users Only"
        AuthLDAPBindAuthoritative off
        AuthLDAPURL "ldap://…"
        Require valid-user
        RequestHeader set REMOTE_USER %{AUTHENTICATE_uid}e
XSendFile on
        XSendFilePath /
SetOutputFilter DEFLATE
        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI /history/export_archive no-gzip
dont-vary
    </Proxy>
<Proxy balancer://galaxy-noauth/*>
        BalancerMember  http://localhost:8080
        BalancerMember  http://localhost:8081
        BalancerMember  http://localhost:8082
        BalancerMember  http://localhost:8083
RequestHeader set X-URL-SCHEME https
Satisfy any
XSendFile on
        XSendFilePath /
SetOutputFilter DEFLATE
        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI /history/export_archive no-gzip
dont-vary
    </Proxy>
What doesn't work when configured this way is data libraries' import
from user directory function. Can you tell if my apache configuration
is equivalent to yours?
Your configuration /looks/ equivalent, yes, but I may have overlooked
something, I have since forgotten some of the semantics of proxy
balancers.
...
Does the import functionality in the data libraries work for you?
Import datasets from data libraries to history? Yes, this does for me.
In what way does it break for you? Unauthenticated /api requests? It
seems odd that data libraries would be the only affected thing here,
since other things like the tool panel also make /api requests.
This configuration worked fine with release 15.10, but doesn't
anymore since I upgraded. Between those two version the default data
libraries interface changed. What used to be "beta" before is now the
default and only available option.
Old data libraries were deprecated
<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.galaxyproject.org_en_master_releases_15.10-5Fannounce.html-23deprecation-2Dnotices&d=CwMF-g&c=ODFT-G5SujMiGrKuoJJjVg&r=p9uZby14OqW9zcjBSjiDKw&m=pwYDgdauhpWnUSnnqVAB6Xyn2OzpGsIMkWdC8fyNRu0&s=p6Y66XyId9RfOGHSilb-eCggUDdkXaloeEyrXZBLQbM&e=>
in favour of the new ones and removed in 16.01.
Best regards,
Sarah
----
Sarah Diehl
HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
sarah.diehl@uni.lu <mailto:sarah.diehl@uni.lu>     http://lcsb.uni.lu
-----
This message is confidential and may contain privileged information.
It is intended for the named recipient only. If you receive it in
error please notify me and permanently delete the original message
and any copies.
-----
From: Eric Rasche <esr@tamu.edu>
Organization: TAMU
Date: Monday 8 August 2016 15:09
To: Sarah DIEHL <sarah.diehl@uni.lu>,
"galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>"
<galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: Re: [galaxy-dev] External user auth and API
Hi Sarah,
On 08. aug. 2016 07:44, Sarah DIEHL wrote:
...
Dear all,
since no one replied so far to the main problem I had and it might
have gotten lost in the conversation, I ask again: Does somebody
know how to configure external user auth with apache such that API
(from external, e.g. bioblend) and dataset import in the data
libraries work? When I configure apache to require auth for
everything, the API does not work. If I except the API from the
apache auth, the dataset import does not work.
Our configuration looks like the following (just switching CAS for LDAP.)
<Location "/galaxy/api/">
        Satisfy Any
        Allow from all
    </Location>
<Location "/galaxy">
        AuthName "CAS"
        AuthType CAS
        Require valid-user
        RequestHeader set X-URL-SCHEME https
        XSendFile on
        XSendFilePath /
        RequestHeader set CAS-User "%{REMOTE_USER}s@tamu.edu"
    </Location>
    ProxyPass /galaxy uwsgi://127.0.0.1:4001/
I.e. we disable authentication on the /api route. On 16.01+ (I think
it was patched then, but 16.04 is a safer bet) this will work
correctly and your users will be able to use the API. On previous
versions the /api route would fail for web users if exposed in this
manner.
...
If I switch to the new galaxy-internal LDAP auth features, will that
solve this problem?
Yes, this is an alternate solution.
...
Any hints are appreciated!
Best regards,
Sarah
----
Sarah Diehl
HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
<mailto:sarah.diehl@uni.lu>sarah.diehl@uni.lu     http://lcsb.uni.lu
-----
This message is confidential and may contain privileged information.
It is intended for the named recipient only. If you receive it in
error please notify me and permanently delete the original message
and any copies.
-----
From: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> on
behalf of Sarah DIEHL <sarah.diehl@uni.lu <mailto:sarah.diehl@uni.lu>>
Date: Monday 1 August 2016 13:06
To: Nicola Soranzo <nsoranzo@tiscali.it>,
"galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>"
<galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: Re: [galaxy-dev] Remote user auth and API
Hi Nicola,
thanks a lot for the help! Yes, it's a self-signed certificate, I
didn't bother with letsencrypt yet ;-).
So now the error turned to
ConnectionError: GET: error 401: b'<!DOCTYPE HTML PUBLIC
"-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>401 Authorization
Required</title>\n</head><body>\n<h1>Authorization
Required</h1>\n<p>This server could not verify that you\nare
authorized to access the document\nrequested. Either you supplied
the wrong\ncredentials (e.g., bad password), or your\nbrowser
doesn\'t understand how to supply\nthe credentials
required.</p>\n</body></html>\n', 0 attempts left: None
which is what I expected, since apache now wants the authentication
through LDAP.
So anybody know what the right settings are to get both the dataset
import and the API working with external user auth over apache and LDAP?
Thanks,
Sarah
----
Sarah Diehl
HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
<mailto:sarah.diehl@uni.lu>sarah.diehl@uni.lu     http://lcsb.uni.lu
-----
This message is confidential and may contain privileged information.
It is intended for the named recipient only. If you receive it in
error please notify me and permanently delete the original message
and any copies.
-----
From: Nicola Soranzo <nicola.soranzo@gmail.com
<mailto:nicola.soranzo@gmail.com>> on behalf of Nicola Soranzo
<nsoranzo@tiscali.it>
Date: Monday 1 August 2016 12:58
To: Sarah DIEHL <sarah.diehl@uni.lu <mailto:sarah.diehl@uni.lu>>,
"galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>"
<galaxy-dev@lists.galaxyproject.org
<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: Re: [galaxy-dev] Remote user auth and API
Hi Sarah!
I guess that your problem is with an untrusted certificate, you can
get one for free at https://letsencrypt.org/
You can disable certificate verification in bioblend as in the
example below:
import bioblend.galaxy
gi = bioblend.galaxy.GalaxyInstance(url=my_server, key=my_key)
gi.verify = False
Cheers,
Nicola
On 01/08/16 09:08, Sarah DIEHL wrote:
...
Dear all,
since the recent update to 16.04 I get the following error when
trying to import a file from a user directory to a data library:
AssertionError: use_remote_user is set but HTTP_REMOTE_USER header
was not provided
I use apache as a proxy and use an LDAP server for authentication.
In order to get the API to work previously the apache had to be set
to not check authentication for the requests to /api. In the logs I
can see that the dataset import is an request to the API, so since
the auth is not checked then, there is also no REMOTE_USER header set.
What is the recommended way to solve this issue with the current
Galaxy version? I disabled the special settings for /api and the
dataset import works now.
I tried to check the API with an old test script based on bioblend,
but I now get the following error:
ConnectionError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:645), 0 attempts left: None
Previously I could disable it with
import requests
requests.packages.urllib3.disable_warnings()
but that doesn't seem to work anymore (switched to Python 3 now).
Since bioblend wraps all the requests methods, I cannot apply any
of the common solutions I found online (e.g. set verify=False).
Any help to solve these issues is highly appreciated :-).
Best regards,
Sarah
----
Sarah Diehl
HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
<mailto:sarah.diehl@uni.lu>sarah.diehl@uni.lu     http://lcsb.uni.lu
-----
This message is confidential and may contain privileged
information. It is intended for the named recipient only. If you
receive it in error please notify me and permanently delete the
original message and any copies.
-----
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
-- 
Eric Rasche
Programmer II
Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
esr@tamu.edu <mailto:esr@tamu.edu>
-- 
Eric Rasche
Programmer II
Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
esr@tamu.edu <mailto:esr@tamu.edu>
-- 
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
esr@tamu.edu <mailto:esr@tamu.edu>