Hi Eric,

thanks for the hint regarding uWSGI. What doesn't work is importing files from the local disk (of the galaxy server) to the data library (see attached screenshot). Everything else seems to be fine, I haven't encountered any other issues. Maybe it's a bug just in that function? Here is the error:

10.184.132.10 - - [30/Jul/2016:18:09:27 +0200] "POST /api/libraries/datasets?encoded_folder_id=F7b46bd6d01de922f&source=userdir_file&path=160308_WTCHG_254732_201.bam&file_type=auto&dbkey=? HTTP/1.1" 500 - "https://galaxy-server.uni.lu/library/list" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
Error - <type 'exceptions.AssertionError'>: use_remote_user is set but HTTP_REMOTE_USER header was not provided
URL: https://galaxy-server.uni.lu/api/libraries/datasets?encoded_folder_id=F7b46bd6d01de922f&source=userdir_file&path=160308_WTCHG_254732_201.bam&file_type=auto&dbkey=?
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 151 in __call__
  app_iter = self.application(environ, sr_checker)
File '/mnt/gaiagpfs/projects/galaxy/internal/.venv/local/lib/python2.7/site-packages/paste/recursive.py', line 85 in __call__
  return self.application(environ, start_response)
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py', line 76 in __call__
  return self.app( environ, start_response )
File '/mnt/gaiagpfs/projects/galaxy/internal/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py', line 640 in __call__
  return self.application(environ, start_response)
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 131 in __call__
  return self.handle_request( environ, start_response )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 158 in handle_request
  trans = self.transaction_factory( environ )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 68 in <lambda>
  self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 99 in transaction_chooser
  return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 198 in __init__
  self.error_message = self._authenticate_api( session_cookie )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 380 in _authenticate_api
  self._ensure_valid_session( session_cookie )
File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 432 in _ensure_valid_session
  "use_remote_user is set but %s header was not provided" % self.app.config.remote_user_header
AssertionError: use_remote_user is set but HTTP_REMOTE_USER header was not provided


Best regards,
Sarah

----
Sarah Diehl
HPC System Administrator
 
UNIVERSITÉ DU LUXEMBOURG
 
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
-----
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
-----


From: Eric Rasche <esr@tamu.edu>
Organization: TAMU
Date: Monday 8 August 2016 15:43
To: Sarah DIEHL <sarah.diehl@uni.lu>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] External user auth and API



On 08. aug. 2016 13:25, Sarah DIEHL wrote:
Hi Eric,

thanks a lot for the info and help! I'm running version 16.04 and my apache conf is a bit different because I balance over multiple galaxy web servers:
Just as an aside, if you are still using this configuration method, you might consider switching to uWSGI, it will drastically simplify your configuration and should be just as performant (if not more).



    # API requests get routed through balancer with no authentication
    RewriteCond %{QUERY_STRING} key=
    RewriteRule ^/api/(.*) balancer://galaxy-noauth/api/$1 [P]

    # Regular requests get routed through balancer with LDAP authentication
    RewriteRule ^(.*) balancer://galaxy$1 [P]

    <Proxy balancer://galaxy/*>
        BalancerMember  http://localhost:8080
        BalancerMember  http://localhost:8081
        BalancerMember  http://localhost:8082
        BalancerMember  http://localhost:8083

RequestHeader set X-URL-SCHEME https

        AuthType Basic
        AuthBasicProvider ldap
        AuthName               "UL HPC Platform Authorized Users Only"
        AuthLDAPBindAuthoritative off
        AuthLDAPURL "ldap://…"
        Require valid-user
        RequestHeader set REMOTE_USER %{AUTHENTICATE_uid}e

        XSendFile on
        XSendFilePath /

        SetOutputFilter DEFLATE
        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary
    </Proxy>

    <Proxy balancer://galaxy-noauth/*>
        BalancerMember  http://localhost:8080
        BalancerMember  http://localhost:8081
        BalancerMember  http://localhost:8082
        BalancerMember  http://localhost:8083

RequestHeader set X-URL-SCHEME https

        Satisfy any

        XSendFile on
        XSendFilePath /

        SetOutputFilter DEFLATE
        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary
        SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary
    </Proxy>

What doesn't work when configured this way is data libraries' import from user directory function. Can you tell if my apache configuration is equivalent to yours?
Your configuration looks equivalent, yes, but I may have overlooked something, I have since forgotten some of the semantics of proxy balancers.
Does the import functionality in the data libraries work for you?
Import datasets from data libraries to history? Yes, this does for me. In what way does it break for you? Unauthenticated /api requests? It seems odd that data libraries would be the only affected thing here, since other things like the tool panel also make /api requests.

This configuration worked fine with release 15.10, but doesn't anymore since I upgraded. Between those two version the default data libraries interface changed. What used to be "beta" before is now the default and only available option.
Old data libraries were deprecated in favour of the new ones and removed in 16.01.

Best regards,
Sarah


----
Sarah Diehl
HPC System Administrator
 
UNIVERSITÉ DU LUXEMBOURG
 
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
-----
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
-----


From: Eric Rasche <esr@tamu.edu>
Organization: TAMU
Date: Monday 8 August 2016 15:09
To: Sarah DIEHL <sarah.diehl@uni.lu>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] External user auth and API

Hi Sarah,


On 08. aug. 2016 07:44, Sarah DIEHL wrote:
Dear all,

since no one replied so far to the main problem I had and it might have gotten lost in the conversation, I ask again: Does somebody know how to configure external user auth with apache such that API (from external, e.g. bioblend) and dataset import in the data libraries work? When I configure apache to require auth for everything, the API does not work. If I except the API from the apache auth, the dataset import does not work.

Our configuration looks like the following (just switching CAS for LDAP.)

    <Location "/galaxy/api/">
        Satisfy Any
        Allow from all
    </Location>

    <Location "/galaxy">
        AuthName "CAS"
        AuthType CAS
        Require valid-user
        RequestHeader set X-URL-SCHEME https
        XSendFile on
        XSendFilePath /
        RequestHeader set CAS-User "%{REMOTE_USER}s@tamu.edu"
    </Location>
    ProxyPass /galaxy uwsgi://127.0.0.1:4001/

I.e. we disable authentication on the /api route. On 16.01+ (I think it was patched then, but 16.04 is a safer bet) this will work correctly and your users will be able to use the API. On previous versions the /api route would fail for web users if exposed in this manner.

If I switch to the new galaxy-internal LDAP auth features, will that solve this problem?
Yes, this is an alternate solution.

Any hints are appreciated!

Best regards,
Sarah


----
Sarah Diehl
HPC System Administrator
 
UNIVERSITÉ DU LUXEMBOURG
 
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
-----
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
-----


From: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> on behalf of Sarah DIEHL <sarah.diehl@uni.lu>
Date: Monday 1 August 2016 13:06
To: Nicola Soranzo <nsoranzo@tiscali.it>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] Remote user auth and API

Hi Nicola,

thanks a lot for the help! Yes, it's a self-signed certificate, I didn't bother with letsencrypt yet ;-).

So now the error turned to
ConnectionError: GET: error 401: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>401 Authorization Required</title>\n</head><body>\n<h1>Authorization Required</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested.  Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn\'t understand how to supply\nthe credentials required.</p>\n</body></html>\n', 0 attempts left: None
which is what I expected, since apache now wants the authentication through LDAP.

So anybody know what the right settings are to get both the dataset import and the API working with external user auth over apache and LDAP?

Thanks,
Sarah

----
Sarah Diehl
HPC System Administrator
 
UNIVERSITÉ DU LUXEMBOURG
 
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
-----
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
-----


From: Nicola Soranzo <nicola.soranzo@gmail.com> on behalf of Nicola Soranzo <nsoranzo@tiscali.it>
Date: Monday 1 August 2016 12:58
To: Sarah DIEHL <sarah.diehl@uni.lu>, "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] Remote user auth and API

Hi Sarah!
I guess that your problem is with an untrusted certificate, you can get one for free at https://letsencrypt.org/

You can disable certificate verification in bioblend as in the example below:

import bioblend.galaxy
gi = bioblend.galaxy.GalaxyInstance(url=my_server, key=my_key)
gi.verify = False

Cheers,
Nicola

On 01/08/16 09:08, Sarah DIEHL wrote:
Dear all,

since the recent update to 16.04 I get the following error when trying to import a file from a user directory to a data library:

AssertionError: use_remote_user is set but HTTP_REMOTE_USER header was not provided

I use apache as a proxy and use an LDAP server for authentication. In order to get the API to work previously the apache had to be set to not check authentication for the requests to /api. In the logs I can see that the dataset import is an request to the API, so since the auth is not checked then, there is also no REMOTE_USER header set.

What is the recommended way to solve this issue with the current Galaxy version? I disabled the special settings for /api and the dataset import works now.

I tried to check the API with an old test script based on bioblend, but I now get the following error:

ConnectionError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645), 0 attempts left: None

Previously I could disable it with 

import requests
requests.packages.urllib3.disable_warnings()

but that doesn't seem to work anymore (switched to Python 3 now). Since bioblend wraps all the requests methods, I cannot apply any of the common solutions I found online (e.g. set verify=False).

Any help to solve these issues is highly appreciated :-).

Best regards,
Sarah



----
Sarah Diehl
HPC System Administrator
 
UNIVERSITÉ DU LUXEMBOURG
 
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE
Campus Belval | Biotech II
6, avenue du Swing
L-4371 Belvaux
T +352 46 66 44 5360
-----
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
-----



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
esr@tamu.edu

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
esr@tamu.edu