Hi Sarah,
On 08. aug. 2016 13:58, Sarah DIEHL wrote:
Hi Eric,
thanks for the hint regarding uWSGI. What doesn't work is importing files from the local disk (of the galaxy server) to the data library (see attached screenshot). Everything else seems to be fine, I haven't encountered any other issues. Maybe it's a bug just in that function? Here is the error:
10.184.132.10 - - [30/Jul/2016:18:09:27 +0200] "POST /api/libraries/datasets?encoded_folder_id=F7b46bd6d01de922f&source=userdir_file&path=160308_WTCHG_254732_201.bam&file_type=auto&dbkey=? HTTP/1.1" 500 - "https://galaxy-server.uni.lu/library/list" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0" Error - <type 'exceptions.AssertionError'>: use_remote_user is set but HTTP_REMOTE_USER header was not provided URL: https://galaxy-server.uni.lu/api/libraries/datasets?encoded_folder_id=F7b46b... File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 151 in __call__ app_iter = self.application(environ, sr_checker) File '/mnt/gaiagpfs/projects/galaxy/internal/.venv/local/lib/python2.7/site-packages/paste/recursive.py', line 85 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py', line 76 in __call__ return self.app( environ, start_response ) File '/mnt/gaiagpfs/projects/galaxy/internal/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py', line 640 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 131 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 158 in handle_request trans = self.transaction_factory( environ ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 68 in <lambda> self.set_transaction_factory( lambda e: self.transaction_chooser( e, galaxy_app, session_cookie ) ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 99 in transaction_chooser return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 198 in __init__ self.error_message = self._authenticate_api( session_cookie ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 380 in _authenticate_api self._ensure_valid_session( session_cookie ) File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 432 in _ensure_valid_session "use_remote_user is set but %s header was not provided" % self.app.config.remote_user_header AssertionError: use_remote_user is set but HTTP_REMOTE_USER header was not provided
I'm afraid I do not have any help for this. Hopefully another galaxy person can chime in here and help debug this.
Best regards, Sarah
Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu mailto:sarah.diehl@uni.lu http://lcsb.uni.lu
https://urldefense.proofpoint.com/v2/url?u=http-3A__lcsb.uni.lu_&d=CwMF-g&c=ODFT-G5SujMiGrKuoJJjVg&r=p9uZby14OqW9zcjBSjiDKw&m=pwYDgdauhpWnUSnnqVAB6Xyn2OzpGsIMkWdC8fyNRu0&s=Ll-0NRWHk-0Kb159GLi8cgxakoFwK4BQyp39IRY0S48&e=
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
From: Eric Rasche <esr@tamu.edu mailto:esr@tamu.edu> Organization: TAMU Date: Monday 8 August 2016 15:43 To: Sarah DIEHL <sarah.diehl@uni.lu mailto:sarah.diehl@uni.lu>, "galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] External user auth and API
On 08. aug. 2016 13:25, Sarah DIEHL wrote:
Hi Eric,
thanks a lot for the info and help! I'm running version 16.04 and my apache conf is a bit different because I balance over multiple galaxy web servers:
Just as an aside, if you are still using this configuration method, you might consider switching to uWSGI https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.galaxyproject.org_Admin_Config_Performance_Scaling&d=CwMF-g&c=ODFT-G5SujMiGrKuoJJjVg&r=p9uZby14OqW9zcjBSjiDKw&m=pwYDgdauhpWnUSnnqVAB6Xyn2OzpGsIMkWdC8fyNRu0&s=FcalyPc9I-0TF6rCq1c3yxRT6-Rrzmf7mNE3GVfCNBI&e=, it will drastically simplify your configuration and should be just as performant (if not more).
# API requests get routed through balancer with no authentication RewriteCond %{QUERY_STRING} key= RewriteRule ^/api/(.*) balancer://galaxy-noauth/api/$1 [P] # Regular requests get routed through balancer with LDAPauthentication RewriteRule ^(.*) balancer://galaxy$1 [P]
<Proxy balancer://galaxy/*> BalancerMember http://localhost:8080 BalancerMember http://localhost:8081 BalancerMember http://localhost:8082 BalancerMember http://localhost:8083RequestHeader set X-URL-SCHEME https
AuthType Basic AuthBasicProvider ldap AuthName "UL HPC Platform Authorized Users Only" AuthLDAPBindAuthoritative off AuthLDAPURL "ldap://…" Require valid-user RequestHeader set REMOTE_USER %{AUTHENTICATE_uid}e XSendFile on XSendFilePath / SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary SetEnvIfNoCase Request_URI /history/export_archive no-gzipdont-vary </Proxy>
<Proxy balancer://galaxy-noauth/*> BalancerMember http://localhost:8080 BalancerMember http://localhost:8081 BalancerMember http://localhost:8082 BalancerMember http://localhost:8083RequestHeader set X-URL-SCHEME https
Satisfy any XSendFile on XSendFilePath / SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary SetEnvIfNoCase Request_URI /history/export_archive no-gzipdont-vary </Proxy>
What doesn't work when configured this way is data libraries' import from user directory function. Can you tell if my apache configuration is equivalent to yours?
Your configuration /looks/ equivalent, yes, but I may have overlooked something, I have since forgotten some of the semantics of proxy balancers.
Does the import functionality in the data libraries work for you?
Import datasets from data libraries to history? Yes, this does for me. In what way does it break for you? Unauthenticated /api requests? It seems odd that data libraries would be the only affected thing here, since other things like the tool panel also make /api requests.
This configuration worked fine with release 15.10, but doesn't anymore since I upgraded. Between those two version the default data libraries interface changed. What used to be "beta" before is now the default and only available option.
Old data libraries were deprecated https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.galaxyproject.org_en_master_releases_15.10-5Fannounce.html-23deprecation-2Dnotices&d=CwMF-g&c=ODFT-G5SujMiGrKuoJJjVg&r=p9uZby14OqW9zcjBSjiDKw&m=pwYDgdauhpWnUSnnqVAB6Xyn2OzpGsIMkWdC8fyNRu0&s=p6Y66XyId9RfOGHSilb-eCggUDdkXaloeEyrXZBLQbM&e= in favour of the new ones and removed in 16.01.
Best regards, Sarah
Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360 sarah.diehl@uni.lu mailto:sarah.diehl@uni.lu http://lcsb.uni.lu
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
From: Eric Rasche esr@tamu.edu Organization: TAMU Date: Monday 8 August 2016 15:09 To: Sarah DIEHL sarah.diehl@uni.lu, "galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] External user auth and API
Hi Sarah,
On 08. aug. 2016 07:44, Sarah DIEHL wrote:
Dear all,
since no one replied so far to the main problem I had and it might have gotten lost in the conversation, I ask again: Does somebody know how to configure external user auth with apache such that API (from external, e.g. bioblend) and dataset import in the data libraries work? When I configure apache to require auth for everything, the API does not work. If I except the API from the apache auth, the dataset import does not work.
Our configuration looks like the following (just switching CAS for LDAP.)
<Location "/galaxy/api/"> Satisfy Any Allow from all </Location> <Location "/galaxy"> AuthName "CAS" AuthType CAS Require valid-user RequestHeader set X-URL-SCHEME https XSendFile on XSendFilePath / RequestHeader set CAS-User "%{REMOTE_USER}s@tamu.edu" </Location> ProxyPass /galaxy uwsgi://127.0.0.1:4001/I.e. we disable authentication on the /api route. On 16.01+ (I think it was patched then, but 16.04 is a safer bet) this will work correctly and your users will be able to use the API. On previous versions the /api route would fail for web users if exposed in this manner.
If I switch to the new galaxy-internal LDAP auth features, will that solve this problem?
Yes, this is an alternate solution.
Any hints are appreciated!
Best regards, Sarah
Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360
mailto:sarah.diehl@uni.lusarah.diehl@uni.lu http://lcsb.uni.lu
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
From: galaxy-dev galaxy-dev-bounces@lists.galaxyproject.org on behalf of Sarah DIEHL <sarah.diehl@uni.lu mailto:sarah.diehl@uni.lu> Date: Monday 1 August 2016 13:06 To: Nicola Soranzo nsoranzo@tiscali.it, "galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] Remote user auth and API
Hi Nicola,
thanks a lot for the help! Yes, it's a self-signed certificate, I didn't bother with letsencrypt yet ;-).
So now the error turned to ConnectionError: GET: error 401: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>401 Authorization Required</title>\n</head><body>\n<h1>Authorization Required</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested. Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn't understand how to supply\nthe credentials required.</p>\n</body></html>\n', 0 attempts left: None which is what I expected, since apache now wants the authentication through LDAP.
So anybody know what the right settings are to get both the dataset import and the API working with external user auth over apache and LDAP?
Thanks, Sarah
Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360
mailto:sarah.diehl@uni.lusarah.diehl@uni.lu http://lcsb.uni.lu
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
From: Nicola Soranzo <nicola.soranzo@gmail.com mailto:nicola.soranzo@gmail.com> on behalf of Nicola Soranzo nsoranzo@tiscali.it Date: Monday 1 August 2016 12:58 To: Sarah DIEHL <sarah.diehl@uni.lu mailto:sarah.diehl@uni.lu>, "galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.galaxyproject.org mailto:galaxy-dev@lists.galaxyproject.org> Subject: Re: [galaxy-dev] Remote user auth and API
Hi Sarah! I guess that your problem is with an untrusted certificate, you can get one for free at https://letsencrypt.org/
You can disable certificate verification in bioblend as in the example below:
import bioblend.galaxy gi = bioblend.galaxy.GalaxyInstance(url=my_server, key=my_key) gi.verify = False
Cheers, Nicola
On 01/08/16 09:08, Sarah DIEHL wrote:
Dear all,
since the recent update to 16.04 I get the following error when trying to import a file from a user directory to a data library:
AssertionError: use_remote_user is set but HTTP_REMOTE_USER header was not provided
I use apache as a proxy and use an LDAP server for authentication. In order to get the API to work previously the apache had to be set to not check authentication for the requests to /api. In the logs I can see that the dataset import is an request to the API, so since the auth is not checked then, there is also no REMOTE_USER header set.
What is the recommended way to solve this issue with the current Galaxy version? I disabled the special settings for /api and the dataset import works now.
I tried to check the API with an old test script based on bioblend, but I now get the following error:
ConnectionError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645), 0 attempts left: None
Previously I could disable it with
import requests requests.packages.urllib3.disable_warnings()
but that doesn't seem to work anymore (switched to Python 3 now). Since bioblend wraps all the requests methods, I cannot apply any of the common solutions I found online (e.g. set verify=False).
Any help to solve these issues is highly appreciated :-).
Best regards, Sarah
Sarah Diehl HPC System Administrator
UNIVERSITÉ DU LUXEMBOURG
LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE Campus Belval | Biotech II 6, avenue du Swing L-4371 Belvaux T +352 46 66 44 5360
mailto:sarah.diehl@uni.lusarah.diehl@uni.lu http://lcsb.uni.lu
This message is confidential and may contain privileged information. It is intended for the named recipient only. If you receive it in error please notify me and permanently delete the original message and any copies.
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
-- Eric Rasche Programmer II
Center for Phage Technology Rm 312A, Biochemistry & Biophysics Texas A&M University College Station, TX 77843 esr@tamu.edu mailto:esr@tamu.edu
-- Eric Rasche Programmer II
Center for Phage Technology Rm 312A, Biochemistry & Biophysics Texas A&M University College Station, TX 77843 esr@tamu.edu mailto:esr@tamu.edu