November 2018 - galaxy-dev - lists.galaxyproject.org

Nginx+Shibboleth+UCSC
by Martin Demko 19 Apr '19

19 Apr '19

Dear list, I have been trying to manage an access to UCSC genome browser from our Galaxy instance which uses Nginx as a proxy with external authentication via Shibboleth and IdP service. Not even the configuration of nginx+shibboleth was quite complicated to achieve (but I succeeded after a week or so) but now during a testing phase, we have found out that a cooperation with external UCSC servers is forbidden due to our auth. configuration. I have found a way how to do that but only for Apache (https://docs.galaxyproject.org/en/master/admin/special_topics/apache.html#d…) so I tried to do something similar in Nginx but only with IP addresses as Nginx doesn't support using of named domains and defined a location: location /display_as { satisfy any; deny all; allow 128.114.119.131; allow 128.114.119.132; allow 128.114.119.133; allow 128.114.119.134; allow 128.114.119.135; allow 128.114.119.136; } Used IP addresses should be the right ones for UCSC servers according to few sources such as (https://genome.soe.ucsc.narkive.com/sll2JSk2/ucsc-ip-address) hgw1.cse.ucsc.edu hgw2.cse.ucsc.edu hgw3.cse.ucsc.edu hgw4.cse.ucsc.edu hgw5.cse.ucsc.edu hgw6.cse.ucsc.edu I tried to change the location also to "/display_application" but didn't help (actually, I don't know why in the documentation there is used as the location that "/display_as" as I have never encountered such location in Galaxy which is my first question). In galaxy.ini I set: display_servers = hgw1.cse.ucsc.edu,hgw2.cse.ucsc.edu,hgw3.cse.ucsc.edu,hgw4.cse.ucsc.edu,hgw5.cse.ucsc.edu,hgw6.cse.ucsc.edu,hgw7.cse.ucsc.edu,hgw8.cse.ucsc.edu,lowepub.cse.ucsc.edu,128.114.119.131,128.114.119.132,128.114.119.133,128.114.119.134,128.114.119.135,128.114.119.136 Just for case I used both, the domains and also IP addresses. Also I have tried to use only the domains or only the addresses but nothing helped. Then I found out after reading of the following (https://www.switch.ch/aai/guides/sp/access-rules/) that the main obstacle should be actually shibboleth so I defined in /etc/shibboleth/shibboleth2.xml an unrestricted path: <RequestMapper type="XML"> <RequestMap> <Host name="our.galaxy.something" authType="shibboleth" requireSession="true" redirectToSSL="443" > <Path name="display_application" requireSession="false" redirectToSSL="443" />  </Host> </RequestMap> </RequestMapper> Again, I have tried to use both locations "display_as" and "display_application", but it haven't solved my problem entirely. I moved forward I suppose because now the requests should be going around the authentication but UCSC stil warns me that: Expected 200 https://our.galaxy.something/display_application/e1304269a2f56a52/ucsc_bigw…: 403 Forbidden Before that it was a different message containing the address of our idp server so I believe I'am on the right way here. The presence of attribute 'authType="shibboleth"' seams to be useless as 'requireSession="false"' is used. Then I tried to teach Nginx to use also the domain names with the using of 3rd party nginx-http-rdns module from (https://github.com/flant/nginx-http-rdns) so my nginx.conf looked like: ....only the part of nginx.conf.... ############## Shibboleth authentication conf ################# # FastCGI authorizer for Shibboleth Auth Request module location = /shibauthorizer { internal; include fastcgi_params; fastcgi_pass unix:/var/run/shibboleth/shibauthorizer.sock; } # FastCGI responder for SSO location /Shibboleth.sso { include fastcgi_params; fastcgi_pass unix:/var/run/shibboleth/shibresponder.sock; } ############### Entry point for Galaxy ############### # Location secured by Shibboleth location / { shib_request /shibauthorizer; more_clear_input_headers 'Variable-*' 'Shib-*' 'Remote-User' 'REMOTE_USER' 'Auth-Type' 'AUTH_TYPE' 'GX_SECRET'; include uwsgi_params; shib_request_set $shib_id $upstream_http_variable_eppn; uwsgi_param HTTP_REMOTE_USER $shib_id; uwsgi_param HTTP_GX_SECRET $our_secret; uwsgi_pass 127.0.0.1:4001; # resolver $correct_resolver_acc_to_resolv.conf; # rdns on; # satisfy any; # rdns_allow genome\.ucsc\.edu; # rdns_allow hgw1\.cse\.ucsc\.edu; # rdns_allow hgw2\.cse\.ucsc\.edu; # rdns_allow hgw3\.cse\.ucsc\.edu; # rdns_allow hgw4\.cse\.ucsc\.edu; # rdns_allow hgw5\.cse\.ucsc\.edu; # rdns_allow hgw6\.cse\.ucsc\.edu; # rdns_allow hgw7\.cse\.ucsc\.edu; # rdns_allow hgw8\.cse\.ucsc\.edu; } # location /display_as { # resolver $correct_resolver_acc_to_resolv.conf; # rdns on; # satisfy any; # rdns_allow genome\.ucsc\.edu; # rdns_allow hgw1\.cse\.ucsc\.edu; # rdns_allow hgw2\.cse\.ucsc\.edu; # rdns_allow hgw3\.cse\.ucsc\.edu; # rdns_allow hgw4\.cse\.ucsc\.edu; # rdns_allow hgw5\.cse\.ucsc\.edu; # rdns_allow hgw6\.cse\.ucsc\.edu; # rdns_allow hgw7\.cse\.ucsc\.edu; # rdns_allow hgw8\.cse\.ucsc\.edu; # } # location /display_application { # resolver $correct_resolver_acc_to_resolv.conf; # rdns on; # satisfy any; # rdns_allow genome\.ucsc\.edu; # rdns_allow hgw1\.cse\.ucsc\.edu; # rdns_allow hgw2\.cse\.ucsc\.edu; # rdns_allow hgw3\.cse\.ucsc\.edu; # rdns_allow hgw4\.cse\.ucsc\.edu; # rdns_allow hgw5\.cse\.ucsc\.edu; # rdns_allow hgw6\.cse\.ucsc\.edu; # rdns_allow hgw7\.cse\.ucsc\.edu; # rdns_allow hgw8\.cse\.ucsc\.edu; # } ....END of the part of nginx.conf.... Everything commented out, I tried to use together or one at the time but nothing helped. But there are differences: 1) when uncommented only location /display_application - I will get just a blank page with: 404 Not Found. 2) when uncommented only location /display_as - nothing changes. 3) when uncommented part inside the location "/" - I will get a new message: Error unexpected end of input reading http header on https://our.galaxy.something/display_application/e1304269a2f56a52/ucsc_bigw… And now I'm out of ideas. I would really appreciate any help. PS: I stil don't get why in the documentation for Apache configuration (https://docs.galaxyproject.org/en/master/admin/special_topics/apache.html#d…) there is used the /display_as location. Can somebody explain, please? Thank you in advance, Martin Demko

2 1

Galaxy Administrator Training, Jan 28 - Feb 1, Penn State University
by Dave Clements 20 Dec '18

20 Dec '18

Hello all, We are pleased to announce that* the 2019 Galaxy Admin Training <https://galaxyproject.org/events/2019-admin-training/> will be offered January 28 through February 1 at Penn State University* (where the Galaxy Project got started over ten years ago). The workshop offers a 2 day introductory session <https://github.com/galaxyproject/dagobah-training#basic-sessions> followed by a 3 day advanced topics session <https://github.com/galaxyproject/dagobah-training#advanced-sessions>. Participants can register for one or both sessions. This workshop will cover what you need to know to set up your own high-performance and multi-user production Galaxy instance. Sessions will be *intensive and hands-on*, and taught by experienced instructors from the Galaxy Community. Participants will learn how to install, configure, customize, and extend their own Galaxy servers. Topics include tool configuration, authentication and user management, using heterogeneous storage and compute services, and many other topics that will enable you to get your own Galaxy server up and running, performing well, and used by your community. Registration <https://galaxyproject.org/events/2019-admin-training/#registration> is now open and starts at $40 / day for participants from non-profits and academia. Early registration ends October 31. However, the 2016 and 2018 admin training were both full, so you are strongly encouraged to *register now*. *The workshop does have prerequisites <https://galaxyproject.org/events/2019-admin-training/#prerequisites>. Please read them before you register.* Cheers, Dave C. *About Galaxy * Galaxy is an open web based platform for biomedical data integration and analysis. It is deployed at large and small organizations around the world and used in a broad range of biomedical research domains, and across the tree of life. -- https://galaxyproject.org/ https://usegalaxy.org/

1 2

XMLs not reloading in all processes (UWSGI)?
by D K 03 Dec '18

03 Dec '18

I'm running version 18.05 and have noticed that when updating the tool_conf.xml that changes don't always get loaded in all processes (e.g. if I hit reload sometimes my changes show up and sometimes they do not). Also, if I try to run a new tool I've added sometimes this will fail presumably because that process doesn't have the update. The only fix for this seems to be a galaxy restart. I've turned watchdog on and am using the UWSGI + mules strategy. Any suggestions?

2 2

installation is not finishing
by Jochen Bick 30 Nov '18

30 Nov '18

Hi, I wanted to install two tools but my installation is stuck in Installing tool dependencies. So I found out that the xmls are cloned but the dependencies are not getting installed. It looks like anything related to python is not getting installed? anyone needs some log file, please tell me. Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

3 7

number of elements in collections and multiple-true-params
by Matthias Bernt 30 Nov '18

30 Nov '18

Dear list, in a tool I want to process a set of (bam) files. For the convenience of the user I thought that it would be nice to allow collections as well as manually selected files (in a param with multiple=true). Like so: ``` <conditional name="input_type"> <param name="input_type_selector" type="select" label="BAM files"> <option value="list">data set list</option> <option value="manual">data sets</option> </param> <when value="manual"> <param name="input_bam" format="bam" type="data" multiple="true" optional="false" label="BAM files" /> </when> <when value="list"> <param name="input_bam" format="bam" type="data_collection" collection_type="list" label="BAM files" /> </when> </conditional> ``` Is there a way to get the number of data sets in both cases (in the collection or selection) in a consistent way? Currently the only way I found is: ``` #if $input_type.input_type_selector == 'manual': #set count = len($input_type.input_bam) #else #set count = len($input_type.input_bam.keys()) #end if ``` Appart from this looping works like a charm: ``` #for $bam in $input_type.input_bam: OPERATIONS... #end for ``` Cheers, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch -------------------------------------------

2 2

Multifactor RNA-Seq Analysis and interaction terms
by Lance Parsons 13 Nov '18

13 Nov '18

One of the labs I work with regularly uses Galaxy for their RNA-Seq analysis. They have been doing very simple analyses up to this point, using DESeq2. However, they are moving toward more complicated experimental designs that will include additional factors, some of which (genotype for example) will be interaction terms <http://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DES…>. I have a few questions: 1. I know Marius van den Beek <https://github.com/mvdbeek> added group tags <https://github.com/galaxyproject/galaxy/pull/5457> in 18.09, which seem like they are at least one way to use multiple factors along with collections. Is there any primer/tutorial on using these? It’s not quite clear to me just yet. 2. Does anyone have any thoughts or suggestions as to how one could add interaction terms (/e.g./ |condition:genotype|) to the DESeq2 wrapper? Does this functionality exist in a different Galaxy tool? Thanks in advance. -- Lance Parsons - Scientific Programmer Carl C. Icahn Laboratory - Room 136 Lewis-Sigler Institute for Integrative Genomics Princeton University

1 0

Adding the Ilifu cloud to launch.usegalaxy.org
by Peter van Heusden 13 Nov '18

13 Nov '18

Hi there While I will continue to work on getting a Cloudlaunch server running locally, could the Ilifu cloud (http://www.ilifu.ac.za/) be added as an option on https://launch.usegalaxy.org/catalog ? What details would be required? And then - what are the requirements for adding appliances to the appliance catalogue? Thanks, Peter

1 0

Re: [galaxy-dev] Running Cloudlaunch against a new OpenStack cloud
by Peter van Heusden 13 Nov '18

13 Nov '18

Hi there I've got a bit further, but now I've got a problem - how do you access the admin panel when you have the complete site set up? 1) If Debug = False (as it is for a production setup), Django doesn't load static files (e.g. CSS). 2) When going via the nginx proxy, I can't figure out what URL will direct me to the admin page. Peter P.S. I'm doing some work on the ansible to get this all set up - I'll do a PR once everything seems to be working. On Fri, 19 Oct 2018 at 01:25, Alexandru Mahmoud <amahmou4(a)jhu.edu> wrote: > Hello! > > I am new to the team, so this will be a brief response to only one of your > questions while waiting for someone else to get to the email (and I am not > replying all so that people don’t dismiss it as answered): > > For 3 specifically, you can access the admin panel at http://127.0.0.1(or > localhost):8000/admin, where you should be able to login with the user > created with the ‘createsuperuser’ command. Under “Appliances” or > “Applications” (depending on the version of cloudlaunch) you’ll be able to > see and change/add configurations for apps. > If you want to see in more detail how the appliances are being set up and > how the launch config is used in code, look for the "Backend component > name", then find the corresponding backend plugin in the cloudlaunch repo. > I hope that is a helpful start, and feel free to email me if nobody > responds in a reasonable amount of time to your other questions, as I can > get more answers in person and relay them to you. > > Best, > Alex Mahmoud > > > On Oct 18, 2018, at 3:45 PM, Peter van Heusden <pvh(a)sanbi.ac.za> wrote: > > Hi there > > I'd like to set up cloudlaunch to be able to set up on-demand Galaxy > servers and associated clusters on the new Ilifu OpenStack cloud. From > looking around the repositories it looks like Cloudlaunch is at least 2 > components: > > 1. The cloudlaunch server > 2. The UI - either cloudlaunch-ui or cloudlaunch-cli > > I'm trying to get cloudlaunch server installed on a Ubuntu 18.04 VM. The > only requirement seems to be python3-pip. For postgresql support I > installed the python3-psycopg2 module, the postgresql-10 server and create > a user and db (both named cloud). Then I installed the server with `pip3 > install cloudlaunch-server` and ran commands: > > cloudlaunch-server django migrate > cloudlaunch-server django createsuperuser > cloudlaunch-server django runserver > > The next step (https://github.com/galaxyproject/cloudlaunch) refers to > redis-server. Installing this package seemed to satisfy that requirement, > running it manually didn't work so I just ran the celery command as > illustrated. And finally relaunched the django server. > > Then nodejs and npm were installed, the cloudlaunch-ui cloned from github, > the various npm package specified installed and finally npm start brings up > the cloudlaunch user interface. > > So far so good but: > > 1. Is there more extensive documentation anywhere? Specifically > documentation about running this in production. > > 2. Is there a docker container that can be used instead of all this manual > installation? > > 3. How are appliances configured and added to the Appliance Catalogue? > > Thanks! > Peter > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ > > >

1 0

Re: [galaxy-dev] Search box in tools menu / uncomplete wrappers list
by Matthias Bernt 12 Nov '18

12 Nov '18

Hi, I observed similar Problems on my Galaxy. But I think not all cases can be explained by Martins answer. For instance, if I search for "hisat" I don't get any result, but HISAT2 is definitely installed. Cheers, Matthias > Message: 1 > Date: Fri, 9 Nov 2018 15:54:16 +0000 > From: Sarah Maman <sarah.maman(a)inra.fr> > To: "galaxy-dev(a)lists.galaxyproject.org" > <galaxy-dev(a)lists.galaxyproject.org> > Subject: [galaxy-dev] Search box in tools menu / uncomplete wrappers > list > Message-ID: <1541778855881.43440(a)inra.fr> > Content-Type: text/plain; charset="iso-8859-1" > > > Hello, > > > When I'm searching a keyword in the « search » box in left « tools » menu, results are sometimes false because the wrappers' list generated by the query thanks to a keyword is not full, some tools are missing. In fact, some wrappers are not listed even if the keyword is in the tool name and/or the tool description. > > Could you please help us to understand why ? > > > Thanks, > > Sarah Maman > > > > > _____________________________ > Sarah Maman > > Bioinformatique/Bioinformatics > > INRA | SIGENAE | GenPhySE > > Centre Inra Occitanie-Toulouse > Tél. : +33(0)5.61.28.57.08 > > _____________________________ > > Chemin de Borde Rouge CS 52627 > 31326 Castanet Tolosan cedex > > www.toulouse.inra.fr<http://www.toulouse.inra.fr/> > > Suivez-nous sur Twitter ! >

2 1

Bug in data manager for the STAR aligner indices
by Previti 12 Nov '18

12 Nov '18

Dear all, I've been wanting to point this out for a while, but the latest update to the STAR data manager didn't fix it either (or maybe I messed up again). The .loc file indicating which genomes have been installed/indices have been made for STAR uses the last column to indicate if there is a gene model/annotation that it should use (0 - for no, 1 - for yes). This seems to be empty...there is a tab at the end of the row, but no number. It works if you manually but in 1 or 0, but not that's not very convenient. Please let me know if I'm missing something... Cheers, Christopher -- *Dr. Christopher Previti* Genomics and Proteomics Core Facility High Throughput Sequencing (W190) Bioinformatician German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany Room: B2.102 (INF580/TP3) Phone: +49 6221 42-4661 christopher.previti(a)dkfz.de <http://www.dkfz.de/> www.dkfz.de <http://www.dkfz.de/> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta VAT-ID No.: DE143293537 Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist. Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung. Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.

1 0