galaxy-dev February 2019

galaxy-dev@lists.galaxyproject.org

9 participants
11 discussions

URL prefix doesn't work properly in Galaxy v18.09
by Ping Luo 20 May '19

20 May '19

I configured URL prefix in galaxy 18.09. It has problems of resolving some of the links, including: workflow visualize -> create visualization shared data -> workflows shared data -> visualizations I will use workflow as an example to illustrate what happens. When the mouse cursor is put on workflow, the URL shown in at the left bottom of the browser shows https://xxx.xxx.tamu.edu/galaxy/workflows/list But after I clicked 'workflow', the real URL shown in the address bar is actually https://xxx.xxxx.tamu.edu/galaxy/galaxy//workflows/list?__identifer=sk0xsyw… Of course, there is no such a page and the browser shows a blank page. If I manually remove one galaxy in the URL, the page is correctly shown. Beside the four mentioned above, other links work fine. If galaxy is configured without prefix, then there is no problem. I also tested v18.05. With all the configuration being the same, this version doesn't have the problem with prefix like v18.09 does. So I think this is a bug in v18.09 with URL prefix. Ping

2 3

Nginx+Shibboleth+UCSC
by Martin Demko 18 Apr '19

18 Apr '19

Dear list, I have been trying to manage an access to UCSC genome browser from our Galaxy instance which uses Nginx as a proxy with external authentication via Shibboleth and IdP service. Not even the configuration of nginx+shibboleth was quite complicated to achieve (but I succeeded after a week or so) but now during a testing phase, we have found out that a cooperation with external UCSC servers is forbidden due to our auth. configuration. I have found a way how to do that but only for Apache (https://docs.galaxyproject.org/en/master/admin/special_topics/apache.html#d…) so I tried to do something similar in Nginx but only with IP addresses as Nginx doesn't support using of named domains and defined a location: location /display_as { satisfy any; deny all; allow 128.114.119.131; allow 128.114.119.132; allow 128.114.119.133; allow 128.114.119.134; allow 128.114.119.135; allow 128.114.119.136; } Used IP addresses should be the right ones for UCSC servers according to few sources such as (https://genome.soe.ucsc.narkive.com/sll2JSk2/ucsc-ip-address): hgw1.cse.ucsc.edu hgw2.cse.ucsc.edu hgw3.cse.ucsc.edu hgw4.cse.ucsc.edu hgw5.cse.ucsc.edu hgw6.cse.ucsc.edu I tried to change the location also to "/display_application" but didn't help (actually, I don't know why in the documentation there is used as the location that "/display_as" as I have never encountered such location in Galaxy which is my first question). In galaxy.ini I set: display_servers = hgw1.cse.ucsc.edu,hgw2.cse.ucsc.edu,hgw3.cse.ucsc.edu,hgw4.cse.ucsc.edu,hgw5.cse.ucsc.edu,hgw6.cse.ucsc.edu,hgw7.cse.ucsc.edu,hgw8.cse.ucsc.edu,lowepub.cse.ucsc.edu,128.114.119.131,128.114.119.132,128.114.119.133,128.114.119.134,128.114.119.135,128.114.119.136 Just for case I used both, the domains and also IP addresses. Also I have tried to use only the domains or only the addresses but nothing helped. Then I found out after reading of the following (https://www.switch.ch/aai/guides/sp/access-rules/) that the main obstacle should be actually shibboleth so I defined in /etc/shibboleth/shibboleth2.xml an unrestricted path: <RequestMapper type="XML"> <RequestMap> <Host name="our.galaxy.something" authType="shibboleth" requireSession="true" redirectToSSL="443" > <Path name="display_application" requireSession="false" redirectToSSL="443" />  </Host> </RequestMap> </RequestMapper> Again, I have tried to use both locations "display_as" and "display_application", but it haven't solved my problem entirely. I moved forward I suppose because now the requests should be going around the authentication but UCSC stil warns me that: Expected 200 https://our.galaxy.something/display_application/e1304269a2f56a52/ucsc_bigw…: 403 Forbidden Before that it was a different message containing the address of our idp server so I believe I'am on the right way here. The presence of attribute 'authType="shibboleth"' seams to be useless as 'requireSession="false"' is used. Then I tried to teach Nginx to use also the domain names with the using of 3rd party nginx-http-rdns module from (https://github.com/flant/nginx-http-rdns), so my nginx.conf looked like: ....only the part of nginx.conf.... ############## Shibboleth authentication conf ################# # FastCGI authorizer for Shibboleth Auth Request module location = /shibauthorizer { internal; include fastcgi_params; fastcgi_pass unix:/var/run/shibboleth/shibauthorizer.sock; } # FastCGI responder for SSO location /Shibboleth.sso { include fastcgi_params; fastcgi_pass unix:/var/run/shibboleth/shibresponder.sock; } ############### Entry point for Galaxy ############### # Location secured by Shibboleth location / { shib_request /shibauthorizer; more_clear_input_headers 'Variable-*' 'Shib-*' 'Remote-User' 'REMOTE_USER' 'Auth-Type' 'AUTH_TYPE' 'GX_SECRET'; include uwsgi_params; shib_request_set $shib_id $upstream_http_variable_eppn; uwsgi_param HTTP_REMOTE_USER $shib_id; uwsgi_param HTTP_GX_SECRET $our_secret; uwsgi_pass 127.0.0.1:4001; # resolver $correct_resolver_acc_to_resolv.conf; # rdns on; # satisfy any; # rdns_allow genome\.ucsc\.edu; # rdns_allow hgw1\.cse\.ucsc\.edu; # rdns_allow hgw2\.cse\.ucsc\.edu; # rdns_allow hgw3\.cse\.ucsc\.edu; # rdns_allow hgw4\.cse\.ucsc\.edu; # rdns_allow hgw5\.cse\.ucsc\.edu; # rdns_allow hgw6\.cse\.ucsc\.edu; # rdns_allow hgw7\.cse\.ucsc\.edu; # rdns_allow hgw8\.cse\.ucsc\.edu; } # location /display_as { # resolver $correct_resolver_acc_to_resolv.conf; # rdns on; # satisfy any; # rdns_allow genome\.ucsc\.edu; # rdns_allow hgw1\.cse\.ucsc\.edu; # rdns_allow hgw2\.cse\.ucsc\.edu; # rdns_allow hgw3\.cse\.ucsc\.edu; # rdns_allow hgw4\.cse\.ucsc\.edu; # rdns_allow hgw5\.cse\.ucsc\.edu; # rdns_allow hgw6\.cse\.ucsc\.edu; # rdns_allow hgw7\.cse\.ucsc\.edu; # rdns_allow hgw8\.cse\.ucsc\.edu; # } # location /display_application { # resolver $correct_resolver_acc_to_resolv.conf; # rdns on; # satisfy any; # rdns_allow genome\.ucsc\.edu; # rdns_allow hgw1\.cse\.ucsc\.edu; # rdns_allow hgw2\.cse\.ucsc\.edu; # rdns_allow hgw3\.cse\.ucsc\.edu; # rdns_allow hgw4\.cse\.ucsc\.edu; # rdns_allow hgw5\.cse\.ucsc\.edu; # rdns_allow hgw6\.cse\.ucsc\.edu; # rdns_allow hgw7\.cse\.ucsc\.edu; # rdns_allow hgw8\.cse\.ucsc\.edu; # } ....END of the part of nginx.conf.... Everything commented out, I tried to use together or one at the time but nothing helped. But there are differences: 1) when uncommented only location /display_application - I will get just a blank page with: 404 Not Found. 2) when uncommented only location /display_as - nothing changes. 3) when uncommented part inside the location "/" - I will get a new message: Error unexpected end of input reading http header on https://our.galaxy.something/display_application/e1304269a2f56a52/ucsc_bigw… And now I'm out of ideas. I would really appreciate any help. PS: I stil don't get why in the documentation for Apache configuration (https://docs.galaxyproject.org/en/master/admin/special_topics/apache.html#d…) there is used the /display_as location. Can somebody explain, please? Thank you in advance, Martin Demko

2 1

Galaxy Release 19.01
by Dave Clements 28 Feb '19

28 Feb '19

We are pleased to announce the *release of Galaxy 19.01 <https://docs.galaxyproject.org/en/release_19.01/releases/19.01_announce.html>*. A few release highlights are: <https://galaxyproject.org/news/2019-1-galaxy-release/#revised-ui-style> Revised UI Style The Galaxy stylesheet has been substantially reworked resulting in the largest visual refresh to Galaxy in years. <https://galaxyproject.org/news/2019-1-galaxy-release/#more-colorful-tags> More Colorful Tags Use consistently hashed randomized colors for history dataset name tags to increase visual distinction between tags, making it easier to track data flow in complex histories. Thanks to @erasche <https://github.com/erasche>. <https://galaxyproject.org/news/2019-1-galaxy-release/#extensive-workflow-en…> Extensive Workflow Enhancements (continued) Galaxy 18.09 featured a bunch of important and diverse fixes and enhancements to workflows on both the frontend and backend. That progress continues in release 19.01. The workflow editor now features explicit step parameter nodes for non-file data (integers, strings, booleans, etc) and the ability to connect these to all tool inputs - not just files. The YAML-based Galaxy Format 2 workflow language is now included in Galaxy as a beta option and allows import and export of human readable workflows as well as extensions to allow Galaxy to serve as a file editor for such workflows. The workflow editor features more helpful labels on inputs and outputs and important fixes including logic for connecting collections and dealing with missing tools. The workflow run form now deals with default values more correctly. <https://galaxyproject.org/news/2019-1-galaxy-release/#enhanced-support-for-…> Enhanced Support for Singularity Singularity <https://singularity.lbl.gov/> container support in Galaxy has been brought to parity with Docker container support. Specifically, Galaxy can now be configured to fetch Biocontainers <https://biocontainers.pro/> automatically for enabled job destinations. Moreover, explicitly annotated Docker containers (either in tools or job destinations) can be used with Singularity. Thanks to @mvdbeek <https://github.com/mvdbeek> Please see the full release notes <https://docs.galaxyproject.org/en/release_19.01/releases/19.01_announce.html> for more information, including how to upgrade today! *Thanks for using Galaxy!* -- https://galaxyproject.org/ https://usegalaxy.org/

1 0

A question about Galaxy
by Abdullah El Kurdi 21 Feb '19

21 Feb '19

Dear all, I would like to ask if is there any way that I could copy my galaxy docker instance from a server to another including the "export" directory, without having to re-install all the tools again. Thanks in advance.

2 1

[GSoC 2019] [Open Genome Informatics] Google Summer of Code 2019 Participation
by Robin Haw 11 Feb '19

11 Feb '19

Dear All, The Open Genome Informatics team serves as an “umbrella" organization to support the efforts of many open access open-source bioinformatics projects for Google Summer of Code (GSoC)<https://summerofcode.withgoogle.com/>. Among this list of projects are GMOD and its software projects -- GBrowse, JBrowse; Galaxy; Reactome; WormBase; DockStore; Bioconda; and others. Call for 2019 Project Ideas and Mentors: We are seeking project ideas to post and attract talented students to this year’s Summer of Code competition. If you have a project idea for which you would like to mentor a student, please contact Robin Haw, Marc Gillespie, Dave Clements, Dannon Baker, and Scott Cain (emails above). You can also submit your ideas here<http://gmod.org/wiki/GSOC_Project_Ideas_2019> or this online form<https://docs.google.com/forms/d/1uQlpsbPCnhNwO5OFKAB-IiY9DLWlmaHZATnaeP6KHhw>. For more information please refer to the Open Genome Informatics page<http://gmod.org/wiki/GSoC>. The mentoring organization application deadline with GSoC is February 6th, 2019 at 3 pm EST. So, if you are interested in taking part with the team please let us know as soon as possible. Please forward this to others who might be interested in taking part. If you have any questions please let us know. Thanks, Robin, Marc, Dave, Dannon, and Scott.

1 1

Uncaught exception in exposed API method:
by Jochen Bick 07 Feb '19

07 Feb '19

Hi, I'm having this error since some minutes. The error message is coming in the moment I click to any tool (e.g.: featurecounts) that provides data collections. I think it has to do with a collection in my history. Here some detail of my log file: galaxy.web.framework.decorators ERROR 2019-02-06 15:50:09,026 Uncaught exception in exposed API method: Traceback (most recent call last): File "/opt/galaxy/production/galaxy-dist/lib/galaxy/web/framework/decorators.py", line 283, in decorator rval = func(self, trans, *args, **kwargs) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/webapps/galaxy/api/tools.py", line 113, in build return tool.to_json(trans, kwd.get('inputs', kwd)) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/__init__.py", line 1903, in to_json self.populate_model(request_context, self.inputs, state_inputs, tool_model['inputs']) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/__init__.py", line 1951, in populate_model tool_dict = input.to_dict(request_context) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/parameters/grouping.py", line 701, in to_dict cond_dict["cases"] = list(map(nested_to_dict, self.cases)) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/parameters/grouping.py", line 699, in nested_to_dict return input.to_dict(trans) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/parameters/grouping.py", line 719, in to_dict when_dict["inputs"] = list(map(input_to_dict, self.inputs.values())) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/parameters/grouping.py", line 717, in input_to_dict return input.to_dict(trans) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/parameters/basic.py", line 1966, in to_dict match = dataset_collection_matcher.hdca_match(hdca) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/tools/parameters/dataset_matcher.py", line 226, in hdca_match converted_ext, _ = datatypes_registry.find_conversion_destination_for_dataset_by_extensions(extension, formats) File "/opt/galaxy/production/galaxy-dist/lib/galaxy/datatypes/registry.py", line 892, in find_conversion_destination_for_dataset_by_extensions self.log.warning("Datatype class not found for extension '%s', which is used as target for conversion from datatype '%s'" % (convert_ext, dataset.ext)) AttributeError: 'NoneType' object has no attribute 'ext' 82.130.95.243 - - [06/Feb/2019:15:50:08 +0200] "GET /api/tools/toolshed.g2.bx.psu.edu/repos/iuc/featurecounts/featurecounts/1.6.3%2Bgalaxy2/build?version=1.6.3%2Bgalaxy2&__identifer=ku6ydfn1c8&tool_version=1.6.3%252Bgalaxy2 HTTP/1.1" 500 - "http://green-l-bio.ethz.ch:8090/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fi…" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36" Any ideas? Errors only occur if I'm in that history. I can also provide more information. This is running on a Galaxy 18.09. The data collections contains 8 fastq files. Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

1 2

Question using sftp to upload file to galaxy
by Rui Wang 06 Feb '19

06 Feb '19

Hey Folks, I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference? What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo). Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong! Cheers, Rui modules: $ sbin/proftpd -l Compiled-in modules: mod_core.c mod_xfer.c mod_rlimit.c mod_auth_unix.c mod_auth.c mod_ls.c mod_log.c mod_site.c mod_delay.c mod_facts.c mod_sql.c mod_sql_postgres.c mod_sql_passwd.c mod_sftp.c mod_cap.c etc/proftpd.conf ServerType standalone # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost. <VirtualHost 10.3.17.42> Port 2222 SFTPEngine on AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects SFTPHostKey /etc/ssh/ssh_host_rsa_key RequireValidShell no MaxLoginAttempts 6 ServerName "Galaxy SFTP" DefaultServer on Umask 077 User bioinfoadmin Group bioinfoadmin UseFtpUsers off DefaultRoot ~ AllowOverwrite on AllowStoreRestart on SQLEngine on SQLGroupInfo sftp_groups name id members # Do not authenticate against real (system) users <IfModule mod_auth_pam.c> AuthPAM off </IfModule> # Common SQL authentication options SQLPasswordEngine on SQLBackend postgres SQLConnectInfo galaxy@galaxy.my.org:5432 bioinfoadmin dbpwd SQLAuthenticate users # Configuration that handles PBKDF2 encryption # Set up mod_sql to authenticate against the Galaxy database SQLAuthTypes PBKDF2 SQLPasswordPBKDF2 SHA256 10000 24 SQLPasswordEncoding base64 SQLPasswordUserSalt sql:/GetUserSalt # Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" # Define custom query to fetch the password salt SQLNamedQuery GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'" </VirtualHost> # Don't use IPv6 support by default. UseIPv6 off MaxInstances 30 # To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit> # Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> ~

2 4

featureCounts with GFF
by Jochen Bick 05 Feb '19

05 Feb '19

Hi, our group would like to switch to featureCounts <http://green-l-bio.ethz.ch:8090/tool_runner?tool_id=toolshed.g2.bx.psu.edu%…> but we are using only GFF from NCBI. It looks like featureCounts <http://green-l-bio.ethz.ch:8090/tool_runner?tool_id=toolshed.g2.bx.psu.edu%…> provides GFF but out of the box it does not provide a proper count table. The output looks like this: Status HISAT2 on data 33 Assigned 5186527 Unassigned_Unmapped 518985 Unassigned_MappingQuality 1353602 Unassigned_Chimera 0 Unassigned_FragmentLength 0 Unassigned_Duplicate 0 Unassigned_MultiMapping 0 Unassigned_Secondary 0 Unassigned_Nonjunction 0 Unassigned_NoFeatures 7515020 Unassigned_Overlapping_Length 0 Unassigned_Ambiguity 0 Geneid HISAT2 on data 33 5186527 I guess I need some advanced options? Which are different to the default run settings? Does anyone have a tutorial or example on how to configure the tool correctly with a standard GFF file, or does anyone also use GFF files? Or is there a better tool to use? Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

1 0

add labels to tools
by Jochen Bick 04 Feb '19

04 Feb '19

Hi, I'm interested to add labels to tool such as shown on usegalaxy.org for example for the tool hifive <https://usegalaxy.org/tool_runner?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2F…>. How can I add these labels. Cheers Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick(a)usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch

2 3

Re: [galaxy-dev] [External] Copying files to FTP upload directory
by Keith Suderman 01 Feb '19

01 Feb '19

I have been creating directories for the users, but putting files in (for example) /home/galaxy/database/ftp/user(a)example.com <mailto:home/galaxy/database/ftp/user@example.com> does not make them available for the user(a)example.com <mailto:user@example.com> user in Galaxy. Also for the list admins... I never did see my message appear on the list and the message Cicada replied to appears to be from galaxy-dev-bounces(a)lists.galaxyproject.org <mailto:galaxy-dev-bounces@lists.galaxyproject.org>. Any ideas why my message(s) appear to be bouncing. I am pretty sure I am sending from the correct address. Cheers, Keith > On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw <hbrokaw(a)iu.edu> wrote: > > I think you have to create the user's directories under /home/galaxy/database/ftp. I don't think Galaxy will create them if they are not there. > Cicada > > ________________________________________ > From: galaxy-dev <galaxy-dev-bounces(a)lists.galaxyproject.org> on behalf of Keith Suderman <suderman(a)cs.vassar.edu> > Sent: Friday, January 25, 2019 4:01 PM > To: Galaxy Dev List > Subject: [External] [galaxy-dev] Copying files to FTP upload directory > > Hello, > > I am trying to (ab)use the FTP file upload functionality to get data from an external service into Galaxy. From my reading of the docs I simply need to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't migrated to yaml yet) and Galaxy will expect to find subdirectories named after the user's account (email) in the ftp_upload_dir. > > I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) and the "Choose FTP File" button is now available in the Upload dialog, however regardless of where I put files, or what I name them, Galaxy always reports that "Your FTP directory does not contain any files." For example, if I have the user account user(a)example.org I would expect to be able to place files in /home/galaxy/database/ftp/user(a)example.org and they will be available in the Upload dialog for the user(a)example.org user. However, this is not working; am I missing something? > > Thanks, > Keith > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/

2 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

galaxy-dev February 2019