- galaxy-dev - lists.galaxyproject.org

Speedup Galaxy
by Jochen Bick 19 Dec '16

19 Dec '16

We are running a Galaxy instance at our institute (version 15.10). Since some month the speed/reaction time of the interface drastically decrease. Sometimes there is a lack of more than a minute loading a single tool. I restarted galaxy without any improvements. We are working with up to 3 people on this instance that should not be the problem. Does anyone have any idea? Disk-space? uptime of our server? big full database? ?? Cheers Jochen

3 5

Galaxy Security Vulnerability - Arbitrary code execution
by Nate Coraor 19 Dec '16

19 Dec '16

*DESCRIPTION* A security vulnerability was recently discovered by David Wyde in two Galaxy tools that are provided with and enabled by default in the Galaxy server. These vulnerabilities allow malicious actors to execute arbitrary code as the user running the Galaxy server. The vulnerable tools are "Filter GFF data by attribute" and "Filter GFF data by feature count", two tools which share code with each other and the more general "Filter data on any column using simple expressions" tool. The latter was fixed in a previous security disclosure but these GFF variants of the tool were missed when updating the Filter tool. These tools use the Python `eval` and `exec` functions and do not properly sanitize input to these functions. Additionally, an XSS vulnerability with the upload tool: An uploaded file's name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. This vulnerability has already been fixed in the 16.07 and 16.10 Galaxy releases. The Galaxy Committers would like to thank David for disclosing these vulnerabilities. If you discover security vulnerabilities in Galaxy, please report them to the Galaxy Committers at galaxy-committers(a)lists.galaxyproject.org. *AFFECTED VERSIONS* These arbitrary code execution vulnerability is present in all known releases of Galaxy using the default tool config or a tool config in which these tools are loaded. To determine if your Galaxy server is affected, check your tool_conf.xml for the following lines: <tool file="filters/gff/gff_filter_by_attribute.xml" /> <tool file="filters/gff/gff_filter_by_feature_count.xml" /> If you are unsure, you can also check to see if the tools have loaded by searching for them in your Galaxy server logs, e.g.: % grep gff_filter paster.log galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: gff_filter_by_attribute, version: 0.1 into tool panel.. galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: gff_filter_by_feature_count, version: 0.1 into tool panel.. *IMPACT* This vulnerability can be exploited to execute arbitrary code as the user that runs Galaxy jobs on any system where Galaxy runs those jobs (potentially a cluster). Such code can be used to provide a remote shell to an attacker, or read/write/delete files that the Galaxy user has appropriate permissions on. *SOLUTION* Multiple solutions exist for the arbitrary code execution vulnerability: 1. Update your Galaxy code. A fix for the issue has been applied to stable releases of Galaxy back to version 14.10. 2. Disable the affected tools. 3. Patch the affected tools. Patches to fix the vulnerabilities (depending on the version of Galaxy that you are running) can be found below. *INSTRUCTIONS* 1. To apply the fix, first identify your current Galaxy release version using the `git branch` command. If you are on a 'release_YY.MM' branch, you can update with: % git pull The process above can also be used to update to the 16.10 release if you are on the 'master' git branch. If you are on the 'master' branch and wish to remain on your current Galaxy major release, check the 'lib/galaxy/version.py' file to determine your major release version, then update to the appropriate branch: % git fetch origin % git checkout -b release_YY.MM origin/release_YY.MM % git pull 2. To disable the tools, locate and remove the following lines from your tool_conf.xml, then restart Galaxy: <tool file="filters/gff/gff_filter_by_attribute.xml" /> <tool file="filters/gff/gff_filter_by_feature_count.xml" /> 3. To manually patch the tools, use the appropriate patch file below: For Galaxy releases 16.01 to dev: https://depot.galaxyproject.org/patch/filter_security_16_01.patch For Galaxy releases 15.07 to 15.10: https://depot.galaxyproject.org/patch/filter_security_15_07.patch For Galaxy releases 14.10 to 15.05: https://depot.galaxyproject.org/patch/filter_security_14_10.patch To apply the patch, navigate to the root of your Galaxy directory, then execute (replacing <patch_url> with the url above appropriate for your current Galaxy release): % wget -O filter_security.patch <patch_url> or: % curl -o filter_security.patch <patch_url> and then test applying the patch: % patch -p1 --dry-run < filter_security.patch checking file tools/filters/gff/gff_filter_by_attribute.py checking file tools/filters/gff/gff_filter_by_attribute.xml checking file tools/filters/gff/gff_filter_by_feature_count.py checking file tools/filters/gff/gff_filter_by_feature_count.xml If this succeeds without error (applying successfully with fuzz/offset is fine), apply with: % patch -p1 < filter_security.patch patching file tools/filters/gff/gff_filter_by_attribute.py patching file tools/filters/gff/gff_filter_by_attribute.xml patching file tools/filters/gff/gff_filter_by_feature_count.py patching file tools/filters/gff/gff_filter_by_feature_count.xml In all cases, for the changes to take effect, *YOU MUST RESTART ALL GALAXY SERVER PROCESSES*. On behalf of the Galaxy Committers, --nate

4 3

Announcing the October 2016 (v16.10) Galaxy Release
by Nate Coraor 15 Dec '16

15 Dec '16

The Galaxy Committers team <https://github.com/galaxyproject/galaxy/blob/dev/doc/source/project/organiz…> is pleased to announce the *October 2016 (v16.10) release of Galaxy <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html>*. Security <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html#secur…> An arbitrary code execution vulnerability in two tools and an XSS vulnerability with the upload tool were identified this release cycle and have been fixed concurrently with the release. In addition, the fixes have been backported to older releases. The Galaxy Committers would like to thank David Wyde for disclosing these vulnerabilities. Details follow: 1. The vulnerable tools are “Filter GFF data by attribute” and “Filter GFF data by feature count”, both of which are provided with and enabled by default in the Galaxy server. These two tools share code with each other and the more general “Filter data on any column using simple expressions” tool. The latter was fixed in a previous security disclosure but these GFF variants of the tool were missed when updating the Filter tool. These tools use the Python eval and exec functions and do not properly sanitize input to these functions. The fix for this issue has been applied to Galaxy releases back to v14.10 and can be found in Commit c1e3087 2. An uploaded file’s name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. The fix for this issue has been applied to Galaxy releases back to v16.07 and can be found in Pull Request 3278. Highlighted Enhancements <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html#highl…> - Overhaul of charts visualization - more visualizations, more options, and better user interface. - Paginate contents of large histories. - Implement a collection operation tool for merging collections (thanks to @Takadonet). - Replace reference documentation for tool XML files with automatically generated documentation from a now official Galaxy XSD documentation (with help from many). - Add a password strength evaluation bar (thanks to @benfulton). - Implement a GoDocker job runner (thanks to @varunshankar). - Support for API batch requests. - Allow JSONP to be returned from API endpoints. - Add “Save as” as an option in the workflow editor (thanks to @tmcgowan). - Allow naming input datasets and collections during workflow extraction. - Various enhancements for API driven installation of tool shed repositories. Get Galaxy <http://getgalaxy.org/> The code lives at Github <https://github.com/galaxyproject/galaxy> and you should have Git <https://git-scm.com/> to obtain it. *To get a new Galaxy repository run:* $ git clone -b release_16.10 https://github.com/galaxyproject/galaxy.git *To update an existing Galaxy repository run:* $ git checkout release_16.10 && git pull --ff-only origin release_16.10 Release Notes <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html> For full details on all of the enhancements and fixes in this release, please see the full release notes <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html>. On behalf of the Galaxy Committers <https://github.com/galaxyproject/galaxy/blob/dev/doc/source/project/organiz…> , *Thanks for using Galaxy!* --nate

1 0

devteam dropped support for installing tool dependencies from toolshed?
by Peter Briggs 15 Dec '16

15 Dec '16

Dear developers It looks like some updates to 'devteam'-maintained toolshed tools have dropped support for tool_dependencies.xml (e.g. fastqc, freebayes), so their dependencies cannot be automatically installed from the toolshed. Is this correct? I assume that it's part of the move to enforcing conda dependency resolution. Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 3

New datatypes with code dependencies
by Cristel Thomas 14 Dec '16

14 Dec '16

Hello, We'd like to add some datatypes to Galaxy specific to flow cytometry (finally). One of the datatype is a binary format which requires rpy2/R to be read (no good python module available to do that). How do you guys deal with that? Thanks, Cris

2 2

multiple datatypes input in 16.07
by Cristel Thomas 14 Dec '16

14 Dec '16

Hi Galaxy Devs, We have a few tools with multiple datatypes as input. They work fine in 16.01. In 16.07 though, the selection of files in the tools' drop-downs include every single file in the history, instead of just the files from the specified data types. Anyone else run into this? Any idea of what's going on? Thanks much, Cris

2 11

Testing workflows
by Eric Rasche 14 Dec '16

14 Dec '16

Wondering if anyone has tooling to test a workflow as it exists in a specific galaxy instance? I'm looking to test some of my user's manually built workflows in our galaxy instance to make sure that we can detect issues when they arise (e.g. due to changes in the interfaces of our non-toolshed installed tools) Ideally something like: $ my_tool $galaxy_url $api_key $workflow_id <set of files and their input IDs in the workflow> > report.xunit.xml If not we'll build something soon. Cheers, Eric

2 1

Addind new datatype - best practice
by Floreline TOUCHARD 11 Dec '16

11 Dec '16

Hi Galaxy community, I would like to add custom datatypes to my Galaxy server without editing any Galaxy files if possible. I've already created and renamed a second config/datatype.conf file. Both files are listed into galaxy.ini (variable datatypes_config_file). But what about the new datatype Class ? For example if it's a binary class, how can I add this class without editing the binary.py file or any other file (if it's possible) ? Thanks in advance, Floreline

2 1

Broken logo 'src' in masthead for Galaxy 16.07 served from subdirectory
by Peter Briggs 08 Dec '16

08 Dec '16

Dear developers Since upgrading to release_16.07 I've noticed a broken link to the masthead icon (i.e. the Galaxy logo 'galaxyIcon_noText.png') in our local Galaxy installations, which are all served from subdirectories of the webserver, e.g. https://centaurus.itservices.manchester.ac.uk/galaxy/ The icon is visible when this URL is visited but appears to be broken when using parts of Galaxy served from their own subdirectories e.g. the login page (under https://centaurus.itservices.manchester.ac.uk/galaxy/user/login?...) or the admin interface (https://centaurus.itservices.manchester.ac.uk/galaxy/admin) At the top level the logo <img> "src" is points correctly to "/galaxy/static/images/galaxyIcon_noText.png"; in the subdirs (users,admin etc) it points to "../../../static/images/galaxyIcon_noText.png" (which cannot be loaded). As far as I can work out, in 16.07 the "src" location is generated from templates/webapps/galaxy/galaxy.masthead.mako, but takes its value from the 'logo_src' parameter that is set in lib/galaxy/managers/configuration.py: ... # TODO: this doesn't seem right 'logo_url' : lambda i, k, **c: self.url_for( i.get( k, '/' ) ), 'logo_src' : lambda i, k, **c: self.url_for( '/static/images/galaxyIcon_noText.png' ), ... However I have no idea how to try and fix it - presumably it needs to incorporate the proxy-prefix setting somehow. (Btw the 'logo_url' does seem to point to the correct location however.) Can someone suggest a fix? Thanks Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 2

Error when "Export History to File"
by D K 06 Dec '16

06 Dec '16

Hi galaxy-devs, I get an "Internal Server Error" when trying to "Export History to File". I get the following in my paster.log file: > .... Error - <type 'exceptions.AttributeError'>: 'NoneType' object has no > attribute 'execute' > File '/opt/galaxy/lib/galaxy/web/framework/middleware/error.py', line 151 > in __call__ > app_iter = self.application(environ, sr_checker) > File '/opt/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py', > line 85 in __call__ > return self.application(environ, start_response) > File > '/opt/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py', > line 640 in __call__ > return self.application(environ, start_response) > File '/opt/galaxy/lib/galaxy/web/framework/base.py', line 135 in __call__ > return self.handle_request( environ, start_response ) > File '/opt/galaxy/lib/galaxy/web/framework/base.py', line 194 in > handle_request > body = method( trans, **kwargs ) > File '/opt/galaxy/lib/galaxy/webapps/galaxy/controllers/history.py', line > 1260 in export_archive > self.queue_history_export( trans, history, gzip=gzip, > include_hidden=include_hidden, include_deleted=include_deleted ) > File '/opt/galaxy/lib/galaxy/web/base/controller.py', line 393 in > queue_history_export > history_exp_tool.execute( trans, incoming=params, history=history, > set_output_hid=True ) > AttributeError: 'NoneType' object has no attribute 'execute' > > CGI Variables > ------------- > CONTENT_LENGTH: '0' > HTTP_ACCEPT: > 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' > HTTP_ACCEPT_ENCODING: 'gzip, deflate, sdch' > HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8,fr;q=0.6' > HTTP_CONNECTION: 'keep-alive' > HTTP_COOKIE: > 'galaxysession=c6ca0ddb55be603a84da157bd892a9e699b161141ab9ce96d24ae4db9c34333de097bd8cf0510125; > SAPPORTALSDB0=urn%253Acom.sapportals.portal%253Adsm%2526debug%3D*; > SAPWP_active=1; > SMSESSION=rPd78cw60F92I7vTkZJ+rcZrVastzVz1RAzoApqSPjO0kh8JFHBilMICVdggRAPy9UTdeCHbOUPoeVqey55n2a31uBfTDi1SrwY/c8UtqGxmYgqYbWixw5r//fMBqUNRCLwG7hdvWkY+oJxhcxQNuhvwPurtoDZRZgRW42tXnb8o5KEFSbuELqdb6d4yVUGHnYJhllennq+qD4JGp1yB0KJdCQ3DKQYz6aR3234JjstwV6iHod/Xi40FZJZlNskjWSFPu7G0u0HFsNl0KEHlIWmmdm46rpuY4b5VxRfCS3O+OBdAOvkBEQO+h2EUcnLmaoXutSO10/hO2fRs69w3wio3NPRSi9FxWi6o+PbjcOkuXOxL5li8RbuO5319v3WHEpQYlv6jlatvVukXbVQ7/904WYuDQdeNFU0GReO6io3FuLAw7zYKq/IHGquDTCIWY9V2Ujue5GTNPcW/yLsLJWWQi5CuQRqZFK7mYmdKwbQTTCmp7OaqZGPaEa9oLIffBt37GKsKdN6ab1HyO3zwt5iT2A9MvCBsh2GvtLl0jnI7MSTq8LjyaA0jRxDZJQrUogIOleDIC/1WdtN3koYZSUgSGYNqInmeZtQXlXmYRdlRNxoO/7c+hoSUCwhDxbgdrgE3LW3Q/lSBJVChrsIKYOifzSNyCIR8Ved1DnTjnqL7XoAuroENr5vOLxr35KatwhzLgowUwtfXIBnk79IomPkBRVR4BYnJQi2yOdTbu7cLe8Y2VdQRbA5J1geG1Q8B0nMOWUxr9OLzhf+v4gkZMu...17118.1452298948' > HTTP_HOST: 'xxx.com:8080' > HTTP_REFERER: 'xxx.com:8080/galaxy' > HTTP_UPGRADE_INSECURE_REQUESTS: '1' > HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 > (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' > PATH_INFO: '/history/export_archive' > QUERY_STRING: 'preview=True' > REMOTE_ADDR: '10.20.52.124' > REQUEST_METHOD: 'GET' > SCRIPT_NAME: '/galaxy' > SERVER_NAME: '0.0.0.0' > SERVER_PORT: '8080' > SERVER_PROTOCOL: 'HTTP/1.1' > > WSGI Variables > -------------- > application: <paste.recursive.RecursiveMiddleware object at 0xc88eb50> > controller_action_key: u'web.history.export_archive' > is_api_request: False > paste.cookies: (<SimpleCookie: > SAPPORTALSDB0='urn%253Acom.sapportals.portal%253Adsm%2526debug%3D*' > SAPWP_active='1' > SMSESSION='rPd78cw60F92I7vTkZJ+rcZrVastzVz1RAzoApqSPjO0kh8JFHBilMICVdggRAPy9UTdeCHbOUPoeVqey55n2a31uBfTDi1SrwY/c8UtqGxmYgqYbWixw5r//fMBqUNRCLwG7hdvWkY+oJxhcxQNuhvwPurtoDZRZgRW42tXnb8o5KEFSbuELqdb6d4yVUGHnYJhllennq+qD4JGp1yB0KJdCQ3DKQYz6aR3234JjstwV6iHod/Xi40FZJZlNskjWSFPu7G0u0HFsNl0KEHlIWmmdm46rpuY4b5VxRfCS3O+OBdAOvkBEQO+h2EUcnLmaoXutSO10/hO2fRs69w3wio3NPRSi9FxWi6o+PbjcOkuXOxL5li8RbuO5319v3WHEpQYlv6jlatvVukXbVQ7/904WYuDQdeNFU0GReO6io3FuLAw7zYKq/IHGquDTCIWY9V2Ujue5GTNPcW/yLsLJWWQi5CuQRqZFK7mYmdKwbQTTCmp7OaqZGPaEa9oLIffBt37GKsKdN6ab1HyO3zwt5iT2A9MvCBsh2GvtLl0jnI7MSTq8LjyaA0jRxDZJQrUogIOleDIC/1WdtN3koYZSUgSGYNqInmeZtQXlXmYRdlRNxoO/7c+hoSUCwhDxbgdrgE3LW3Q/lSBJVChrsIKYOifzSNyCIR8Ved1DnTjnqL7XoAuroENr5vOLxr35KatwhzLgowUwtfXIBnk79IomPkBRVR4BYnJQi2yOdTbu7cLe8Y2VdQRbA5J1geG1Q8B0nMOWUxr9OLzhf+v4gkZMujpesKRr1BUuBvSidQSn3p7kdKWD4RiRkWHNsN3E7rFQLMJclY0j47P06ms3KUG1T0JNVJWdWI1MzwN...7118.1452298948') > paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] > paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object > at 0xc88ead0> > paste.httpserver.proxy.host: 'dummy' > paste.httpserver.proxy.scheme: 'http' > paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at > 0xb8ae210> > paste.parsed_querystring: ([('preview', 'True')], 'preview=True') > paste.recursive.forward: <paste.recursive.Forwarder from /galaxy> > paste.recursive.include: <paste.recursive.Includer from /galaxy> > paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from > /galaxy> > paste.recursive.script_name: '/galaxy' > paste.throw_errors: True > request_id: '6abfe982bc1011e69fd4005056ab0b73' > webob._parsed_query_vars: (GET([(u'preview', u'True')]), 'preview=True') > wsgi process: 'Multithreaded' > ------------------------------------------------------------

1 0