December 2016 - galaxy-dev - lists.galaxyproject.org

GalaxyAdmins Online Meetup 15 December
by Dave Clements 03 Jan '17

03 Jan '17

Hello all, The next GalaxyAdmins online meetup <https://new.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_12_15/> is this Thursday, 15 December at [see your local time <http://bit.ly/2h2QOsT>]. The meetup will feature two presentations: - Galaxy Training Materials: Community creation, curation, and use, by Björn Grüning and Bérénice Batut, University of Freiburg - The new Galaxy Hub: Community creation, curation, and use, by Dannon Baker, Johns Hopkins University. The call will last around an hour, and will include discussion. Hope to see you there, Dave C & HAns-Rudolf -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 1

Job launching slowed down by conda auto-install
by Tiago Macedo 21 Dec '16

21 Dec '16

Hello all, I recently realized that setting conda_auto_install to true in galaxy.ini has a noticeable effect in slowing down the start of job execution. On the first job run after activating conda_auto_install the job takes a lot longer as conda is actually installing packages, which is expected. On the following runs, even if there are no packages to be installed jobs still take longer to start. What's more surprising is that after deactivating conda_auto_install jobs still take longer to run. My particular test case was a simple small (12MB) file upload. It consistently went from 15s to over 30s after activating conda_auto_install and remained at that even after deactivating it. We tested running a few tools which presented a noticeable and irreversible delay after activation of conda_auto_install, but I didn't measure these delays. I also tested this on fresh installs of galaxy 16.07 and 16.10 with very similar results. Is this supposed to happen? Is there a way around it besides never activating conda_auto_install? Thanks in advance, Tiago

2 1

tmp file under database/tmp get regenerated after restarting Galaxy
by Mark Xiao 20 Dec '16

20 Dec '16

Dear Galaxy Developers, I am using Galaxy version 15.10 on my server, load-balancing is enabled. I found a problem when running the jobs. When there are running jobs and if I restart the Galaxy process at this time, the job will fail at the end of the job with the error message such as: IOError: [Errno 2] No such file or directory: '/galaxy/database/tmp/tmpleXN4b' That tmp file doesn't exist, I guess it is because when Galaxy process get restarted, the tmp file get regenerated, so the job cannot find the correct tmp file, then it was failed. Do you think this is a known issue, and is there a fix for this? Thank you in advance! Mark

1 1

Tools pick up Python modules from Galaxy core rather than installed tool dependencies
by Peter Briggs 20 Dec '16

20 Dec '16

Dear devs I was wondering if anyone had any suggestions as to the cause of a peculiar error I'm seeing when running the latest versions of the devteam "join" and "fastq_paired_end_joiner". The tools appear to be picking up Python modules from the Galaxy library, rather than from the installed dependencies. The PYTHONPATH appears to be correct (i.e. tool dependency appears before Galaxy library path) but sys.path shows the dependency's Python egg appearing after the Galaxy library path. E.g. for the paired-end joiner: PYTHONPATH=/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python:/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib sys.path = ['', '/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python', '/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib', '/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python/galaxy_sequence_utils-1.0.1-py2.7.egg', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python27.zip', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/plat-linux2', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-tk', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-old', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-dynload', '/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7', '/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/plat-linux2', '/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/lib-tk', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/site-packages'] Error: Traceback (most recent call last): File "/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py", line 158, in <module> main() File "/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py", line 136, in main joiner = fq.fastqJoiner( input1_type, paste=paste ) TypeError: __init__() got an unexpected keyword argument 'paste' I know it's a long shot but can anyone suggest where to start looking at where this is going wrong? Thanks, Peter Ps I'm using Galaxy v16.07 and the problem occurs on our production platform but not in a local Galaxy install on my desktop. -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 3

Connection to mariaDB generates tables, but does not migrate information
by evan clark 20 Dec '16

20 Dec '16

I connected our galaxy instance to mariadb with no connection issues and it began to migrate the data, I noticed however when galaxy was finally started the tables were generated, but no data was migrated. It completed the migration without error so I am not sure if there is a setting I missed in mariadb.

3 2

Installation of packages failing
by James Mullan 19 Dec '16

19 Dec '16

Hello, I am trying to install packages with dependencies from Galaxy Main tool shed. This worked Thursday and since Friday it no longer works for me. It seems this url is no longer reachable. Examples of what I see in the log during download are shown below. Any help would be much appreciated. package_bowtie_0_12_7 Error downloading from URL http://depot.galaxyproject.org/package/linux/x86_64/bowtie/bowtie-0.12.7-li… package_samtools_0_1_19 Exception: Error downloading from URL https://depot.galaxyproject.org/software/samtools/samtools_0.1.19_src_all.t… Regards James J P Mullan

4 6

Speedup Galaxy
by Jochen Bick 19 Dec '16

19 Dec '16

We are running a Galaxy instance at our institute (version 15.10). Since some month the speed/reaction time of the interface drastically decrease. Sometimes there is a lack of more than a minute loading a single tool. I restarted galaxy without any improvements. We are working with up to 3 people on this instance that should not be the problem. Does anyone have any idea? Disk-space? uptime of our server? big full database? ?? Cheers Jochen

3 5

Galaxy Security Vulnerability - Arbitrary code execution
by Nate Coraor 19 Dec '16

19 Dec '16

*DESCRIPTION* A security vulnerability was recently discovered by David Wyde in two Galaxy tools that are provided with and enabled by default in the Galaxy server. These vulnerabilities allow malicious actors to execute arbitrary code as the user running the Galaxy server. The vulnerable tools are "Filter GFF data by attribute" and "Filter GFF data by feature count", two tools which share code with each other and the more general "Filter data on any column using simple expressions" tool. The latter was fixed in a previous security disclosure but these GFF variants of the tool were missed when updating the Filter tool. These tools use the Python `eval` and `exec` functions and do not properly sanitize input to these functions. Additionally, an XSS vulnerability with the upload tool: An uploaded file's name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. This vulnerability has already been fixed in the 16.07 and 16.10 Galaxy releases. The Galaxy Committers would like to thank David for disclosing these vulnerabilities. If you discover security vulnerabilities in Galaxy, please report them to the Galaxy Committers at galaxy-committers(a)lists.galaxyproject.org. *AFFECTED VERSIONS* These arbitrary code execution vulnerability is present in all known releases of Galaxy using the default tool config or a tool config in which these tools are loaded. To determine if your Galaxy server is affected, check your tool_conf.xml for the following lines: <tool file="filters/gff/gff_filter_by_attribute.xml" /> <tool file="filters/gff/gff_filter_by_feature_count.xml" /> If you are unsure, you can also check to see if the tools have loaded by searching for them in your Galaxy server logs, e.g.: % grep gff_filter paster.log galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: gff_filter_by_attribute, version: 0.1 into tool panel.. galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: gff_filter_by_feature_count, version: 0.1 into tool panel.. *IMPACT* This vulnerability can be exploited to execute arbitrary code as the user that runs Galaxy jobs on any system where Galaxy runs those jobs (potentially a cluster). Such code can be used to provide a remote shell to an attacker, or read/write/delete files that the Galaxy user has appropriate permissions on. *SOLUTION* Multiple solutions exist for the arbitrary code execution vulnerability: 1. Update your Galaxy code. A fix for the issue has been applied to stable releases of Galaxy back to version 14.10. 2. Disable the affected tools. 3. Patch the affected tools. Patches to fix the vulnerabilities (depending on the version of Galaxy that you are running) can be found below. *INSTRUCTIONS* 1. To apply the fix, first identify your current Galaxy release version using the `git branch` command. If you are on a 'release_YY.MM' branch, you can update with: % git pull The process above can also be used to update to the 16.10 release if you are on the 'master' git branch. If you are on the 'master' branch and wish to remain on your current Galaxy major release, check the 'lib/galaxy/version.py' file to determine your major release version, then update to the appropriate branch: % git fetch origin % git checkout -b release_YY.MM origin/release_YY.MM % git pull 2. To disable the tools, locate and remove the following lines from your tool_conf.xml, then restart Galaxy: <tool file="filters/gff/gff_filter_by_attribute.xml" /> <tool file="filters/gff/gff_filter_by_feature_count.xml" /> 3. To manually patch the tools, use the appropriate patch file below: For Galaxy releases 16.01 to dev: https://depot.galaxyproject.org/patch/filter_security_16_01.patch For Galaxy releases 15.07 to 15.10: https://depot.galaxyproject.org/patch/filter_security_15_07.patch For Galaxy releases 14.10 to 15.05: https://depot.galaxyproject.org/patch/filter_security_14_10.patch To apply the patch, navigate to the root of your Galaxy directory, then execute (replacing <patch_url> with the url above appropriate for your current Galaxy release): % wget -O filter_security.patch <patch_url> or: % curl -o filter_security.patch <patch_url> and then test applying the patch: % patch -p1 --dry-run < filter_security.patch checking file tools/filters/gff/gff_filter_by_attribute.py checking file tools/filters/gff/gff_filter_by_attribute.xml checking file tools/filters/gff/gff_filter_by_feature_count.py checking file tools/filters/gff/gff_filter_by_feature_count.xml If this succeeds without error (applying successfully with fuzz/offset is fine), apply with: % patch -p1 < filter_security.patch patching file tools/filters/gff/gff_filter_by_attribute.py patching file tools/filters/gff/gff_filter_by_attribute.xml patching file tools/filters/gff/gff_filter_by_feature_count.py patching file tools/filters/gff/gff_filter_by_feature_count.xml In all cases, for the changes to take effect, *YOU MUST RESTART ALL GALAXY SERVER PROCESSES*. On behalf of the Galaxy Committers, --nate

4 3

Announcing the October 2016 (v16.10) Galaxy Release
by Nate Coraor 15 Dec '16

15 Dec '16

The Galaxy Committers team <https://github.com/galaxyproject/galaxy/blob/dev/doc/source/project/organiz…> is pleased to announce the *October 2016 (v16.10) release of Galaxy <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html>*. Security <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html#secur…> An arbitrary code execution vulnerability in two tools and an XSS vulnerability with the upload tool were identified this release cycle and have been fixed concurrently with the release. In addition, the fixes have been backported to older releases. The Galaxy Committers would like to thank David Wyde for disclosing these vulnerabilities. Details follow: 1. The vulnerable tools are “Filter GFF data by attribute” and “Filter GFF data by feature count”, both of which are provided with and enabled by default in the Galaxy server. These two tools share code with each other and the more general “Filter data on any column using simple expressions” tool. The latter was fixed in a previous security disclosure but these GFF variants of the tool were missed when updating the Filter tool. These tools use the Python eval and exec functions and do not properly sanitize input to these functions. The fix for this issue has been applied to Galaxy releases back to v14.10 and can be found in Commit c1e3087 2. An uploaded file’s name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. The fix for this issue has been applied to Galaxy releases back to v16.07 and can be found in Pull Request 3278. Highlighted Enhancements <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html#highl…> - Overhaul of charts visualization - more visualizations, more options, and better user interface. - Paginate contents of large histories. - Implement a collection operation tool for merging collections (thanks to @Takadonet). - Replace reference documentation for tool XML files with automatically generated documentation from a now official Galaxy XSD documentation (with help from many). - Add a password strength evaluation bar (thanks to @benfulton). - Implement a GoDocker job runner (thanks to @varunshankar). - Support for API batch requests. - Allow JSONP to be returned from API endpoints. - Add “Save as” as an option in the workflow editor (thanks to @tmcgowan). - Allow naming input datasets and collections during workflow extraction. - Various enhancements for API driven installation of tool shed repositories. Get Galaxy <http://getgalaxy.org/> The code lives at Github <https://github.com/galaxyproject/galaxy> and you should have Git <https://git-scm.com/> to obtain it. *To get a new Galaxy repository run:* $ git clone -b release_16.10 https://github.com/galaxyproject/galaxy.git *To update an existing Galaxy repository run:* $ git checkout release_16.10 && git pull --ff-only origin release_16.10 Release Notes <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html> For full details on all of the enhancements and fixes in this release, please see the full release notes <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html>. On behalf of the Galaxy Committers <https://github.com/galaxyproject/galaxy/blob/dev/doc/source/project/organiz…> , *Thanks for using Galaxy!* --nate

1 0

devteam dropped support for installing tool dependencies from toolshed?
by Peter Briggs 15 Dec '16

15 Dec '16

Dear developers It looks like some updates to 'devteam'-maintained toolshed tools have dropped support for tool_dependencies.xml (e.g. fastqc, freebayes), so their dependencies cannot be automatically installed from the toolshed. Is this correct? I assume that it's part of the move to enforcing conda dependency resolution. Best wishes Peter -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 3