- galaxy-dev - lists.galaxyproject.org

Dump SQlite to mySQL
by evan clark 03 Jan '17

03 Jan '17

Is there a script someone has designed to properly dump the galaxy sqlite database to a mySQL database?

1 0

psycopg2 module and SSL connection to postgresql
by Floreline TOUCHARD 03 Jan '17

03 Jan '17

Hi Galaxy Community, I have the same problem than Nikolay few years ago ( http://dev.list.galaxyproject.org/connecting-to-a-DB-on-a-different-host-td…) : I would like to connect my Galaxy instance to my postgresql database using SSL encryption and Auth method md5. I have no problems connecting to the DB via the command line with SSL encryption. When setting the galaxy.ini file with postgresql database connection to "postgresql://<USERNAME>:<PWD>@<HOST>:5432/<DB_NAME>?sslmode=require", I get the following message : "OperationalError: (psycopg2.OperationalError) sslmode value "require" invalid when SSL support is not compiled in" Regarding the former post I thought that it might come from the module psycopg2's compilation. I followed the steps of the post but encounter a different error. What I did: 1. Downloaded a tar.gz of psycopg2 2.5.1 (I saw the needed version in the file eggs.ini) to <install dir>/galaxy/ and unpacked it there 2. cd into psycopg2, modified the setup.py according to the instructions (added the line containing scramble_lib) 3. export the path to pg_config (export PATH=<path/to/postgresql>/bin:$PATH) 4. added the file ez_setup.py (downloaded from https://bootstrap.pypa.io/ez_setup.py) to <install dir>/galaxy/scripts/scramble/lib/, the same dir where scramble_lib.py is located 5. ran PYTHONPATH=<install dir>/galaxy/scripts/scramble/lib python setup.py egg_info --tag-build=_9.4.4_static bdist_egg And get the following error : " Traceback (most recent call last): File "setup.py", line 57, in <module> from scramble_lib import * File "<install dir>/galaxy/scripts/scramble/lib/scramble_lib.py", line 150, in <module> use_setuptools( download_delay=8, to_dir=os.path.dirname( __file__ ) ) File "<install dir>/galaxy/scripts/scramble/lib/ez_setup.py", line 160, in use_setuptools version = _resolve_version(version) File "<install dir>/galaxy/scripts/scramble/lib/ez_setup.py", line 368, in _resolve_version resp = urlopen(meta_url) File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open raise URLError(err) urllib2.URLError: <urlopen error [Errno -2] Name or service not known> " Could it be because I have no internet access ? What module should I install to enable SSL encryption ? Additional information : - Postgresql version 9.4.4 - Galaxy version 15.10 - I use virtualenv for python environment Thanks in advance, Floreline

2 1

GIE security implications
by Tamir,Ido 03 Jan '17

03 Jan '17

Hi, what are the security implications of GIE? I saw the overview on the GIE page, but its not clear to me how dangerous they are. "They have complex interactions with numerous services, you’ll need to be a fairly competent SysAdmin to debug all of the possible problems that can occur during deployment” Is it possible to describe what the docker container has access to and what could possibly go wrong? thank you very much, ido

2 1

Job launching slowed down by conda auto-install
by Tiago Macedo 21 Dec '16

21 Dec '16

Hello all, I recently realized that setting conda_auto_install to true in galaxy.ini has a noticeable effect in slowing down the start of job execution. On the first job run after activating conda_auto_install the job takes a lot longer as conda is actually installing packages, which is expected. On the following runs, even if there are no packages to be installed jobs still take longer to start. What's more surprising is that after deactivating conda_auto_install jobs still take longer to run. My particular test case was a simple small (12MB) file upload. It consistently went from 15s to over 30s after activating conda_auto_install and remained at that even after deactivating it. We tested running a few tools which presented a noticeable and irreversible delay after activation of conda_auto_install, but I didn't measure these delays. I also tested this on fresh installs of galaxy 16.07 and 16.10 with very similar results. Is this supposed to happen? Is there a way around it besides never activating conda_auto_install? Thanks in advance, Tiago

2 1

tmp file under database/tmp get regenerated after restarting Galaxy
by Mark Xiao 20 Dec '16

20 Dec '16

Dear Galaxy Developers, I am using Galaxy version 15.10 on my server, load-balancing is enabled. I found a problem when running the jobs. When there are running jobs and if I restart the Galaxy process at this time, the job will fail at the end of the job with the error message such as: IOError: [Errno 2] No such file or directory: '/galaxy/database/tmp/tmpleXN4b' That tmp file doesn't exist, I guess it is because when Galaxy process get restarted, the tmp file get regenerated, so the job cannot find the correct tmp file, then it was failed. Do you think this is a known issue, and is there a fix for this? Thank you in advance! Mark

1 1

Tools pick up Python modules from Galaxy core rather than installed tool dependencies
by Peter Briggs 20 Dec '16

20 Dec '16

Dear devs I was wondering if anyone had any suggestions as to the cause of a peculiar error I'm seeing when running the latest versions of the devteam "join" and "fastq_paired_end_joiner". The tools appear to be picking up Python modules from the Galaxy library, rather than from the installed dependencies. The PYTHONPATH appears to be correct (i.e. tool dependency appears before Galaxy library path) but sys.path shows the dependency's Python egg appearing after the Galaxy library path. E.g. for the paired-end joiner: PYTHONPATH=/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python:/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib sys.path = ['', '/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python', '/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib', '/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python/galaxy_sequence_utils-1.0.1-py2.7.egg', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python27.zip', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/plat-linux2', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-tk', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-old', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-dynload', '/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7', '/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/plat-linux2', '/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/lib-tk', '/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/site-packages'] Error: Traceback (most recent call last): File "/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py", line 158, in <module> main() File "/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py", line 136, in main joiner = fq.fastqJoiner( input1_type, paste=paste ) TypeError: __init__() got an unexpected keyword argument 'paste' I know it's a long shot but can anyone suggest where to start looking at where this is going wrong? Thanks, Peter Ps I'm using Galaxy v16.07 and the problem occurs on our production platform but not in a local Galaxy install on my desktop. -- Peter Briggs peter.briggs(a)manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482

2 3

Connection to mariaDB generates tables, but does not migrate information
by evan clark 20 Dec '16

20 Dec '16

I connected our galaxy instance to mariadb with no connection issues and it began to migrate the data, I noticed however when galaxy was finally started the tables were generated, but no data was migrated. It completed the migration without error so I am not sure if there is a setting I missed in mariadb.

3 2

Installation of packages failing
by James Mullan 19 Dec '16

19 Dec '16

Hello, I am trying to install packages with dependencies from Galaxy Main tool shed. This worked Thursday and since Friday it no longer works for me. It seems this url is no longer reachable. Examples of what I see in the log during download are shown below. Any help would be much appreciated. package_bowtie_0_12_7 Error downloading from URL http://depot.galaxyproject.org/package/linux/x86_64/bowtie/bowtie-0.12.7-li… package_samtools_0_1_19 Exception: Error downloading from URL https://depot.galaxyproject.org/software/samtools/samtools_0.1.19_src_all.t… Regards James J P Mullan

4 6

Speedup Galaxy
by Jochen Bick 19 Dec '16

19 Dec '16

We are running a Galaxy instance at our institute (version 15.10). Since some month the speed/reaction time of the interface drastically decrease. Sometimes there is a lack of more than a minute loading a single tool. I restarted galaxy without any improvements. We are working with up to 3 people on this instance that should not be the problem. Does anyone have any idea? Disk-space? uptime of our server? big full database? ?? Cheers Jochen

3 5

Galaxy Security Vulnerability - Arbitrary code execution
by Nate Coraor 19 Dec '16

19 Dec '16

*DESCRIPTION* A security vulnerability was recently discovered by David Wyde in two Galaxy tools that are provided with and enabled by default in the Galaxy server. These vulnerabilities allow malicious actors to execute arbitrary code as the user running the Galaxy server. The vulnerable tools are "Filter GFF data by attribute" and "Filter GFF data by feature count", two tools which share code with each other and the more general "Filter data on any column using simple expressions" tool. The latter was fixed in a previous security disclosure but these GFF variants of the tool were missed when updating the Filter tool. These tools use the Python `eval` and `exec` functions and do not properly sanitize input to these functions. Additionally, an XSS vulnerability with the upload tool: An uploaded file's name was not properly sanitized, and so a specially crafted filename uploaded to the Galaxy server could be used as an XSS attack vector. This vulnerability has already been fixed in the 16.07 and 16.10 Galaxy releases. The Galaxy Committers would like to thank David for disclosing these vulnerabilities. If you discover security vulnerabilities in Galaxy, please report them to the Galaxy Committers at galaxy-committers(a)lists.galaxyproject.org. *AFFECTED VERSIONS* These arbitrary code execution vulnerability is present in all known releases of Galaxy using the default tool config or a tool config in which these tools are loaded. To determine if your Galaxy server is affected, check your tool_conf.xml for the following lines: <tool file="filters/gff/gff_filter_by_attribute.xml" /> <tool file="filters/gff/gff_filter_by_feature_count.xml" /> If you are unsure, you can also check to see if the tools have loaded by searching for them in your Galaxy server logs, e.g.: % grep gff_filter paster.log galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: gff_filter_by_attribute, version: 0.1 into tool panel.. galaxy.tools.toolbox.base DEBUG 2016-12-08 13:25:16,930 Loaded tool id: gff_filter_by_feature_count, version: 0.1 into tool panel.. *IMPACT* This vulnerability can be exploited to execute arbitrary code as the user that runs Galaxy jobs on any system where Galaxy runs those jobs (potentially a cluster). Such code can be used to provide a remote shell to an attacker, or read/write/delete files that the Galaxy user has appropriate permissions on. *SOLUTION* Multiple solutions exist for the arbitrary code execution vulnerability: 1. Update your Galaxy code. A fix for the issue has been applied to stable releases of Galaxy back to version 14.10. 2. Disable the affected tools. 3. Patch the affected tools. Patches to fix the vulnerabilities (depending on the version of Galaxy that you are running) can be found below. *INSTRUCTIONS* 1. To apply the fix, first identify your current Galaxy release version using the `git branch` command. If you are on a 'release_YY.MM' branch, you can update with: % git pull The process above can also be used to update to the 16.10 release if you are on the 'master' git branch. If you are on the 'master' branch and wish to remain on your current Galaxy major release, check the 'lib/galaxy/version.py' file to determine your major release version, then update to the appropriate branch: % git fetch origin % git checkout -b release_YY.MM origin/release_YY.MM % git pull 2. To disable the tools, locate and remove the following lines from your tool_conf.xml, then restart Galaxy: <tool file="filters/gff/gff_filter_by_attribute.xml" /> <tool file="filters/gff/gff_filter_by_feature_count.xml" /> 3. To manually patch the tools, use the appropriate patch file below: For Galaxy releases 16.01 to dev: https://depot.galaxyproject.org/patch/filter_security_16_01.patch For Galaxy releases 15.07 to 15.10: https://depot.galaxyproject.org/patch/filter_security_15_07.patch For Galaxy releases 14.10 to 15.05: https://depot.galaxyproject.org/patch/filter_security_14_10.patch To apply the patch, navigate to the root of your Galaxy directory, then execute (replacing <patch_url> with the url above appropriate for your current Galaxy release): % wget -O filter_security.patch <patch_url> or: % curl -o filter_security.patch <patch_url> and then test applying the patch: % patch -p1 --dry-run < filter_security.patch checking file tools/filters/gff/gff_filter_by_attribute.py checking file tools/filters/gff/gff_filter_by_attribute.xml checking file tools/filters/gff/gff_filter_by_feature_count.py checking file tools/filters/gff/gff_filter_by_feature_count.xml If this succeeds without error (applying successfully with fuzz/offset is fine), apply with: % patch -p1 < filter_security.patch patching file tools/filters/gff/gff_filter_by_attribute.py patching file tools/filters/gff/gff_filter_by_attribute.xml patching file tools/filters/gff/gff_filter_by_feature_count.py patching file tools/filters/gff/gff_filter_by_feature_count.xml In all cases, for the changes to take effect, *YOU MUST RESTART ALL GALAXY SERVER PROCESSES*. On behalf of the Galaxy Committers, --nate

4 3