February 2016 - galaxy-dev - lists.galaxyproject.org

external_chown_script.py and the local job runner
by Peter Cock 15 Mar '16

15 Mar '16

On Mon, Jan 25, 2016 at 11:33 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hello all, > > We're currently looking at changing our Galaxy setup to link user accounts > with Linux user accounts for better cluster integration (running jobs as the > actual user on SGE). As part of this, we've tried setting up a fresh > installation on a new VM which has thrown up some issues. > > [snip] We eventually got Galaxy to talk to SGE and submit jobs successfully as the individual user's Linux account, with external_chown_script.py being called to handle ownership of the files. However, we would like to have some jobs (like the upload tool) configured to run on the Galaxy server itself - and the easiest way to do that is via the local job runner. We tried both the "upload1" and "Convert characters1" tools. These jobs would start and seem to run, but then fail with a file permission error (I don't have a stack trace to hand). From watching the Galaxy terminal output from run.sh we could see external_chown_script.py being called. As a test, when we disabled the external_chown_script setting in config/galaxy.ini then the local jobs would work. When using the local job runner, does Galaxy run the child process as the Galaxy user (my guess - no chown needed), or as the job owner's Linux account (calling chown would be needed)? Thanks, Peter

2 5

Scaling tools: data available?
by Peter van Heusden 14 Mar '16

14 Mar '16

Hi there We're researching the resource (CPU time and memory) requirements for RNA STAR at the moment. Specifically, we'd like to build up a database of input size to resource usage so that we can use this to feed a dynamic destination mapper (currently our STAR configuration uses a thumbsuck for RAM and CPU requirements that is not optimal but works in many cases). Has anyone collected these metrics before? And more generally, is anyone collecting metrics for a range of bioinformatics tools? A database like this could be very useful! Thanks! Peter

2 3

error loading files into galaxy
by Hakeem Almabrazi 04 Mar '16

04 Mar '16

Hi, I started getting the following error whenever I try to load a file into Galaxy local. Traceback (most recent call last): File "/usr/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 8, in <module> from galaxy import eggs ImportError: cannot import name eggs I will appreciate if someone can tell me what could happen to cause such issue and how to resolve it. Regards,

2 1

nginx data upload request never reaches /api/tools
by Preussner, Jens 03 Mar '16

03 Mar '16

Hi all, I set up the data upload using nginx as described in the wiki (https://wiki.galaxyproject.org/Admin/Config/nginxProxy, using the subdirectory /galaxy). The uploaded file also appears inside the uploade_store folder on the galaxy server, but galaxy does not add the file. Investigating the log files, I found the corresponding lines in the nginx.error.log: # Post request is received 2016/02/29 12:55:37 [notice] 13156#0: *4 "^(GET|HEAD|POST)$" matches "POST", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: http://172.16.12.122/galaxy/ # Upload starts 2016/02/29 12:55:37 [info] 13156#0: *4 started uploading file "boxes.txt" to "/mnt/idrive1/database/tmp/upload_store/0020132707" (field "files_0|file_data", content type "text/plain"), client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: http://172.16.12.122/galaxy/ # Upload done, the file is present in the given path 2016/02/29 12:55:37 [info] 13156#0: *4 finished uploading file "boxes.txt" to "/mnt/idrive1/database/tmp/upload_store/0020132707", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" # Redirection starts? 2016/02/29 12:55:37 [notice] 13156#0: *4 "^(GET|HEAD|POST)$" matches "POST", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" 2016/02/29 12:55:37 [notice] 13156#0: *4 "nginx_redir=([^&]+)" does not match "", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" 2016/02/29 12:55:37 [notice] 13156#0: *4 "" matches "/_upload_done", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" # Redirection should be done here 2016/02/29 12:55:37 [notice] 13156#0: *4 rewritten data: "/galaxy/api/tools", args: "", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" # ModSecurity seems to block the redirect? 2016/02/29 12:55:37 [info] 13156#0: *4 ModSecurity: invalid "Content-Length" header, client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" 2016/02/29 12:55:37 [notice] 13156#0: *4 "^(GET|HEAD|POST)$" matches "GET", client: 172.16.17.17, server: , request: "POST /_upload HTTP/1.1", host: "172.16.12.122", referrer: "http://172.16.12.122/galaxy/" A ModSecurityEnabled off; within the /_upload and /_upload_done of the nginx config did not help. Any idea why the Content-Length header causes problems and is blocked by modSec? Any workaround for ModSec to accept everything that goes from internal to internal? Thanks for your help! Best, Jens

2 2

Galaxy on Azure
by Alex Lenail 02 Mar '16

02 Mar '16

Hello, My name is Alex LeNail and I'm a researcher at MIT. I'm looking to set up a Galaxy server, and our lab has a great deal of Microsoft Azure credits, so we were hoping to deploy Galaxy on there. Where should I start if I wanted to set that up? I haven't really found any resources online with any description of how that might be accomplished. Thanks, --Alex

2 2

Dependencies for sudo scripts/drmaa_external_runner.py ?
by Peter Cock 02 Mar '16

02 Mar '16

Hello all, QUESTION: When submitting jobs to the cluster as the real user, how should sudo scripts/drmaa_external_runner.py be told which Python to use, and how would it activate the venv if needed for the DRMAA dependency? BACKGROUND: We're currently trying Galaxy out on a new CentOS 6 VM, with matching CentOS 6 cluster, where jobs are submitted to SGE via DRMAA and run as the Linux user rather than a generic Galaxy Linux user account. This is documented on the wiki here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster This all seemed to be working under Galaxy v15.10 (using eggs), but we're now targeting the recently released Galaxy v16.10 (using wheels) instead and have run into problems. https://github.com/galaxyproject/galaxy/issues/1596 Because Galaxy is deprecating support for Python 2.6 (the default bundled with CentOS 6), we're now using a local copy of Python 2.7 (compiled from source) on a shared mount. This mismatch seems to be the root cause of the problem I will now describe. During job submission to SGE, Galaxy will attempt to run a command like this: $ sudo scripts/drmaa_external_runner.py 1005 /mnt/shared/galaxy/galaxy-dist/database/sge/132.jt_json >From the terminal output from ./run.sh we'd see: RuntimeError: External_runjob failed (exit code 1) Child process reported error: Traceback (most recent call last): File "/mnt/shared/galaxy/galaxy-dist/scripts/drmaa_external_runner.py", line 15, in <module> import drmaa ImportError: No module named drmaa Although a drmaa wheel was installed within the Python 2.7 virtual environment under ~/galaxy-dist/.venv Galaxy makes no attempt to activate the venv for scripts/drmaa_external_runner.py We then installed DRMAA under our local copy of Python 2.7, and realised sudo scripts/drmaa_external_runner.py was not even using this copy of Python. Changing the hash bang line was a crude way to solve that (see below). This in turn lead to finding $DRMAA_LIBRARY_PATH and $SGE_ROOT were not set in the sudo environment. Again, you can hack around this by modifying scripts/drmaa_external_runner.py (see below). In our case, I suspect the least invasive change would be to install the DRMAA libraries under the system provided Python 2.6, and let sudo scripts/drmaa_external_runner.py execute that way. We still need to solve why sudo scripts/drmaa_external_runner.py does not see $DRMAA_LIBRARY_PATH and $SGE_ROOT but we have some clues to follow up on that: http://stackoverflow.com/questions/257616/sudo-changes-path-why Peter P.S. See also https://twitter.com/pjacock/status/704335582651162624 -- Here's our workaround diff - lots of hard coded strings, not portable at all but it worked for testing/debugging: $ git diff scripts/drmaa_external_runner.py diff --git a/scripts/drmaa_external_runner.py b/scripts/drmaa_external_runner.py index a1474fe..61d2383 100755 --- a/scripts/drmaa_external_runner.py +++ b/scripts/drmaa_external_runner.py @@ -1,5 +1,6 @@ +#!/mnt/shared/galaxy/apps/python/2.7.11/bin/python +#Was #!/usr/bin/env python - """ Submit a DRMAA job given a user id and a job template file (in JSON format) defining any or all of the following: args, remoteCommand, outputPath, @@ -12,8 +13,15 @@ import os import pwd import sys +# Hack +# print "$DRMAA_LIBRARY_PATH is %s" % os.environ.get('DRMAA_LIBRARY_PATH') +# print "$SGE_ROOT is %s" % os.environ.get('SGE_ROOT') +os.environ['DRMAA_LIBRARY_PATH'] = '/mnt/sge/lib/lx-amd64/libdrmaa.so' +os.environ['SGE_ROOT'] = '/mnt/sge' + import drmaa DRMAA_jobTemplate_attributes = [ 'args', 'remoteCommand', 'outputPath', 'errorPath', 'nativeSpecification', 'workingDirectory', 'jobName', 'email', 'project' ]

2 4

Python 2.6 Support
by Dannon Baker 01 Mar '16

01 Mar '16

We're currently discussing when we should stop supporting Python2.6 and it'd be useful to know how much this might impact local Galaxy administrators. Does anyone on the list have strong feelings or concerns about this and want to weigh in? Here's the issue on GitHub for more information and discussion: https://github.com/galaxyproject/galaxy/issues/1596 Thanks! Dannon

3 2

tmp directory
by Raphenya, Amogelang 01 Mar '16

01 Mar '16

4 4

Downgrade tool upload tool
by Makis Ladoukakis 01 Mar '16

01 Mar '16

Hello everyone, I was wondering whether it's possible to downgrade the upload tool to the previous version(1.1.4). I copied the upload.py and upload.xml scripts from a previous galaxy version to the tools/data_source/ directory and reloaded the tool as admin but it hasn't changed even when I restarted the server. Do I have to change something else in addition? Thank you, Makis

4 7

Lost prostgresql database
by Raphenya, Amogelang 29 Feb '16

29 Feb '16

3 3