Problem offloading upload with nginx - galaxy-dev

21 Oct 2010


      Good Morning,
I have a galaxy instance at our institution, and we are trying to make the file upload with nginx work.
It appears that when I do a file download, it is getting handled by nginx (running top I see an nginx process appear, do work and disappear when the file is done downloading).  However, when I try to do an upload, I see additional python processes appear in top, and I see the upload.py script go by in my runner0.log file, but I don't see an nginx process appear in top.  Also, when I go to run several large uploads in a row, and then attempt to run some other job (like a sort) it blocks while waiting for the files to upload.  This leads me to believe the upload is running in galaxy, not nginx.
I am also running "watch ls -ltr upload_store" from galaxy-dist/database/tmp while the upload is occurring and I never see anything written to the upload temp directory.
We are running galaxy on a 4 cpu server, which is running SUSE Linux Enterprise Server 11 (x86_64) for an os.
I have galaxy configured to run 3 webapps, and 1 job runner.
We have nginx installed and built with the nginx_upload_module  (detail below)
jaxgalaxydev01:~ # /usr/local/nginx/sbin/nginx -V
nginx version: nginx/0.8.53
built by gcc 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux)
configure arguments: --prefix=/usr/local/nginx-0.8.53 --add-module=/usr/local/src/nginx_upload_module-2.2.0
In both my universe_wsgi.webapp.ini and universe_wsgi.runner.ini  I have the following lines defined in my [app:main]  section:
# Directive for using nginx to do file downloads
nginx_x_accel_redirect_base = /_x_accel_redirect
# Directives to allow nginx to deal with file uploads
nginx_upload_store = database/tmp/upload_store
nginx_upload_path = /_upload
Below is the entire contents of my nginx.conf file:
worker_processes  1;
events {
    worker_connections  1024;
}
user galaxy;
http {
    include       mime.types;
    default_type  application/octet-stream;
#  Added for compression and caching.  According to Galaxy wiki page:
    #  http://bitbucket.org/galaxy/galaxy-central/wiki/Config/nginxProxy
    #  This will decrease downlod and page load times for clients
    gzip  on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_proxied any;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6].(?!.*SV1)";
sendfile        on;
keepalive_timeout  65;
#  Added to support proxying to galaxy server
    upstream galaxy_app {
        server localhost:8080;
        server localhost:8081;
        server localhost:8082;
    }
server {
        listen       80;
        server_name  localhost;
# Line added for galaxy support
client_max_body_size 10G;
#  Added to allow nginx to handle file uploads
        location /_upload {
            upload_store /galaxydata/galaxy-setup/galaxy-dist/database/tmp/upload_store;
            upload_pass_form_field "";
            upload_set_form_field "__${upload_field_name}__is_composite" "true";
            upload_set_form_field "__${upload_field_name}__keys" "name path";
            upload_set_form_field "${upload_field_name}_name" "$upload_file_name";
            upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path";
            upload_pass_args on;
            upload_pass /_upload_done;
        }
        location /_upload_done {
            set $dst /tool_runner/index;
            if ($args ~ nginx_redir=([^&]+)) {
                set $dst $1;
            }
            rewrite "" $dst;
        }
#  added to allow nginx to handle file downloads
        location /_x_accel_redirect/ {
            internal;
            alias /;
        }
#  The following series of "locations" are for off-loading static
#  content from the galaxy server.
        location /static {
            alias /galaxydata/galaxy-setup/galaxy-dist/static;
            #  This will decrease downlod and page load times for clients
            expires 24h;
        }
        location /static/style {
            alias /galaxydata/galaxy-setup/galaxy-dist/static/june_2007_style/blue;
        }
        location /static/scripts {
            alias /galaxydata/galaxy-setup/galaxy-dist/static/scripts/packed;
            #  This will decrease downlod and page load times for clients
            expires 24h;
        }
        location /favicon.ico {
            alias /galaxydata/galaxy-setup/galaxy-dist/static/favicon.ico;
        }
        location /robots.txt {
            alias /galaxydata/galaxy-setup/galaxy-dist/static/robots.txt;
        }
        #  End of static content
location / {
            proxy_pass   http://galaxy_app;
            proxy_set_header   X-Forwarded-Host $host;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        }
# redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}
I have restarted both my galaxy server and my nginx server to confirm that the configurations above have been picked up.
I would be interested in:
1) any other suggestions to confirm where the upload is occurring (assuming I'm mistaken about it running in galaxy).
2) any insight as to how I might have my server(s) misconfigured, in order to get upload working in nginx.
Any assistance or suggestions you could provide would be greatly appreciated.
Thanks,
Dave