Thanks very much Nate.
I do have a question while I have you.
As recommended, we are trying to off-load all work from Galaxy that we can.
Uploads and downloads going through nginx, most jobs running on the cluster.
There is a catch however. We'd like the URL based uploads to get off-loaded to the cluster. However, for importing our NGS data, we are using the mechanism built in to used the shared libraries.
We are doing this because the NGS data generated by our Sequencing Core, is sitting on a share that is visible to our Galaxy server, but not visible to our cluster. While the directory where the galaxy data is stored *is* visible to our cluster. What this means is that to import this data, it has to be a "local" import, not run on the cluster.
Is there anyway to separate the shared library imports from "upload1", so that these can be local, but the URL based uploads can go through our cluster?
Thanks,
Dave
On 10/21/10 3:26 PM, "Nate Coraor" nate@bx.psu.edu wrote:
Dave Walton wrote:
The problem has been solved.
Hooray!
I'm going to own-up to misunderstanding what "file uploads" the proxy server could off load.
I should have been clear in my first email wrt what kind of a file upload we were trying. We were doing an upload by URL.
Ah, that could be explained in the documentation. I'll try and clear this up.
If you have any other questions or issues please let us know.
Thanks, --nate
It turns out that the file upload from the user's desktop was always working (going through the proxy).
I've reconfigured our upload1 to go through our cluster and now everything is working great.
Thanks for your time.
Dave
On 10/21/10 2:32 PM, "Nate Coraor" nate@bx.psu.edu wrote:
Dave Walton wrote:
Cut and paste from a view source of my upload form:
<form id="tool_form" name="tool_form" action="/_upload?nginx_redir=/tool_runner/index" enctype="multipart/form-data" target="galaxy_main" method="post">
When I point my browser at /_upload on my galaxy server I get a: 405 Not allowed Nginx/0.8.53
Could you check the nginx error log for anything interesting?
Thanks, --nate
Dave
On 10/21/10 11:45 AM, "Nate Coraor" nate@bx.psu.edu wrote:
Hi Dave,
Your config looks good. Can you verify that the generated upload form's action is to '/_upload', e.g.:
<form id="tool_form" name="tool_form" action="/_upload?nginx_redir=/tool_runner/index" enctype="multipart/form-data" target="galaxy_main" method="post">
Can you access /_upload in your browser?
--nate
Dave Walton wrote:
Good Morning,
I have a galaxy instance at our institution, and we are trying to make the file upload with nginx work.
It appears that when I do a file download, it is getting handled by nginx (running top I see an nginx process appear, do work and disappear when the file is done downloading). However, when I try to do an upload, I see additional python processes appear in top, and I see the upload.py script go by in my runner0.log file, but I don't see an nginx process appear in top. Also, when I go to run several large uploads in a row, and then attempt to run some other job (like a sort) it blocks while waiting for the files to upload. This leads me to believe the upload is running in galaxy, not nginx.
I am also running "watch ls -ltr upload_store" from galaxy-dist/database/tmp while the upload is occurring and I never see anything written to the upload temp directory.
We are running galaxy on a 4 cpu server, which is running SUSE Linux Enterprise Server 11 (x86_64) for an os.
I have galaxy configured to run 3 webapps, and 1 job runner.
We have nginx installed and built with the nginx_upload_module (detail below)
jaxgalaxydev01:~ # /usr/local/nginx/sbin/nginx -V
nginx version: nginx/0.8.53 built by gcc 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) configure arguments: --prefix=/usr/local/nginx-0.8.53 --add-module=/usr/local/src/nginx_upload_module-2.2.0
In both my universe_wsgi.webapp.ini and universe_wsgi.runner.ini I have the following lines defined in my [app:main] section:
# Directive for using nginx to do file downloads nginx_x_accel_redirect_base = /_x_accel_redirect
# Directives to allow nginx to deal with file uploads nginx_upload_store = database/tmp/upload_store nginx_upload_path = /_upload
Below is the entire contents of my nginx.conf file:
worker_processes 1; events { worker_connections 1024; }
user galaxy; http { include mime.types; default_type application/octet-stream;
# Added for compression and caching. According to Galaxy wiki page: # http://bitbucket.org/galaxy/galaxy-central/wiki/Config/nginxProxy # This will decrease downlod and page load times for clients gzip on; gzip_http_version 1.1; gzip_vary on; gzip_comp_level 4; gzip_proxied any; gzip_types text/plain text/css application/x-javascript text/xmlapplication/xml text/javascript application/json; gzip_buffers 16 8k; gzip_disable "MSIE [1-6].(?!.*SV1)";
sendfile on; keepalive_timeout 65; # Added to support proxying to galaxy server upstream galaxy_app { server localhost:8080; server localhost:8081; server localhost:8082; } server { listen 80; server_name localhost;# Line added for galaxy support client_max_body_size 10G;
# Added to allow nginx to handle file uploads location /_upload { upload_store /galaxydata/galaxy-setup/galaxy-dist/database/tmp/upload_store; upload_pass_form_field ""; upload_set_form_field "__${upload_field_name}__is_composite" "true"; upload_set_form_field "__${upload_field_name}__keys" "name path"; upload_set_form_field "${upload_field_name}_name" "$upload_file_name"; upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path"; upload_pass_args on; upload_pass /_upload_done; } location /_upload_done { set $dst /tool_runner/index; if ($args ~ nginx_redir=([^&]+)) { set $dst $1; } rewrite "" $dst; }
# added to allow nginx to handle file downloads location /_x_accel_redirect/ { internal; alias /; }# The following series of "locations" are for off-loading static # content from the galaxy server. location /static { alias /galaxydata/galaxy-setup/galaxy-dist/static; # This will decrease downlod and page load times for clients expires 24h; } location /static/style { alias /galaxydata/galaxy-setup/galaxy-dist/static/june_2007_style/blue; } location /static/scripts { alias /galaxydata/galaxy-setup/galaxy-dist/static/scripts/packed; # This will decrease downlod and page load times for clients expires 24h; } location /favicon.ico { alias /galaxydata/galaxy-setup/galaxy-dist/static/favicon.ico; } location /robots.txt { alias /galaxydata/galaxy-setup/galaxy-dist/static/robots.txt; } # End of static content
location / { proxy_pass http://galaxy_app; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-For$proxy_add_x_forwarded_for; }
# redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } }}
I have restarted both my galaxy server and my nginx server to confirm that the configurations above have been picked up.
I would be interested in:
- any other suggestions to confirm where the upload is occurring
(assuming I'm mistaken about it running in galaxy). 2) any insight as to how I might have my server(s) misconfigured, in order to get upload working in nginx.
Any assistance or suggestions you could provide would be greatly appreciated.
Thanks,
Dave
galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev