[galaxy-dev] Local Galaxy Install - Multiple Job Handlers - Uploads waiting to run endlessly

11 Apr 2013

      Howdy Galaxy Admins,

I am installing Galaxy on our HPC cluster and running into a strange issue
that I can no longer debug further. I am running the latest galaxy build as
of Tuesday April 9th.
    hg clone https://bitbucket.org/galaxy/galaxy-dist/
    hg update stable

When a user tries to upload a file via the web interface, nginx reverse
proxy setup as described here
http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy, the
upload stays in a state of "Job is waiting to run."

The job stays like this for both FTP uploads, and uploads via browser. The
dataset is a 319 bytes (both FTP and upload via browser)

My universe_wsgi.ini variables are like so:
     .. under [app:main] ....
     new_file_path = /som/galaxy/tmp
     file_path = /som/galaxy/datasets/files
     track_jobs_in_database = True
     job_manager = manager
     job_handlers = handler0,handler1
     ... before [app:main] in [server:main] ....

     [server:main]
     use = egg:Paste#http
     port = 8080
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5

     [server:manager]
     use = egg:Paste#http
     port = 8079
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5

     [server:handler0]
     use = egg:Paste#http
     port = 8090
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5

     [server:handler1]
     use = egg:Paste#http
     port = 8091
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5

Galaxy, for now, is running under the root user and is installed in
/data/apps/galaxy (NFS mount) and the tmp files are stored on a GlusterFS
system under /som/galaxy/.

I have tried removing the extra job managers and handlers so I just have
server:main and this still continues to happen. The paster.log files (with
DEBUG = True) does not show any errors from python or issues writing to
file. All POST and GET responses are recorded and nothing else.

What I find strange is that if I look in new_file_path I do see my dataset
as
    upload_file_data_Zx_1rK
and cat'ing the file shows the correct contents. The items listed in
my file_path (the dataset_*.dat) files are empty/blank..but are created.

Any ideas as to why its failing? Is it nginx? I only have it redirect 8080,
and none of the other job managers / handlers...is that an issue? Do you
think its permissions on the my file_path or new_file_path directories?

A very stumped Adam.....

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebrenne@uci.edu