FYI: Here are the entries in paster.log. If you notice no job_handlers are being sent. I performed tests on both nginx reverse proxy and direct localhost:8080 (limited to our private network on campus)

128.200.34.227 - - [11/Apr/2013:13:24:40 -0700] "GET /tool_runner?tool_id=upload1 HTTP/1.0" 200 - "http://ghtf-hpc.oit.uci.edu/root/tool_menu" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10"

galaxy.tools.actions.upload_common INFO 2013-04-11 13:24:44,173 tool upload1 created job id 8

128.200.34.227 - - [11/Apr/2013:13:24:43 -0700] "POST /tool_runner/index HTTP/1.0" 200 - "http://ghtf-hpc.oit.uci.edu/tool_runner?tool_id=upload1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10"

128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] "GET /history HTTP/1.0" 200 - "http://ghtf-hpc.oit.uci.edu/tool_runner/index" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10"

128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] "GET /api/histories/56959efe847bb360 HTTP/1.0" 200 - "http://ghtf-hpc.oit.uci.edu/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10"

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebrenne@uci.edu

On Thu, Apr 11, 2013 at 12:39 PM, Adam Brenner <aebrenne@uci.edu> wrote:

Howdy Galaxy Admins,

I am installing Galaxy on our HPC cluster and running into a strange issue that I can no longer debug further. I am running the latest galaxy build as of Tuesday April 9th.

    hg clone https://bitbucket.org/galaxy/galaxy-dist/

    hg update stable

When a user tries to upload a file via the web interface, nginx reverse proxy setup as described here http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy, the upload stays in a state of "Job is waiting to run."

The job stays like this for both FTP uploads, and uploads via browser. The dataset is a 319 bytes (both FTP and upload via browser)

My universe_wsgi.ini variables are like so:

     .. under [app:main] ....

     new_file_path = /som/galaxy/tmp

     file_path = /som/galaxy/datasets/files

     track_jobs_in_database = True

     job_manager = manager

     job_handlers = handler0,handler1

     ... before [app:main] in [server:main] ....

     [server:main]

     use = egg:Paste#http
     port = 8080
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5

     [server:manager]

     use = egg:Paste#http
     port = 8079
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5
     

     [server:handler0]
     use = egg:Paste#http
     port = 8090
     host = 127.0.0.1
     use_threadpool = true
     threadpool_workers = 5
     

     [server:handler1]
     use = egg:Paste#http
     port = 8091
     host = 127.0.0.1
     use_threadpool = true

     threadpool_workers = 5

Galaxy, for now, is running under the root user and is installed in /data/apps/galaxy (NFS mount) and the tmp files are stored on a GlusterFS system under /som/galaxy/. 

I have tried removing the extra job managers and handlers so I just have server:main and this still continues to happen. The paster.log files (with DEBUG = True) does not show any errors from python or issues writing to file. All POST and GET responses are recorded and nothing else.

What I find strange is that if I look in new_file_path I do see my dataset as 

    upload_file_data_Zx_1rK

and cat'ing the file shows the correct contents. The items listed in my file_path (the dataset_*.dat) files are empty/blank..but are created.

Any ideas as to why its failing? Is it nginx? I only have it redirect 8080, and none of the other job managers / handlers...is that an issue? Do you think its permissions on the my file_path or new_file_path directories? 

A very stumped Adam.....

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebrenne@uci.edu