Uploading Files on Local Cluster Install
I just finished setting up a local install on our cluster but I'm having trouble uploading files. I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang. Here's the information from the log file: paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow? Let me know if I can provide more info. Thanks, Greg
Hi guys, I wanted to make sure this made it through to the dev list. I'm still stuck on this :-( Thanks, Greg On Wed, Nov 28, 2012 at 1:23 PM, greg <margeemail@gmail.com> wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg
I apologize for potentially sending this 2 (or 3?) times. I just realized I might have sent it to the wrong address? galaxy-dev@lists.bx.psu.edu != galaxy-dev@bx.psu.edu? Thanks again, Greg On Wed, Nov 28, 2012 at 1:23 PM, greg <margeemail@gmail.com> wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg
On Nov 28, 2012, at 1:23 PM, greg wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Hi Greg, Could you try the following: Open tools/data_source/upload.xml In the following line, set 'ajax-upload="false"': <param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator)."> Then restart Galaxy and retry the upload and let us know if it succeeds. --nate
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious) -Greg On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On Nov 29, 2012, at 10:54 AM, greg wrote:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious)
It's due to a bug that we've done quite a bit of work to investigate, but haven't yet found the cause. It seems to have cropped up more frequently in the past few weeks, so we'll be working on it in depth in the next few days. --nate
-Greg
On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Follow up question. What's the deal with the 2GB limit it mentions? I think it would be pretty convienent to allow larger uploads. We're on a fast local network so even 10 or 20 GB uploads pretty fast. Can I increase the limit in Apache? Which browsers are limited? This is an internal install so I can tell users to only use certain browsers for uploads. -Greg On Thu, Nov 29, 2012 at 11:01 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 29, 2012, at 10:54 AM, greg wrote:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious)
It's due to a bug that we've done quite a bit of work to investigate, but haven't yet found the cause. It seems to have cropped up more frequently in the past few weeks, so we'll be working on it in depth in the next few days.
--nate
-Greg
On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
This says Google Chrome can support uploads larger than 4 GB. http://www.motobit.com/help/scptutl/pa98.htm Maybe I just need to configure Apache? I'll let you guys know what I find. -Greg On Thu, Nov 29, 2012 at 11:22 AM, greg <margeemail@gmail.com> wrote:
Follow up question. What's the deal with the 2GB limit it mentions?
I think it would be pretty convienent to allow larger uploads. We're on a fast local network so even 10 or 20 GB uploads pretty fast.
Can I increase the limit in Apache? Which browsers are limited? This is an internal install so I can tell users to only use certain browsers for uploads.
-Greg
On Thu, Nov 29, 2012 at 11:01 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 29, 2012, at 10:54 AM, greg wrote:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious)
It's due to a bug that we've done quite a bit of work to investigate, but haven't yet found the cause. It seems to have cropped up more frequently in the past few weeks, so we'll be working on it in depth in the next few days.
--nate
-Greg
On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Hmm, no luck on the large uploads. I tried using a recent version of Google Chrome. I added these settings to /etc/httpd/conf/httpd.conf Timeout 6000 LimitRequestBody 32147483647 And restarted apache. But when I try to upload a 3.7GB file I get this error in Galaxy: "Bad Gateway The proxy server received an invalid response from an upstream server." Now I do see this error in the Galaxy log, so I'm thinking I'm not hitting an Apache issue, and I'm not hitting a browser issue. Perhaps it's a paste issue? Exception happened during processing of request from ('127.0.0.1', 52128) Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1053, in process_request_in_thread self.finish_request(request, client_address) File "/usr/local/python27/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/local/python27/lib/python2.7/SocketServer.py", line 639, in __init__ self.handle() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 432, in handle BaseHTTPRequestHandler.handle(self) File "/usr/local/python27/lib/python2.7/BaseHTTPServer.py", line 337, in handle self.handle_one_request() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 427, in handle_one_request self.wsgi_execute() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 287, in wsgi_execute self.wsgi_start_response) File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/urlmap.py", line 202, in __call__ return app(environ, start_response) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/xforwardedhost.py", line 21, in __call__ return self.app( environ, start_response ) File "/misc/local/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.7.egg/paste/deploy/config.py", line 164, in __call__ app_iter = self.application(environ, start_response) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/translogger.py", line 68, in __call__ return self.application(environ, replacement_start_response) File "/misc/local/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py", line 226, in __call__ return self.respond(environ, start_response) File "/misc/local/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py", line 399, in respond if self.xmlhttp_key in req.params: File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 900, in params params = self.str_params File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 892, in str_params return NestedMultiDict(self.str_GET, self.str_POST) File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 818, in str_POST keep_blank_values=True) File "/usr/local/python27/lib/python2.7/cgi.py", line 508, in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File "/usr/local/python27/lib/python2.7/cgi.py", line 632, in read_multi environ, keep_blank_values, strict_parsing) File "/usr/local/python27/lib/python2.7/cgi.py", line 510, in __init__ self.read_single() File "/usr/local/python27/lib/python2.7/cgi.py", line 647, in read_single self.read_lines() File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py", line 268, in read_lines self.read_lines_to_outerboundary() File "/usr/local/python27/lib/python2.7/cgi.py", line 697, in read_lines_to_outerboundary line = self.fp.readline(1<<16) File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 467, in readline data = self.file.readline(self.length - self._consumed) File "/usr/local/python27/lib/python2.7/socket.py", line 412, in readline bline = buf.readline(size) OverflowError: signed integer is greater than maximum On Thu, Nov 29, 2012 at 11:26 AM, greg <margeemail@gmail.com> wrote:
This says Google Chrome can support uploads larger than 4 GB. http://www.motobit.com/help/scptutl/pa98.htm
Maybe I just need to configure Apache?
I'll let you guys know what I find.
-Greg
On Thu, Nov 29, 2012 at 11:22 AM, greg <margeemail@gmail.com> wrote:
Follow up question. What's the deal with the 2GB limit it mentions?
I think it would be pretty convienent to allow larger uploads. We're on a fast local network so even 10 or 20 GB uploads pretty fast.
Can I increase the limit in Apache? Which browsers are limited? This is an internal install so I can tell users to only use certain browsers for uploads.
-Greg
On Thu, Nov 29, 2012 at 11:01 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 29, 2012, at 10:54 AM, greg wrote:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious)
It's due to a bug that we've done quite a bit of work to investigate, but haven't yet found the cause. It seems to have cropped up more frequently in the past few weeks, so we'll be working on it in depth in the next few days.
--nate
-Greg
On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
I just finished setting up a local install on our cluster but I'm having trouble uploading files.
I was able to (successfully?) upload one 160MB fasta file. But any further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
Here's the information from the log file:
paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool upload1 created job id 4 galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 /home/galaxy/job_working_directory/000/4 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/4' paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) ave time N/A, max time 0.00sec, killed 0 workers
The first upload that worked was around 12:20 (above), and the second failed one was at 12:40. The log file doesn't seem to even mention it? Could that error have killed the server somehow?
Let me know if I can provide more info.
Thanks,
Greg ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
LimitRequestBody has a max value of 2147483647 (2GB). You probably want to set it to zero to disable the limit: http://httpd.apache.org/docs/current/mod/core.html#limitrequestbody Also, you may be running into memory limits on the server, especially if it's running a 32-bit OS (where each process is limited to about 2GB of memory): http://stackoverflow.com/questions/1922414/file-upload-limit-in-http Ilya On 11/29/12 12:57 PM, "greg" <margeemail@gmail.com> wrote:
Hmm, no luck on the large uploads.
I tried using a recent version of Google Chrome.
I added these settings to /etc/httpd/conf/httpd.conf
Timeout 6000 LimitRequestBody 32147483647
And restarted apache.
But when I try to upload a 3.7GB file I get this error in Galaxy:
"Bad Gateway
The proxy server received an invalid response from an upstream server."
Now I do see this error in the Galaxy log, so I'm thinking I'm not hitting an Apache issue, and I'm not hitting a browser issue. Perhaps it's a paste issue?
Exception happened during processing of request from ('127.0.0.1', 52128) Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver. py", line 1053, in process_request_in_thread self.finish_request(request, client_address) File "/usr/local/python27/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/local/python27/lib/python2.7/SocketServer.py", line 639, in __init__ self.handle() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver. py", line 432, in handle BaseHTTPRequestHandler.handle(self) File "/usr/local/python27/lib/python2.7/BaseHTTPServer.py", line 337, in handle self.handle_one_request() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver. py", line 427, in handle_one_request self.wsgi_execute() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver. py", line 287, in wsgi_execute self.wsgi_start_response) File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/urlmap.py", line 202, in __call__ return app(environ, start_response) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/xforwa rdedhost.py", line 21, in __call__ return self.app( environ, start_response ) File "/misc/local/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.7.egg/paste/dep loy/config.py", line 164, in __call__ app_iter = self.application(environ, start_response) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/transl ogger.py", line 68, in __call__ return self.application(environ, replacement_start_response) File "/misc/local/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/eval exception/middleware.py", line 226, in __call__ return self.respond(environ, start_response) File "/misc/local/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/eval exception/middleware.py", line 399, in respond if self.xmlhttp_key in req.params: File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 900, in params params = self.str_params File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 892, in str_params return NestedMultiDict(self.str_GET, self.str_POST) File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 818, in str_POST keep_blank_values=True) File "/usr/local/python27/lib/python2.7/cgi.py", line 508, in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File "/usr/local/python27/lib/python2.7/cgi.py", line 632, in read_multi environ, keep_blank_values, strict_parsing) File "/usr/local/python27/lib/python2.7/cgi.py", line 510, in __init__ self.read_single() File "/usr/local/python27/lib/python2.7/cgi.py", line 647, in read_single self.read_lines() File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py", line 268, in read_lines self.read_lines_to_outerboundary() File "/usr/local/python27/lib/python2.7/cgi.py", line 697, in read_lines_to_outerboundary line = self.fp.readline(1<<16) File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver. py", line 467, in readline data = self.file.readline(self.length - self._consumed) File "/usr/local/python27/lib/python2.7/socket.py", line 412, in readline bline = buf.readline(size) OverflowError: signed integer is greater than maximum
On Thu, Nov 29, 2012 at 11:26 AM, greg <margeemail@gmail.com> wrote:
This says Google Chrome can support uploads larger than 4 GB. http://www.motobit.com/help/scptutl/pa98.htm
Maybe I just need to configure Apache?
I'll let you guys know what I find.
-Greg
On Thu, Nov 29, 2012 at 11:22 AM, greg <margeemail@gmail.com> wrote:
Follow up question. What's the deal with the 2GB limit it mentions?
I think it would be pretty convienent to allow larger uploads. We're on a fast local network so even 10 or 20 GB uploads pretty fast.
Can I increase the limit in Apache? Which browsers are limited? This is an internal install so I can tell users to only use certain browsers for uploads.
-Greg
On Thu, Nov 29, 2012 at 11:01 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 29, 2012, at 10:54 AM, greg wrote:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious)
It's due to a bug that we've done quite a bit of work to investigate, but haven't yet found the cause. It seems to have cropped up more frequently in the past few weeks, so we'll be working on it in depth in the next few days.
--nate
-Greg
On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
> I just finished setting up a local install on our cluster but I'm > having trouble uploading files. > > I was able to (successfully?) upload one 160MB fasta file. But any > further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
> > Here's the information from the log file: > > paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 >starting) > ave time N/A, max time 0.00sec, killed 0 workers > galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 >tool > upload1 created job id 4 > galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched > galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued >as 3718641 > galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 > /home/galaxy/job_working_directory/000/4 delete error [Errno 39] > Directory not empty: '/home/galaxy/job_working_directory/000/4' > paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 >starting) > ave time N/A, max time 0.00sec, killed 0 workers > paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 >starting) > ave time N/A, max time 0.00sec, killed 0 workers > paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 >starting) > ave time N/A, max time 0.00sec, killed 0 workers > paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 >starting) > ave time N/A, max time 0.00sec, killed 0 workers > > > The first upload that worked was around 12:20 (above), and the >second > failed one was at 12:40. The log file doesn't seem to even mention > it? Could that error have killed the server somehow? > > Let me know if I can provide more info. > > Thanks, > > Greg > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
I just tried setting up galaxy to use xSendFile[1] but that didn't make any difference. What's odd is that I'm still getting the same error in the galaxy log. By using xsendfile shouldn't I now be avoiding using Galaxy to process the upload over a socket? Perhaps xsendfile isn't working? -Greg [1] http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy#Sending_files_usin... On Thu, Nov 29, 2012 at 12:57 PM, greg <margeemail@gmail.com> wrote:
Hmm, no luck on the large uploads.
I tried using a recent version of Google Chrome.
I added these settings to /etc/httpd/conf/httpd.conf
Timeout 6000 LimitRequestBody 32147483647
And restarted apache.
But when I try to upload a 3.7GB file I get this error in Galaxy:
"Bad Gateway
The proxy server received an invalid response from an upstream server."
Now I do see this error in the Galaxy log, so I'm thinking I'm not hitting an Apache issue, and I'm not hitting a browser issue. Perhaps it's a paste issue?
Exception happened during processing of request from ('127.0.0.1', 52128) Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 1053, in process_request_in_thread self.finish_request(request, client_address) File "/usr/local/python27/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/local/python27/lib/python2.7/SocketServer.py", line 639, in __init__ self.handle() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 432, in handle BaseHTTPRequestHandler.handle(self) File "/usr/local/python27/lib/python2.7/BaseHTTPServer.py", line 337, in handle self.handle_one_request() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 427, in handle_one_request self.wsgi_execute() File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 287, in wsgi_execute self.wsgi_start_response) File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/urlmap.py", line 202, in __call__ return app(environ, start_response) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/xforwardedhost.py", line 21, in __call__ return self.app( environ, start_response ) File "/misc/local/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.7.egg/paste/deploy/config.py", line 164, in __call__ app_iter = self.application(environ, start_response) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/translogger.py", line 68, in __call__ return self.application(environ, replacement_start_response) File "/misc/local/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py", line 226, in __call__ return self.respond(environ, start_response) File "/misc/local/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py", line 399, in respond if self.xmlhttp_key in req.params: File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 900, in params params = self.str_params File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 892, in str_params return NestedMultiDict(self.str_GET, self.str_POST) File "build/bdist.linux-x86_64/egg/webob/__init__.py", line 818, in str_POST keep_blank_values=True) File "/usr/local/python27/lib/python2.7/cgi.py", line 508, in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File "/usr/local/python27/lib/python2.7/cgi.py", line 632, in read_multi environ, keep_blank_values, strict_parsing) File "/usr/local/python27/lib/python2.7/cgi.py", line 510, in __init__ self.read_single() File "/usr/local/python27/lib/python2.7/cgi.py", line 647, in read_single self.read_lines() File "/misc/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py", line 268, in read_lines self.read_lines_to_outerboundary() File "/usr/local/python27/lib/python2.7/cgi.py", line 697, in read_lines_to_outerboundary line = self.fp.readline(1<<16) File "/misc/local/galaxy/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpserver.py", line 467, in readline data = self.file.readline(self.length - self._consumed) File "/usr/local/python27/lib/python2.7/socket.py", line 412, in readline bline = buf.readline(size) OverflowError: signed integer is greater than maximum
On Thu, Nov 29, 2012 at 11:26 AM, greg <margeemail@gmail.com> wrote:
This says Google Chrome can support uploads larger than 4 GB. http://www.motobit.com/help/scptutl/pa98.htm
Maybe I just need to configure Apache?
I'll let you guys know what I find.
-Greg
On Thu, Nov 29, 2012 at 11:22 AM, greg <margeemail@gmail.com> wrote:
Follow up question. What's the deal with the 2GB limit it mentions?
I think it would be pretty convienent to allow larger uploads. We're on a fast local network so even 10 or 20 GB uploads pretty fast.
Can I increase the limit in Apache? Which browsers are limited? This is an internal install so I can tell users to only use certain browsers for uploads.
-Greg
On Thu, Nov 29, 2012 at 11:01 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 29, 2012, at 10:54 AM, greg wrote:
Thanks. That seemed to have worked. Can you explain what happened? (I'm curious)
It's due to a bug that we've done quite a bit of work to investigate, but haven't yet found the cause. It seems to have cropped up more frequently in the past few weeks, so we'll be working on it in depth in the next few days.
--nate
-Greg
On Thu, Nov 29, 2012 at 8:43 AM, Nate Coraor <nate@bx.psu.edu> wrote:
On Nov 28, 2012, at 1:23 PM, greg wrote:
> I just finished setting up a local install on our cluster but I'm > having trouble uploading files. > > I was able to (successfully?) upload one 160MB fasta file. But any > further uploads I try, they just hang.
Hi Greg,
Could you try the following:
Open tools/data_source/upload.xml
In the following line, set 'ajax-upload="false"':
<param name="file_data" type="file" size="30" label="File" ajax-upload="false" help="TIP: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator).">
Then restart Galaxy and retry the upload and let us know if it succeeds.
--nate
> > Here's the information from the log file: > > paste.httpserver.ThreadPool INFO 2012-11-28 12:15:18,190 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) > ave time N/A, max time 0.00sec, killed 0 workers > galaxy.tools.actions.upload_common INFO 2012-11-28 12:20:36,597 tool > upload1 created job id 4 > galaxy.jobs.handler INFO 2012-11-28 12:20:38,386 (4) Job dispatched > galaxy.jobs.runners.drmaa INFO 2012-11-28 12:20:38,930 (4) queued as 3718641 > galaxy.objectstore CRITICAL 2012-11-28 12:21:03,071 > /home/galaxy/job_working_directory/000/4 delete error [Errno 39] > Directory not empty: '/home/galaxy/job_working_directory/000/4' > paste.httpserver.ThreadPool INFO 2012-11-28 12:21:55,763 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) > ave time N/A, max time 0.00sec, killed 0 workers > paste.httpserver.ThreadPool INFO 2012-11-28 12:28:26,466 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) > ave time N/A, max time 0.00sec, killed 0 workers > paste.httpserver.ThreadPool INFO 2012-11-28 12:35:09,662 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) > ave time N/A, max time 0.00sec, killed 0 workers > paste.httpserver.ThreadPool INFO 2012-11-28 12:46:35,565 > kill_hung_threads status: 10 threads (0 working, 10 idle, 0 starting) > ave time N/A, max time 0.00sec, killed 0 workers > > > The first upload that worked was around 12:20 (above), and the second > failed one was at 12:40. The log file doesn't seem to even mention > it? Could that error have killed the server somehow? > > Let me know if I can provide more info. > > Thanks, > > Greg > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/
participants (3)
-
greg -
Nate Coraor -
Sytchev, Ilya