Problem uploading multigigabyte files via API

27 Apr 2016

      Hi,

I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came
across an issue where uploading files larger than ~3GB via the API into
histories and libraries fails.

Steps to reproduce (using BioBlend 0.7.0):

from bioblend.galaxy.objects import GalaxyInstance
gi = GalaxyInstance("galaxy-dev", "api_key")
h = gi.histories.create()
h.upload_dataset('/path/to/multigigabytefile')

Observed results:

ConnectionError: Unexpected response from galaxy: 504: <html>

<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

It looks like Nginx is accepting the full incoming data transfer over the
network and saving it to disk successfully, and the error occurs during
the subsequent forwarding of the request to Galaxy. There doesn't seem to
be a particular threshold for file size (for example, 2GB and 2.7GB file
can be uploaded successfully). Also, large file uploads do work when
Galaxy is not running behind Nginx.

Relevant log entries and config details are attached. I'd appreciate any
help.

Thanks,
Ilya

Sytchev, Ilya

tags

participants (1)