Just to follow up: we've implemented a workaround for uploading large files into libraries using upload_file_from_url(): https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend .galaxy.libraries.LibraryClient.upload_file_from_url
Also, there is a corresponding function for uploading files into histories: https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend .galaxy.tools.ToolClient.put_url
Ilya
-----Original Message----- From: galaxy-dev galaxy-dev-bounces@lists.galaxyproject.org on behalf of Ilya Sytchev isytchev@hsph.harvard.edu Date: Wednesday, April 27, 2016 at 15:53 To: "galaxy-dev@lists.galaxyproject.org" galaxy-dev@lists.galaxyproject.org Subject: [galaxy-dev] Problem uploading multigigabyte files via API
Hi,
I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came across an issue where uploading files larger than ~3GB via the API into histories and libraries fails.
Steps to reproduce (using BioBlend 0.7.0):
from bioblend.galaxy.objects import GalaxyInstance gi = GalaxyInstance("galaxy-dev", "api_key") h = gi.histories.create() h.upload_dataset('/path/to/multigigabytefile')
Observed results:
ConnectionError: Unexpected response from galaxy: 504: <html>
<head><title>504 Gateway Time-out</title></head> <body bgcolor="white"> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx/1.4.6 (Ubuntu)</center> </body> </html>
It looks like Nginx is accepting the full incoming data transfer over the network and saving it to disk successfully, and the error occurs during the subsequent forwarding of the request to Galaxy. There doesn't seem to be a particular threshold for file size (for example, 2GB and 2.7GB file can be uploaded successfully). Also, large file uploads do work when Galaxy is not running behind Nginx.
Relevant log entries and config details are attached. I'd appreciate any help.
Thanks, Ilya