Re: [galaxy-dev] Problem uploading multigigabyte files via API

12 May 2016


      Just to follow up: we've implemented a workaround for uploading large
files into libraries using upload_file_from_url():
https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend
.galaxy.libraries.LibraryClient.upload_file_from_url


Also, there is a corresponding function for uploading files into
histories: 
https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend
.galaxy.tools.ToolClient.put_url

Ilya


-----Original Message-----
From: galaxy-dev <galaxy-dev-bounces@lists.galaxyproject.org> on behalf of
Ilya Sytchev <isytchev@hsph.harvard.edu>
Date: Wednesday, April 27, 2016 at 15:53
To: "galaxy-dev@lists.galaxyproject.org"
<galaxy-dev@lists.galaxyproject.org>
Subject: [galaxy-dev] Problem uploading multigigabyte files via API
...
Hi,
I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came
across an issue where uploading files larger than ~3GB via the API into
histories and libraries fails.
Steps to reproduce (using BioBlend 0.7.0):
from bioblend.galaxy.objects import GalaxyInstance
gi = GalaxyInstance("galaxy-dev", "api_key")
h = gi.histories.create()
h.upload_dataset('/path/to/multigigabytefile')
Observed results:
ConnectionError: Unexpected response from galaxy: 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
It looks like Nginx is accepting the full incoming data transfer over the
network and saving it to disk successfully, and the error occurs during
the subsequent forwarding of the request to Galaxy. There doesn't seem to
be a particular threshold for file size (for example, 2GB and 2.7GB file
can be uploaded successfully). Also, large file uploads do work when
Galaxy is not running behind Nginx.
Relevant log entries and config details are attached. I'd appreciate any
help.
Thanks,
Ilya

Sytchev, Ilya

tags

participants (1)