From isytchev@hsph.harvard.edu Thu May 12 15:01:32 2016 From: "Sytchev, Ilya" To: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Problem uploading multigigabyte files via API Date: Thu, 12 May 2016 19:01:30 +0000 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7485164365018449188==" --===============7485164365018449188== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Just to follow up: we've implemented a workaround for uploading large files into libraries using upload_file_from_url(): https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend .galaxy.libraries.LibraryClient.upload_file_from_url Also, there is a corresponding function for uploading files into histories: https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend .galaxy.tools.ToolClient.put_url Ilya -----Original Message----- From: galaxy-dev on behalf of Ilya Sytchev Date: Wednesday, April 27, 2016 at 15:53 To: "galaxy-dev(a)lists.galaxyproject.org" Subject: [galaxy-dev] Problem uploading multigigabyte files via API >Hi, > >I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came >across an issue where uploading files larger than ~3GB via the API into >histories and libraries fails. > >Steps to reproduce (using BioBlend 0.7.0): > >from bioblend.galaxy.objects import GalaxyInstance >gi = GalaxyInstance("galaxy-dev", "api_key") >h = gi.histories.create() >h.upload_dataset('/path/to/multigigabytefile') > >Observed results: > >ConnectionError: Unexpected response from galaxy: 504: > >504 Gateway Time-out > >

504 Gateway Time-out

nginx/1.4.6 (Ubuntu) > > > >It looks like Nginx is accepting the full incoming data transfer over the >network and saving it to disk successfully, and the error occurs during >the subsequent forwarding of the request to Galaxy. There doesn't seem to >be a particular threshold for file size (for example, 2GB and 2.7GB file >can be uploaded successfully). Also, large file uploads do work when >Galaxy is not running behind Nginx. > > >Relevant log entries and config details are attached. I'd appreciate any >help. > >Thanks, >Ilya > > --===============7485164365018449188==--