Hello All, We have a local galaxy install at our facility that is running on a VM machine with an allocated memory of 2gigs. I have a user that has been trying to upload a ~6gig fastq file to galaxy for the past couple of days within our same subnet. She tried submitting the upload but the next day it still wasn't finished uploading, so I told her to cancel the upload and try again. The same problem occurred the second day after a day of trying. Should galaxy be able to support an upload of this size?
Thanks, -Hans
Hello Hans,
What version of Galaxy are you running? We recently introduced enhancements to the upload utility for uploading files to a History - these were in change set 2583. These enhancements are currently under development for uploading files to a Data Library.
If you are running an older version of Galaxy, are you running behind an Apache proxy? If so, you may want to increase the value of the TimeOut or ProxyTimeout directives.
Greg Von Kuster Galaxy Development Team
Hans Vasquez-Gross wrote:
Hello All, †† We have a local galaxy install at our facility that is running on a VM machine with an allocated memory of 2gigs.† I have a user that has been trying to upload a ~6gig fastq file to galaxy for the past couple of days within our same subnet.† She tried submitting the upload but the next day it still wasn't finished uploading, so I told her to cancel the upload and try again.† The same problem occurred the second day after a day of trying.† Should galaxy be able to support an upload of this size?
Thanks, -Hans
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Another option is to compress the file before uploading it. It will reduce the bandwidth required in uploading and Galaxy will automatically decompress it once uploaded.
Chris
Greg Von Kuster wrote:
Hello Hans,
What version of Galaxy are you running? We recently introduced enhancements to the upload utility for uploading files to a History - these were in change set 2583. These enhancements are currently under development for uploading files to a Data Library.
If you are running an older version of Galaxy, are you running behind an Apache proxy? If so, you may want to increase the value of the TimeOut or ProxyTimeout directives.
Greg Von Kuster Galaxy Development Team
Hans Vasquez-Gross wrote:
Hello All, †† We have a local galaxy install at our facility that is running on a VM machine with an allocated memory of 2gigs.† I have a user that has been trying to upload a ~6gig fastq file to galaxy for the past couple of days within our same subnet.† She tried submitting the upload but the next day it still wasn't finished uploading, so I told her to cancel the upload and try again.† The same problem occurred the second day after a day of trying.† Should galaxy be able to support an upload of this size?
Thanks, -Hans
We did some testing and found that the upload isn't finishing because the file system that has the /tmp/ folder has too little space. I did a find for the .py files and greped for files containing /tmp/ and found that there are some files that have this path hardcoded in the scripts. We have the space on another file system that is located in /temp/galaxy_space. Is there anyway to change this configuration via a variable? If not, I'll try changing these locations manually to see if this fixes the problem.
Thanks for the help, Hans Vasquez-Gross
On Wed, Sep 9, 2009 at 2:11 PM, Greg Von Kuster ghv2@psu.edu wrote:
Hello Hans,
What version of Galaxy are you running? We recently introduced enhancements to the upload utility for uploading files to a History - these were in change set 2583. These enhancements are currently under development for uploading files to a Data Library.
If you are running an older version of Galaxy, are you running behind an Apache proxy? If so, you may want to increase the value of the TimeOut or ProxyTimeout directives.
Greg Von Kuster Galaxy Development Team
Hans Vasquez-Gross wrote:
Hello All, †† We have a local galaxy install at our facility that is running on a VM machine with an allocated memory of 2gigs.† I have a user that has been trying to upload a ~6gig fastq file to galaxy for the past couple of days within our same subnet.† She tried submitting the upload but the next day it still wasn't finished uploading, so I told her to cancel the upload and try again.† The same problem occurred the second day after a day of trying.† Should galaxy be able to support an upload of this size?
Thanks, -Hans
galaxy-user mailing list galaxy-user@bx.psu.edu http://mail.bx.psu.edu/cgi-bin/mailman/listinfo/galaxy-user
Hans Vasquez-Gross wrote:
We did some testing and found that the upload isn't finishing because the file system that has the /tmp/ folder has too little space. I did a find for the .py files and greped for files containing /tmp/ and found that there are some files that have this path hardcoded in the scripts. We have the space on another file system that is located in /temp/galaxy_space. Is there anyway to change this configuration via a variable? If not, I'll try changing these locations manually to see if this fixes the problem.
Hi Hans,
/tmp is hardcoded as a last result, but everything uses Python's tempfile module, which has a set list of methods for determining where to put stuff.
The easiest solution is to set $TEMP in the shell before starting Galaxy. We do this now to place tempfiles on the same filesystem as the database/files/ directory for faster upload processing (i.e. moves during upload processing just update an inode instead of moving data).
--nate
galaxy-user@lists.galaxyproject.org
-
Chris Cole -
Greg Von Kuster -
Hans Vasquez-Gross -
Nate Coraor