Problem uploading files gt 1 Mbyte to Galaxy Cloud
Hi, I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes. I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ". If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes. These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad. I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page. [image: logs.png] My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files. I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check. Many thanks, Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
Hi Greg, This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process. You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly. -Dannon On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ".
If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page.
<logs.png>
My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check.
Many thanks, Greg E
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Dannon, Thanks. Glad it's not my error ! Could ypu let me know where nginx.conf is, and how to "restart the nginx process" ? This is a prototype proof-of-concept implementation, and I think a once-per-instance custom fix which preserves the simple ability to use Get Data / Upload Files From Your Computer will be best for now. Thanks (and also for being awake at 4am in Pennsylvania, if I have my time zones right) Greg E On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker <dannonbaker@me.com> wrote:
Hi Greg,
This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ".
If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page.
<logs.png>
My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check.
Many thanks, Greg E
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
Hi Dannon, I found nginx.conf at /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf It contains .. client_max_body_size 100m; That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I set it to 10000m, or 1g ? Can't see how to restart nginx yet. Thanks, Greg E On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker <dannonbaker@me.com> wrote:
Hi Greg,
This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ".
If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page.
<logs.png>
My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check.
Many thanks, Greg E
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't think that one you found is used. And it isn't explicitly set at all currently, so the default 1m gets applied. And, to reload with the modified config, I think this should do the trick (as the ubuntu user)- sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid` -Dannon On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:
Hi Dannon,
I found nginx.conf at /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
It contains ..
client_max_body_size 100m;
That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I set it to 10000m, or 1g ?
Can't see how to restart nginx yet.
Thanks, Greg E
On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker <dannonbaker@me.com> wrote: Hi Greg,
This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ".
If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page.
<logs.png>
My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check.
Many thanks, Greg E
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
Dannon, Cool, that seems to have worked. I set server { listen 80; client_max_body_size 500m; in /opt/galaxy/pkg/nginx/conf/nginx.conf and issued sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid` and it's uploading my test files of 2mb to 25mb ok. I think it will be easier to do that in my Cloud setup than instruct my (not very computer keen) wet-lab Proteomist clients how to use a subsidiary FTP package, easy though that is. But I'll try the FTP approach too for compariosn. Thanks, Greg E On Mon, Jan 23, 2012 at 8:58 PM, Dannon Baker <dannonbaker@me.com> wrote:
It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't think that one you found is used. And it isn't explicitly set at all currently, so the default 1m gets applied.
And, to reload with the modified config, I think this should do the trick (as the ubuntu user)-
sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
-Dannon
On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:
Hi Dannon,
I found nginx.conf at /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
It contains ..
client_max_body_size 100m;
That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I set it to 10000m, or 1g ?
Can't see how to restart nginx yet.
Thanks, Greg E
On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker <dannonbaker@me.com> wrote: Hi Greg,
This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ".
If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page.
<logs.png>
My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check.
Many thanks, Greg E
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
Great, glad it worked for you! And, once we release a new AMI, it'll be fixed retroactively. That is, you'll be able to use your existing cloud setup with the new AMI and it should just work. -Dannon On Jan 23, 2012, at 5:35 AM, Greg Edwards wrote:
Dannon,
Cool, that seems to have worked. I set
server { listen 80; client_max_body_size 500m;
in /opt/galaxy/pkg/nginx/conf/nginx.conf
and issued sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
and it's uploading my test files of 2mb to 25mb ok.
I think it will be easier to do that in my Cloud setup than instruct my (not very computer keen) wet-lab Proteomist clients how to use a subsidiary FTP package, easy though that is. But I'll try the FTP approach too for compariosn.
Thanks, Greg E
On Mon, Jan 23, 2012 at 8:58 PM, Dannon Baker <dannonbaker@me.com> wrote: It's the other nginx.conf at /opt/galaxy/pkg/nginx/conf/nginx.conf, I don't think that one you found is used. And it isn't explicitly set at all currently, so the default 1m gets applied.
And, to reload with the modified config, I think this should do the trick (as the ubuntu user)-
sudo kill -HUP `cat /opt/galaxy/pkg/nginx/logs/nginx.pid`
-Dannon
On Jan 23, 2012, at 4:43 AM, Greg Edwards wrote:
Hi Dannon,
I found nginx.conf at /opt/galaxy/pkg/nginx_upload_module-2.0.12/nginx.conf
It contains ..
client_max_body_size 100m;
That looks like 100 Megabyte ?? but seems to result in 1 Megabyte. Can I set it to 10000m, or 1g ?
Can't see how to restart nginx yet.
Thanks, Greg E
On Mon, Jan 23, 2012 at 8:18 PM, Dannon Baker <dannonbaker@me.com> wrote: Hi Greg,
This is a problem with the default client_max_body_size option in nginx being set far too small in the nginx.conf on the cloud AMI. It'll be fixed with our next AMI update, but to fix it you have to edit the nginx.conf on your cloud node to change the client_max_body_size to something more appropriate for Galaxy uploads and then restart the nginx process.
You'd have to do it for every instance, unfortunately, since that section of the filesystem is not persisted after shutdown. As a workaround the URL upload will work correctly with any size file if you're able to host the file you want to upload somewhere local, or you could use FTP upload and that should also function correctly.
-Dannon
On Jan 23, 2012, at 4:09 AM, Greg Edwards wrote:
Hi,
I'm building a Galaxy Cloud implementation, and running into a problem uploading data files > 1 Mbyte, or 1,048,576 bytes.
I've started Galaxy Cloudman and a single server, all according to to the Wiki. I can upload fasta format files < 1048756 bytes, and general CSV format peptide files, and indeed junk data files built with variants of " jot -r -c 1030000 a z | rs -g 0 100 >junk ".
If I upload files (via the built-in Get Data / Upload Files From Your Computer) that are > than approx 2^20 bytes, the upload just hangs in the History and never completes.
These files upload fine to a local Galaxy server, and to Galaxy Main. On the Galaxy local server, they run fine with my custom proteomics code. Ditto on the Cloud galaxy server when the (smaller than 1 Mbtye) files successfully upoad.
I can't see anthing useful in the logs, but I'm not sure which logs I should be looking at. I'm just looking at tyhe logs presented on the Cloudman admin page.
<logs.png>
My environment is Mac, OS X 10.6.8. My data files are terminated with /n, but the problem manifests the same with Windows style /r/n files.
I'll leave it at that in the hope this is a ridiculously simple probelm and needs no more description ! If not, please let me know the logs I should look at and whatever else I should check.
Many thanks, Greg E
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
-- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com
participants (2)
-
Dannon Baker -
Greg Edwards