/dev/xvda1 filling up when downloading large files via ftp
Dear all, this problem has been mentioned before by McDowell@vims.ed, but there was no solution so far posted. I'm using the current cloudman on Amazon AWS. When I download a fastq.gz file via an EBI-SRA ftp link using the "Upload File" function, although the file itself is loading to the /mnt/galaxy drive, at the same time /dev/xvda1 is filling up and at some point the download stops and says "[Errno 28] No space left on device". I've checked in the universe_wsgi.ini and the ftp upload path is set correctly to /mnt/galaxy/tmp/ftp. Here's the df output: Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 10321208 9797012 0 100% / udev 8736124 8 8736116 1% /dev tmpfs 3497960 232 3497728 1% /run none 5120 0 5120 0% /run/lock none 8744896 0 8744896 0% /run/shm cgroup 8744896 0 8744896 0% /sys/fs/cgroup /dev/xvdb 423135208 207440 401433728 1% /mnt /dev/xvdf 104847360 22055572 82791788 22% /mnt/galaxy /dev/xvdg 733900800 685188288 48712512 94% /mnt/galaxyIndices I'm also having another issue: when I delete files in the history in galaxy, they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder. Could the two issues be connected? Are there maybe some rights missing for the galaxy instance to delete files? Thanks in advance for any help Best Stanislas
On Tue, Oct 15, 2013 at 10:13 AM, Stanislas Werfel <Werfel@ipt.med.tum.de> wrote:
I'm also having another issue: when I delete files in the history in galaxy, they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder.
I can't help with the first problem, but I believe the second is easier: The data isn't immediately deleted and therefore your usage figure (and quota) do not get updated right away. This is usually done via a house keeping cron job, see: http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20a... Peter
And I can answer the first part; this is a known issue we're trying to nail down and get a fix out for this week, along with a base volume update to address other issues. Trello card: https://trello.com/c/8XSIeoG0 On Tue, Oct 15, 2013 at 10:23 AM, Peter Cock <p.j.a.cock@googlemail.com>wrote:
On Tue, Oct 15, 2013 at 10:13 AM, Stanislas Werfel <Werfel@ipt.med.tum.de> wrote:
I'm also having another issue: when I delete files in the history in
galaxy,
they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder.
I can't help with the first problem, but I believe the second is easier: The data isn't immediately deleted and therefore your usage figure (and quota) do not get updated right away. This is usually done via a house keeping cron job, see:
http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20a...
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi, thanks for your help. Hope the update comes soon. I have another small question. After I install bowtie2 and tophat2 and set the paths using "export PATH=$PATH:..." the paths are added to the variable but when I terminate and restart they're gone again. What should one do here? Best Stan Von: Dannon Baker [mailto:dannon.baker@gmail.com] Gesendet: Dienstag, 15. Oktober 2013 16:49 An: Peter Cock Cc: Stanislas Werfel; galaxy-dev@lists.bx.psu.edu Betreff: Re: [galaxy-dev] /dev/xvda1 filling up when downloading large files via ftp And I can answer the first part; this is a known issue we're trying to nail down and get a fix out for this week, along with a base volume update to address other issues. Trello card: https://trello.com/c/8XSIeoG0 On Tue, Oct 15, 2013 at 10:23 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote: On Tue, Oct 15, 2013 at 10:13 AM, Stanislas Werfel <Werfel@ipt.med.tum.de> wrote:
I'm also having another issue: when I delete files in the history in
galaxy,
they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder.
I can't help with the first problem, but I believe the second is easier: The data isn't immediately deleted and therefore your usage figure (and quota) do not get updated right away. This is usually done via a house keeping cron job, see: http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20a nd%20Datasets Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
PS: I just tried running all the delete and purge scripts without further paramters as described on http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20a nd%20Datasets. However alls the dat files are still there in the /files/000 folder. How does one execute the scripts correctly to delete files which have been deleted in the history. Shouldn't galaxy execute the scripts by itself? Best Stan Von: Dannon Baker [mailto:dannon.baker@gmail.com] Gesendet: Dienstag, 15. Oktober 2013 16:49 An: Peter Cock Cc: Stanislas Werfel; galaxy-dev@lists.bx.psu.edu Betreff: Re: [galaxy-dev] /dev/xvda1 filling up when downloading large files via ftp And I can answer the first part; this is a known issue we're trying to nail down and get a fix out for this week, along with a base volume update to address other issues. Trello card: https://trello.com/c/8XSIeoG0 On Tue, Oct 15, 2013 at 10:23 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote: On Tue, Oct 15, 2013 at 10:13 AM, Stanislas Werfel <Werfel@ipt.med.tum.de> wrote:
I'm also having another issue: when I delete files in the history in
galaxy,
they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder.
I can't help with the first problem, but I believe the second is easier: The data isn't immediately deleted and therefore your usage figure (and quota) do not get updated right away. This is usually done via a house keeping cron job, see: http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20a nd%20Datasets Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (3)
-
Dannon Baker -
Peter Cock -
Stanislas Werfel