new datasets appear in database/files/000/
Hi, today we found out that some of our files in your galaxy instance are new in "database/files/000/" which is the first folder of your database files. Could some please explain me how galaxy generates new file names? and how this is possible? Thanks in advance Jochen -- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
Coming back to my question: this is how an ls -ls looks in my database folder... 2. Apr 14:40 000/ 9. Nov 2015 005/ 19. Nov 2018 007/ 19. Nov 2018 008/ 3. Dez 2015 010/ 10. Jan 2017 012/ 10. Jan 2017 013/ 29. Jun 2017 014/ 4. Jul 2017 015/ 9. Jan 2018 018/ 29. Jun 2017 020/ 21. Sep 2016 021/ 20. Mär 2018 032/ 29. Jan 09:38 033/ 29. Jan 10:32 034/ 16. Jul 2018 035/ 30. Okt 2018 036/ 30. Okt 2018 037/ 30. Okt 2018 038/ 30. Okt 2018 039/ 29. Aug 2018 070/ 29. Aug 2018 071/ 29. Aug 2018 072/ 29. Aug 2018 073/ 29. Aug 2018 074/ 29. Aug 2018 075/ 29. Aug 2018 076/ 29. Aug 2018 077/ 29. Aug 2018 078/ 29. Aug 2018 079/ 29. Aug 2018 080/ 29. Aug 2018 081/ 29. Aug 2018 082/ 29. Aug 2018 083/ 29. Aug 2018 084/ 29. Aug 2018 085/ 29. Aug 2018 086/ 29. Aug 2018 087/ 29. Aug 2018 088/ 29. Aug 2018 089/ 29. Aug 2018 090/ 29. Aug 2018 091/ 29. Aug 2018 092/ 29. Aug 2018 093/ 29. Aug 2018 094/ 29. Aug 2018 095/ 31. Aug 2018 096/ 30. Aug 2018 097/ 1. Sep 2018 098/ 24. Okt 2018 099/ if you look into the 000 folder it restarts: 10. Dez 19:36 100/ 30. Nov 10:45 101/ 10. Dez 19:39 102/ 10. Dez 19:37 103/ 11. Jan 15:13 104/ 11. Jan 11:31 105/ 22. Jan 14:33 106/ 8. Feb 10:46 107/ 14. Feb 15:47 108/ 21. Mär 11:00 109/ 2. Apr 15:22 110/ 21. Mai 17:17 111/ is this the expected behavior? The only thing we did was we archived some of the very early generated datasets. Is this maybe the problem? Cheers Jochen On 14.05.19 15:32, Jochen Bick wrote:
Hi,
today we found out that some of our files in your galaxy instance are new in "database/files/000/" which is the first folder of your database files. Could some please explain me how galaxy generates new file names? and how this is possible?
Thanks in advance Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
Hi Jochen, Yes, those new directories would coincide with when you passed 100,000 datasets on your instance, so congrats! You can see the code that builds this hashed path here: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/util/__init__.py... So, for example, dataset id `99999` Will be in folder 'datasets/099/'. Dataset '100000' will be in 'datasets/000/100/'. Sorry for the slow response, hope this clears up the storage path logic! -Dannon On Tue, May 21, 2019 at 11:38 AM Jochen Bick <jochen.bick@usys.ethz.ch> wrote:
Coming back to my question:
this is how an ls -ls looks in my database folder...
2. Apr 14:40 000/ 9. Nov 2015 005/ 19. Nov 2018 007/ 19. Nov 2018 008/ 3. Dez 2015 010/ 10. Jan 2017 012/ 10. Jan 2017 013/ 29. Jun 2017 014/ 4. Jul 2017 015/ 9. Jan 2018 018/ 29. Jun 2017 020/ 21. Sep 2016 021/ 20. Mär 2018 032/ 29. Jan 09:38 033/ 29. Jan 10:32 034/ 16. Jul 2018 035/ 30. Okt 2018 036/ 30. Okt 2018 037/ 30. Okt 2018 038/ 30. Okt 2018 039/ 29. Aug 2018 070/ 29. Aug 2018 071/ 29. Aug 2018 072/ 29. Aug 2018 073/ 29. Aug 2018 074/ 29. Aug 2018 075/ 29. Aug 2018 076/ 29. Aug 2018 077/ 29. Aug 2018 078/ 29. Aug 2018 079/ 29. Aug 2018 080/ 29. Aug 2018 081/ 29. Aug 2018 082/ 29. Aug 2018 083/ 29. Aug 2018 084/ 29. Aug 2018 085/ 29. Aug 2018 086/ 29. Aug 2018 087/ 29. Aug 2018 088/ 29. Aug 2018 089/ 29. Aug 2018 090/ 29. Aug 2018 091/ 29. Aug 2018 092/ 29. Aug 2018 093/ 29. Aug 2018 094/ 29. Aug 2018 095/ 31. Aug 2018 096/ 30. Aug 2018 097/ 1. Sep 2018 098/ 24. Okt 2018 099/
if you look into the 000 folder it restarts:
10. Dez 19:36 100/ 30. Nov 10:45 101/ 10. Dez 19:39 102/ 10. Dez 19:37 103/ 11. Jan 15:13 104/ 11. Jan 11:31 105/ 22. Jan 14:33 106/ 8. Feb 10:46 107/ 14. Feb 15:47 108/ 21. Mär 11:00 109/ 2. Apr 15:22 110/ 21. Mai 17:17 111/
is this the expected behavior?
The only thing we did was we archived some of the very early generated datasets. Is this maybe the problem?
Cheers Jochen
On 14.05.19 15:32, Jochen Bick wrote:
Hi,
today we found out that some of our files in your galaxy instance are new in "database/files/000/" which is the first folder of your database files. Could some please explain me how galaxy generates new file names? and how this is possible?
Thanks in advance Jochen
-- ETH Zurich *Jochen Bick* Animal Physiology Institute of Agricultural Sciences Postal address: Universitätstrasse 2 / LFW B 58.1 8092 Zurich, Switzerland Office: Eschikon 27 8315 Lindau, Switzerland
Phone +41 52 354 92 06 jochen.bick@usys.ethz.ch <mailto:jochen.bick@usys.ethz.ch> www.ap.ethz.ch
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
participants (2)
-
Dannon Baker -
Jochen Bick