Dear all, I am really sorry for the late reminder but, our July meeting is today... July 18, 2024, 3PM CEST (https://www.timeanddate.com/worldclock/fixedtime.html?msg=Small+Scale+Galaxy+Admins+Meeting&iso=20240718T15&p1=945&ah=1) Here is the zoom link: https://epfl.zoom.us/j/69775900853?pwd=R1gzbllUZzg0RFUrM0Mzd2MvczE3Zz09 Here is our working document, do not hesitate to put your topic suggestions: https://docs.google.com/document/d/1KTz4qxZBG4Z3sF23dVKPWXYz1mCNo2YDygSUHqtA... Regards, Hans-Rudolf and Lucille PS: the upcoming calls for your calendar: 19. September 3PM CEST (1PM UTC) 21. November 3PM CET (2PM UTC) -- Lucille Delisle, PhD PostDoc Duboule Lab Laboratory of Developmental Genomics EPFL SV ISREC UPDUB SV 2843 CH-1015 Lausanne Tel : +41 21 693 97 08
Hi all, Data Library Synchronisation * I don't remember who mentioned a/several project(s) that synchronise data libraries with a given folder? * Is there perhaps some public information/links? * I searched but only found an old link from 2013 [ https://lists.galaxyproject.org/archives/list/galaxy-dev@lists.galaxyproject... | https://lists.galaxyproject.org/archives/list/galaxy-dev@lists.galaxyproject... ] Poll slides * Also, I forgot to mention it separately, but the poll slides are published [ https://training.galaxyproject.org/training-material/topics/admin/tutorials/... | https://training.galaxyproject.org/training-material/topics/admin/tutorials/... ] * Thanks to Helena for her detailed feedback. * GTN News post PR [ https://github.com/galaxyproject/training-material/pull/5177 | https://github.com/galaxyproject/training-material/pull/5177 ] * I'm not sure how prospective admins could be informed of its existence (since the news post is for existing admins) * But I did "manually" suggest this to Jasper & Jacques yesterday Thanks, Vlad De: "Delisle Lucille via Small-scale-admins" <small-scale-admins@lists.galaxyproject.org> À: "small-scale-admins" <small-scale-admins@lists.galaxyproject.org> Envoyé: Jeudi 18 Juillet 2024 08:57:10 Objet: [Small-scale-admins] Today: Meeting of 'small scale servers' group Dear all, I am really sorry for the late reminder but, our July meeting is today... July 18, 2024, 3PM CEST (https://www.timeanddate.com/worldclock/fixedtime.html?msg=Small+Scale+Galaxy+Admins+Meeting&iso=20240718T15&p1=945&ah=1) Here is the zoom link: https://epfl.zoom.us/j/69775900853?pwd=R1gzbllUZzg0RFUrM0Mzd2MvczE3Zz09 Here is our working document, do not hesitate to put your topic suggestions: https://docs.google.com/document/d/1KTz4qxZBG4Z3sF23dVKPWXYz1mCNo2YDygSUHqtA... Regards, Hans-Rudolf and Lucille PS: the upcoming calls for your calendar: 19. September 3PM CEST (1PM UTC) 21. November 3PM CET (2PM UTC) -- Lucille Delisle, PhD PostDoc Duboule Lab Laboratory of Developmental Genomics EPFL SV ISREC UPDUB SV 2843 CH-1015 Lausanne Tel : +41 21 693 97 08 _______________________________________________ Small-scale-admins mailing list -- small-scale-admins@lists.galaxyproject.org To unsubscribe send an email to small-scale-admins-leave@lists.galaxyproject.org
Hi Vlad
* I don't remember who mentioned a/several project(s) that synchronise data libraries with a given folder?
well, since I did not attend yesterday's meeting, I couldn't mention it... In our case, all fastq files generated by our sequencing facility are linked (not copied) to Data Libraries. There is a python script (running every night) checking the facilitiy's LIMS for any new data. And then based on the information in the LIMS, several calls to the Galaxy API generate new sub-folders and link the data. So it is a very customized solution fitting our local LIMS, storage set-up, etc. Regards, Hans-Rudolf
Hi Hans, all, Thank you for sharing your solution. I've been looking into the API and have used [ https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_ | this endpoint ] [ https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_ ] (strangely BioBlend seems to wrap around [ https://usegalaxy.org/api/docs#/undocumented/post_api_libraries__library_id_... | this less expressive one ] instead), very useful! If enough people have this synchronisation use-case, there could be a case for a Galaxy-provided link between a File Source (assuming your LIMS can be imported through one) and a Data Library. Regards, Vlad De: "Hans-Rudolf Hotz" <hrh@fmi.ch> À: "VLAD VISAN" <vlad.visan@univ-grenoble-alpes.fr>, "small-scale-admins" <small-scale-admins@lists.galaxyproject.org> Envoyé: Vendredi 19 Juillet 2024 14:07:08 Objet: Re: [Small-scale-admins] Re: Today: Meeting of 'small scale servers' group Hi Vlad
* I don't remember who mentioned a/several project(s) that synchronise data libraries with a given folder?
well, since I did not attend yesterday's meeting, I couldn't mention it... In our case, all fastq files generated by our sequencing facility are linked (not copied) to Data Libraries. There is a python script (running every night) checking the facilitiy's LIMS for any new data. And then based on the information in the LIMS, several calls to the Galaxy API generate new sub-folders and link the data. So it is a very customized solution fitting our local LIMS, storage set-up, etc. Regards, Hans-Rudolf
Hi Vlad, it was me who mentioned the library sync. I've heard of similar cases that Hans-Rudolf is describing and some more sophisticated ones that tried to do bi-directional sync. I couldn't find a link or any more information so I asked in one of the Galaxy channels -- maybe someone has a better memory ( link <https://matrix.to/#/!fVQwotRzniGGJbeUUi:gitter.im/$-y1pfvymkrrvGXOvxhVwVky4ZWdNOI1lGM94iZFztEU?via=gitter.im&via=matrix.org>). The thing is, that it is likely always going to be very custom and heavily based on what your other systems look like. Long time ago we had attempted to implement more full-featured LIMS in Galaxy, but that has long been gone since it was always just too general. Cheers, Martin On Fri, Jul 19, 2024 at 3:03 PM VLAD VISAN < vlad.visan@univ-grenoble-alpes.fr> wrote:
Hi Hans, all,
Thank you for sharing your solution. I've been looking into the API and have used this endpoint <https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_> <https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_>(strangely BioBlend seems to wrap around this less expressive one <https://usegalaxy.org/api/docs#/undocumented/post_api_libraries__library_id__contents> instead), very useful! If enough people have this synchronisation use-case, there could be a case for a Galaxy-provided link between a File Source (assuming your LIMS can be imported through one) and a Data Library.
Regards,
Vlad
------------------------------ *De: *"Hans-Rudolf Hotz" <hrh@fmi.ch> *À: *"VLAD VISAN" <vlad.visan@univ-grenoble-alpes.fr>, "small-scale-admins" <small-scale-admins@lists.galaxyproject.org> *Envoyé: *Vendredi 19 Juillet 2024 14:07:08 *Objet: *Re: [Small-scale-admins] Re: Today: Meeting of 'small scale servers' group
Hi Vlad
* I don't remember who mentioned a/several project(s) that synchronise data libraries with a given folder?
well, since I did not attend yesterday's meeting, I couldn't mention it...
In our case, all fastq files generated by our sequencing facility are linked (not copied) to Data Libraries. There is a python script (running every night) checking the facilitiy's LIMS for any new data. And then based on the information in the LIMS, several calls to the Galaxy API generate new sub-folders and link the data.
So it is a very customized solution fitting our local LIMS, storage set-up, etc.
Regards, Hans-Rudolf _______________________________________________ Small-scale-admins mailing list -- small-scale-admins@lists.galaxyproject.org To unsubscribe send an email to small-scale-admins-leave@lists.galaxyproject.org
Hi Vlad, all, https://github.com/usegalaxy-eu/shared-data was one attempt at uni-directional sync, yaml files describing data / URLs are kept continually in sync with target galaxies. Maybe not something very reusable but potentially useful as inspiration? mvg Helena On Fri, 19 Jul 2024, 15:38 Martin Čech, <cech.marten@gmail.com> wrote:
Hi Vlad,
it was me who mentioned the library sync. I've heard of similar cases that Hans-Rudolf is describing and some more sophisticated ones that tried to do bi-directional sync. I couldn't find a link or any more information so I asked in one of the Galaxy channels -- maybe someone has a better memory ( link <https://matrix.to/#/!fVQwotRzniGGJbeUUi:gitter.im/$-y1pfvymkrrvGXOvxhVwVky4ZWdNOI1lGM94iZFztEU?via=gitter.im&via=matrix.org>). The thing is, that it is likely always going to be very custom and heavily based on what your other systems look like. Long time ago we had attempted to implement more full-featured LIMS in Galaxy, but that has long been gone since it was always just too general.
Cheers, Martin
On Fri, Jul 19, 2024 at 3:03 PM VLAD VISAN < vlad.visan@univ-grenoble-alpes.fr> wrote:
Hi Hans, all,
Thank you for sharing your solution. I've been looking into the API and have used this endpoint <https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_> <https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_>(strangely BioBlend seems to wrap around this less expressive one <https://usegalaxy.org/api/docs#/undocumented/post_api_libraries__library_id__contents> instead), very useful! If enough people have this synchronisation use-case, there could be a case for a Galaxy-provided link between a File Source (assuming your LIMS can be imported through one) and a Data Library.
Regards,
Vlad
------------------------------ *De: *"Hans-Rudolf Hotz" <hrh@fmi.ch> *À: *"VLAD VISAN" <vlad.visan@univ-grenoble-alpes.fr>, "small-scale-admins" <small-scale-admins@lists.galaxyproject.org> *Envoyé: *Vendredi 19 Juillet 2024 14:07:08 *Objet: *Re: [Small-scale-admins] Re: Today: Meeting of 'small scale servers' group
Hi Vlad
* I don't remember who mentioned a/several project(s) that synchronise data libraries with a given folder?
well, since I did not attend yesterday's meeting, I couldn't mention it...
In our case, all fastq files generated by our sequencing facility are linked (not copied) to Data Libraries. There is a python script (running every night) checking the facilitiy's LIMS for any new data. And then based on the information in the LIMS, several calls to the Galaxy API generate new sub-folders and link the data.
So it is a very customized solution fitting our local LIMS, storage set-up, etc.
Regards, Hans-Rudolf _______________________________________________ Small-scale-admins mailing list -- small-scale-admins@lists.galaxyproject.org To unsubscribe send an email to small-scale-admins-leave@lists.galaxyproject.org
_______________________________________________ Small-scale-admins mailing list -- small-scale-admins@lists.galaxyproject.org To unsubscribe send an email to small-scale-admins-leave@lists.galaxyproject.org
Hi all, Thank you Martin and Helena for the information. From [ https://github.com/usegalaxy-eu/shared-data | https://github.com/usegalaxy-eu/shared-data ] I found a few useful links: * [ https://ephemeris.readthedocs.io/en/latest/commands/setup-data-libraries.htm... | a util tool ] to populate a data library from the command line * [ https://github.com/galaxyproject/training-material/blob/3a1c6f78aae2e4b7b010... | an example of the format ] expected for the above tool Regards, Vlad De: "Helena Rasche" <helena.rasche@gmail.com> À: "Martin Čech" <cech.marten@gmail.com> Cc: "VLAD VISAN" <vlad.visan@univ-grenoble-alpes.fr>, "Hans-Rudolf Hotz" <hrh@fmi.ch>, "small-scale-admins" <small-scale-admins@lists.galaxyproject.org> Envoyé: Vendredi 19 Juillet 2024 16:50:52 Objet: Re: [Small-scale-admins] Re: Today: Meeting of 'small scale servers' group Hi Vlad, all, [ https://github.com/usegalaxy-eu/shared-data | https://github.com/usegalaxy-eu/shared-data ] was one attempt at uni-directional sync, yaml files describing data / URLs are kept continually in sync with target galaxies. Maybe not something very reusable but potentially useful as inspiration? mvg Helena On Fri, 19 Jul 2024, 15:38 Martin Čech, < [ mailto:cech.marten@gmail.com | cech.marten@gmail.com ] > wrote: Hi Vlad, it was me who mentioned the library sync. I've heard of similar cases that Hans-Rudolf is describing and some more sophisticated ones that tried to do bi-directional sync. I couldn't find a link or any more information so I asked in one of the Galaxy channels -- maybe someone has a better memory ( [ https://matrix.to/#/!fVQwotRzniGGJbeUUi:gitter.im/$-y1pfvymkrrvGXOvxhVwVky4ZWdNOI1lGM94iZFztEU?via=gitter.im&via=matrix.org | link ] ). The thing is, that it is likely always going to be very custom and heavily based on what your other systems look like. Long time ago we had attempted to implement more full-featured LIMS in Galaxy, but that has long been gone since it was always just too general. Cheers, Martin On Fri, Jul 19, 2024 at 3:03 PM VLAD VISAN < [ mailto:vlad.visan@univ-grenoble-alpes.fr | vlad.visan@univ-grenoble-alpes.fr ] > wrote: BQ_BEGIN Hi Hans, all, Thank you for sharing your solution. I've been looking into the API and have used [ https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_ | this endpoint ] [ https://usegalaxy.org/api/docs#/undocumented/post_api_libraries_datasets_ ] (strangely BioBlend seems to wrap around [ https://usegalaxy.org/api/docs#/undocumented/post_api_libraries__library_id_... | this less expressive one ] instead), very useful! If enough people have this synchronisation use-case, there could be a case for a Galaxy-provided link between a File Source (assuming your LIMS can be imported through one) and a Data Library. Regards, Vlad De: "Hans-Rudolf Hotz" < [ mailto:hrh@fmi.ch | hrh@fmi.ch ] > À: "VLAD VISAN" < [ mailto:vlad.visan@univ-grenoble-alpes.fr | vlad.visan@univ-grenoble-alpes.fr ] >, "small-scale-admins" < [ mailto:small-scale-admins@lists.galaxyproject.org | small-scale-admins@lists.galaxyproject.org ] > Envoyé: Vendredi 19 Juillet 2024 14:07:08 Objet: Re: [Small-scale-admins] Re: Today: Meeting of 'small scale servers' group Hi Vlad
* I don't remember who mentioned a/several project(s) that synchronise data libraries with a given folder?
well, since I did not attend yesterday's meeting, I couldn't mention it... In our case, all fastq files generated by our sequencing facility are linked (not copied) to Data Libraries. There is a python script (running every night) checking the facilitiy's LIMS for any new data. And then based on the information in the LIMS, several calls to the Galaxy API generate new sub-folders and link the data. So it is a very customized solution fitting our local LIMS, storage set-up, etc. Regards, Hans-Rudolf _______________________________________________ Small-scale-admins mailing list -- [ mailto:small-scale-admins@lists.galaxyproject.org | small-scale-admins@lists.galaxyproject.org ] To unsubscribe send an email to [ mailto:small-scale-admins-leave@lists.galaxyproject.org | small-scale-admins-leave@lists.galaxyproject.org ] _______________________________________________ Small-scale-admins mailing list -- [ mailto:small-scale-admins@lists.galaxyproject.org | small-scale-admins@lists.galaxyproject.org ] To unsubscribe send an email to [ mailto:small-scale-admins-leave@lists.galaxyproject.org | small-scale-admins-leave@lists.galaxyproject.org ] BQ_END
participants (5)
-
Delisle Lucille -
Hans-Rudolf Hotz -
Helena Rasche -
Martin Čech -
VLAD VISAN