Dear All,
During the ESG annual meeting in Freiburg we discussed more in detail about the meta scheduling implementation.
So far we have been discussing adding this to DIRAC. However, the UseGalaxy.eu team felt that it would be better to create a separate API endpoint for the meta scheduling (specifically using https://fastapi.tiangolo.com/ and https://www.openapis.org/). We all agreed this is a better approach.
Also, we discussed that TPV has a global overview of all destinations in Galaxy, and DIRAC only sees the Compute Elements under a Virtual Organistaion. Specifically, this is where the scheduling happens in TPV, after it has filtered out all the candidate destinations for a given job (tool): https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/files/galaxy/tpv/tool_defaults.yml#L47
It is that part of the code that will call the new API endpoint to return the best destination.
We informally discussed that @paul.degeest@psb.vib-ugent.be will contact David Lopez (UseGalaxy.eu) to get started with the FastAPI implementation. @Abdulrahman Azab will be contributing his code for the meta scheduling. I assume that the best first step is to create a GitHub repository under https://github.com/usegalaxy-eu so we can then point interested people to it?
I have added this to our shared T4.3 document:
Please let me know if there are any questions.
Best regards,
Sebastian
--
Sebastian Luna-Valero (he/him/his) | Cloud Community Support Specialist | EGI Foundation (Amsterdam, The Netherlands)
W: www.egi.eu | E: sebastian.luna.valero@egi.eu | Twitter: SLunaValero
EGI: Advanced Computing for Research
The EGI Foundation is ISO 9001:2015 and ISO/IEC 20000-1:2018 certified