Hello,
I have a question regarding the job queuing/scheduling in Galaxy, because although I assured my users that the scheduling is fair (using the "round robin" scheduler), what I'm seeing is not exactly fair (and so users wait a long time for their jobs to run).
I'll try to explain the situation as best as I can, and I'll appreciate any feedback.
The attached diagram shows a situation similar to what I have.
At time "t0", user-A submits a workflow with many steps (the lines represent input-output connections, the numbers are job numbers). After a while, jobs 1,2,3 are done (green). and jobs 4,5 are running (yellow) - assuming I can run only two jobs at a time. If I understand correctly, jobs 6-12 are queued (i.e. their input dataset is "ready", so they are ready to run, and put on my local runner queue). Job 13 is still "new" because its input datasets (11,12,13) are not ready.
At time "t1", (while jobs 4,5 are still running), user-B submits a single job (#14). If I understand correctly, it will go from "new" to "queued" immediately, because it isn't dependent on any input dataset and is ready to run.
My question is: Once jobs 4 or 5 are completed, which job will run next ? If everything is "fair", I'd expect job #14 (from the second user) to run.
But what I think I'm seeing, is that the "queued jobs" queue is actually FIFO, and jobs 6-12 will be run before job 14. (Job 14 however will run before job 13),
Is the "round-robin" queue refers only to the transition from "new" to "queued" ?
I could be completely wrong about this whole thing, but what I'm experiencing is that user-A submitted 5 workflows in parallel (all look very similar to the attached diagram), I have 7 runnings jobs (of user-A), 35 queued job from user-A and 2 queued job from user-B, and galaxy consistently chooses to run queued job from user-A instead of user-B (presumably because they were queued before user-B submitted the jobs).
Is there a way to work around this issue ? a different configuration maybe ? Or, an old request: is it possible to limit the number of jobs-per-user at any single time (i.e. a single user can run 3 jobs at any given time, even if no other users are running jobs and there are 7 workers ready) ?
Thanks for reading so far, -gordon
