Galaxy cluster jobs - automatic priority setting
Dear all, Does Galaxy have any mechanisms to set the priority of jobs submitted to the cluster? Specifically I am interested in SGE via DRMAA, but this is a general issue. If there is some existing code, I might be able to use it for the following situation: The specific motivation is for balancing competing BLAST searches from multiple users: I want to be able to prioritize short jobs (e.g. under 100 queries) over large jobs (tens of thousands of queries). In the (experimental) task splitting code, I would like to be able to give short jobs higher priority (e.g. normal) than large jobs (e.g. low priority) based on the size of the split file. One idea would be a scaling bases on the (split) input file's size (in bytes), or perhaps a per-file format size threshold, e.g. when splitting FASTA files, 1000 sequences might trigger lower priority. The advantage of this kind of assessment is it would also work on both the current split mechanisms, "number_of_parts" and "to_size", as well as any hybrid of the two: <parallelism method="multi" split_mode="number_of_parts" split_size="8" ..." /> <parallelism method="multi" split_mode="to_size" split_size="1000" ... /> http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009647.html So, is there any existing code in Galaxy that would be helpful here - or existing plans in this area? Thanks, Peter
On Jun 4, 2012, at 7:24 AM, Peter Cock wrote:
Dear all,
Does Galaxy have any mechanisms to set the priority of jobs submitted to the cluster? Specifically I am interested in SGE via DRMAA, but this is a general issue. If there is some existing code, I might be able to use it for the following situation:
The specific motivation is for balancing competing BLAST searches from multiple users: I want to be able to prioritize short jobs (e.g. under 100 queries) over large jobs (tens of thousands of queries).
In the (experimental) task splitting code, I would like to be able to give short jobs higher priority (e.g. normal) than large jobs (e.g. low priority) based on the size of the split file.
One idea would be a scaling bases on the (split) input file's size (in bytes), or perhaps a per-file format size threshold, e.g. when splitting FASTA files, 1000 sequences might trigger lower priority.
The advantage of this kind of assessment is it would also work on both the current split mechanisms, "number_of_parts" and "to_size", as well as any hybrid of the two:
<parallelism method="multi" split_mode="number_of_parts" split_size="8" ..." /> <parallelism method="multi" split_mode="to_size" split_size="1000" ... /> http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009647.html
So, is there any existing code in Galaxy that would be helpful here - or existing plans in this area?
Hi Peter, Nothing existing short of separate copies of the same tool, one to be used for short jobs, and one for longer jobs. It should be possible with John Chilton's dynamic job runners[1], which I will accept and merge any day now. --nate [1] https://bitbucket.org/galaxy/galaxy-central/pull-request/12/dynamic-job-runn...
Thanks,
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
On Fri, Jun 8, 2012 at 8:41 PM, Nate Coraor <nate@bx.psu.edu> wrote:
On Jun 4, 2012, at 7:24 AM, Peter Cock wrote:
Dear all,
Does Galaxy have any mechanisms to set the priority of jobs submitted to the cluster? Specifically I am interested in SGE via DRMAA, but this is a general issue. If there is some existing code, I might be able to use it for the following situation:
The specific motivation is for balancing competing BLAST searches from multiple users: I want to be able to prioritize short jobs (e.g. under 100 queries) over large jobs (tens of thousands of queries).
In the (experimental) task splitting code, I would like to be able to give short jobs higher priority (e.g. normal) than large jobs (e.g. low priority) based on the size of the split file.
One idea would be a scaling bases on the (split) input file's size (in bytes), or perhaps a per-file format size threshold, e.g. when splitting FASTA files, 1000 sequences might trigger lower priority.
The advantage of this kind of assessment is it would also work on both the current split mechanisms, "number_of_parts" and "to_size", as well as any hybrid of the two:
<parallelism method="multi" split_mode="number_of_parts" split_size="8" ..." /> <parallelism method="multi" split_mode="to_size" split_size="1000" ... /> http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-May/009647.html
So, is there any existing code in Galaxy that would be helpful here - or existing plans in this area?
Hi Peter,
Nothing existing short of separate copies of the same tool, one to be used for short jobs, and one for longer jobs. It should be possible with John Chilton's dynamic job runners[1], which I will accept and merge any day now.
--nate
Hi Nate, As per my reply on John's thread, http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010088.html I'm looking forward to seeing this merged: https://bitbucket.org/galaxy/galaxy-central/pull-request/47/ Peter
participants (2)
-
Nate Coraor -
Peter Cock