Job metrics configuration
Hi there I'm trying to understand how the job metrics configuration works. Reading through various examples, it looks like it works together with the dynamic destination support. First on the dynamic destination support, this works, as I understand it, by importing modules (files ending in .py) from the directory set as the rules_module, where rules_module is a parameter for the dynamic job plugin. As in this snipped from the usegalaxy configuration: <plugin id="dynamic" type="runner"> <!-- These live in the virtualenv --> <param id="rules_module">usegalaxy.jobs.rules</param> </plugin> So import usegalaxy.jobs.rules, find where this resolves to as a directory, find all .py files in that directory. If rules_module is not specified, the default is galaxy.jobs.rules (under galaxy/lib). So these .py files are imported and within them are functions that are used in dynamic destinations (those with runner="dynamic") and in turn tools are directed to use these destinations. Now the job metrics configuration supplies both a user interface and then a set of keys that are made accessible to dynamic runners as key value pairs on the job.parameters attribute (passed in as the third argument, i.e. job). Specifically the job metrics configuration exists under a key '__job_resource'. Is this all correct? Is there a more straightforward way of moving from job resource specification to job configuration than this? Thanks, Peter
So job metrics are completely different than what you described here. Job metrics are enabled by default and can be configured fairly directly by modifying config/job_metrics_conf.xml. New plugins are relatively easy to write and can be added as python files to lib/galaxy/jobs/metrics/instrumenters/. What you described is what is used on usegalaxy.org to allow users to have some input on how jobs are scheduled. I call these job resources parameters - in job_conf.xml particular tools can be mapped to these parameters and - yes you are correct dynamic job destination functions can consume them. The dynamic job destinations definitely shouldn't be accessing these through job.parameters keyed on __job_resource. The dynamic destination function should just take in a parameter named resource_params and then the function will be passed in. Dynamic Destinations are documented here: https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping You are correct job resource parameters have not been documented*, there is an open issue here: https://github.com/galaxyproject/galaxy/issues/1266 * It should be noted that while this a frequently requested feature and used on usegalaxy.org - the feature is somewhat controversial. The thinking among some is that is exposes a level of detail that Galaxy is supposed to hide. I don't share this particular concern, but the implementation does worry me - what it does to persisted job state and parameter handling is fairly hacky and I'm somewhat surprised we haven't encountered more problems. This is part of the reason not why this hasn't been documented, but why I feel less bad that it hasn't been documented than other stuff I have done that also hasn't been documented :|. -John On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <pvh@sanbi.ac.za> wrote:
Hi there
I'm trying to understand how the job metrics configuration works. Reading through various examples, it looks like it works together with the dynamic destination support.
First on the dynamic destination support, this works, as I understand it, by importing modules (files ending in .py) from the directory set as the rules_module, where rules_module is a parameter for the dynamic job plugin. As in this snipped from the usegalaxy configuration:
<plugin id="dynamic" type="runner"> <!-- These live in the virtualenv --> <param id="rules_module">usegalaxy.jobs.rules</param> </plugin>
So import usegalaxy.jobs.rules, find where this resolves to as a directory, find all .py files in that directory. If rules_module is not specified, the default is galaxy.jobs.rules (under galaxy/lib).
So these .py files are imported and within them are functions that are used in dynamic destinations (those with runner="dynamic") and in turn tools are directed to use these destinations.
Now the job metrics configuration supplies both a user interface and then a set of keys that are made accessible to dynamic runners as key value pairs on the job.parameters attribute (passed in as the third argument, i.e. job). Specifically the job metrics configuration exists under a key '__job_resource'.
Is this all correct? Is there a more straightforward way of moving from job resource specification to job configuration than this?
Thanks, Peter
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi there John Thanks for the clarification. My initial aim was, exactly, to expose those parameters that, as you say, some people don't think should be exposed. My use case is to allow jobs with different native parameters to be scheduled on our clouster from our Galaxy server. Perhaps a better approach is to use dynamic destinations that try and estimate appropriate values for e.g. hvmem, but at present our users are used to setting such parameters themselves. Peter On 11 January 2016 at 17:32, John Chilton <jmchilton@gmail.com> wrote:
So job metrics are completely different than what you described here. Job metrics are enabled by default and can be configured fairly directly by modifying config/job_metrics_conf.xml. New plugins are relatively easy to write and can be added as python files to lib/galaxy/jobs/metrics/instrumenters/.
What you described is what is used on usegalaxy.org to allow users to have some input on how jobs are scheduled. I call these job resources parameters - in job_conf.xml particular tools can be mapped to these parameters and - yes you are correct dynamic job destination functions can consume them.
The dynamic job destinations definitely shouldn't be accessing these through job.parameters keyed on __job_resource. The dynamic destination function should just take in a parameter named resource_params and then the function will be passed in.
Dynamic Destinations are documented here:
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping
You are correct job resource parameters have not been documented*, there is an open issue here: https://github.com/galaxyproject/galaxy/issues/1266
* It should be noted that while this a frequently requested feature and used on usegalaxy.org - the feature is somewhat controversial. The thinking among some is that is exposes a level of detail that Galaxy is supposed to hide. I don't share this particular concern, but the implementation does worry me - what it does to persisted job state and parameter handling is fairly hacky and I'm somewhat surprised we haven't encountered more problems. This is part of the reason not why this hasn't been documented, but why I feel less bad that it hasn't been documented than other stuff I have done that also hasn't been documented :|.
-John
Hi there
I'm trying to understand how the job metrics configuration works. Reading through various examples, it looks like it works together with the dynamic destination support.
First on the dynamic destination support, this works, as I understand it, by importing modules (files ending in .py) from the directory set as the rules_module, where rules_module is a parameter for the dynamic job
As in this snipped from the usegalaxy configuration:
<plugin id="dynamic" type="runner"> <!-- These live in the virtualenv --> <param id="rules_module">usegalaxy.jobs.rules</param> </plugin>
So import usegalaxy.jobs.rules, find where this resolves to as a
find all .py files in that directory. If rules_module is not specified,
default is galaxy.jobs.rules (under galaxy/lib).
So these .py files are imported and within them are functions that are used in dynamic destinations (those with runner="dynamic") and in turn tools are directed to use these destinations.
Now the job metrics configuration supplies both a user interface and
set of keys that are made accessible to dynamic runners as key value
On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <pvh@sanbi.ac.za> wrote: plugin. directory, the then a pairs
on the job.parameters attribute (passed in as the third argument, i.e. job). Specifically the job metrics configuration exists under a key '__job_resource'.
Is this all correct? Is there a more straightforward way of moving from job resource specification to job configuration than this?
Thanks, Peter
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (2)
-
John Chilton -
Peter van Heusden