On 11 January 2016 at 17:32, John Chilton <jmchilton@gmail.com> wrote:

So job metrics are completely different than what you described here.
Job metrics are enabled by default and can be configured fairly
directly by modifying config/job_metrics_conf.xml. New plugins are
relatively easy to write and can be added as python files to
lib/galaxy/jobs/metrics/instrumenters/.

What you described is what is used on usegalaxy.org to allow users to
have some input on how jobs are scheduled. I call these job resources
parameters - in job_conf.xml particular tools can be mapped to these
parameters and - yes you are correct dynamic job destination functions
can consume them.

The dynamic job destinations definitely shouldn't be accessing these
through job.parameters keyed on __job_resource. The dynamic
destination function should just take in a parameter named
resource_params and then the function will be passed in.

Dynamic Destinations are documented here:
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

You are correct job resource parameters have not been documented*,
there is an open issue here:
https://github.com/galaxyproject/galaxy/issues/1266

* It should be noted that while this a frequently requested feature
and used on usegalaxy.org - the feature is somewhat controversial. The
thinking among some is that is exposes a level of detail that Galaxy
is supposed to hide. I don't share this particular concern, but the
implementation does worry me - what it does to persisted job state and
parameter handling is fairly hacky and I'm somewhat surprised we
haven't encountered more problems. This is part of the reason not why
this hasn't been documented, but why I feel less bad that it hasn't
been documented than other stuff I have done that also hasn't been
documented :|.

-John

On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <pvh@sanbi.ac.za> wrote:
> Hi there
>
> I'm trying to understand how the job metrics configuration works. Reading
> through various examples, it looks like it works together with the dynamic
> destination support.
>
> First on the dynamic destination support, this works, as I understand it, by
> importing modules (files ending in .py) from the directory set as the
> rules_module, where rules_module is a parameter for the dynamic job plugin.
> As in this snipped from the usegalaxy configuration:
>
> <plugin id="dynamic" type="runner">
> 
> <param id="rules_module">usegalaxy.jobs.rules</param>
> </plugin>
>
>
> So import usegalaxy.jobs.rules, find where this resolves to as a directory,
> find all .py files in that directory. If rules_module is not specified, the
> default is galaxy.jobs.rules (under galaxy/lib).
>
> So these .py files are imported and within them are functions that are used
> in dynamic destinations (those with runner="dynamic") and in turn tools are
> directed to use these destinations.
>
> Now the job metrics configuration supplies both a user interface and then a
> set of keys that are made accessible to dynamic runners as key value pairs
> on the job.parameters attribute (passed in as the third argument, i.e. job).
> Specifically the job metrics configuration exists under a key
> '__job_resource'.
>
> Is this all correct? Is there a more straightforward way of moving from job
> resource specification to job configuration than this?
>
> Thanks,
> Peter
>

> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/