Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 8d8bf5eaffeda0f229bf56cb45e4adbb05b12d57 https://github.com/galaxyproject/galaxy/commit/8d8bf5eaffeda0f229bf56cb45e4a... Author: John Chilton jmchilton@gmail.com Date: 2018-01-09 (Tue, 09 Jan 2018)

Changed paths: M lib/galaxy/config.py M test/base/driver_util.py A test/functional/tools/mulled_example_explicit.xml A test/functional/tools/mulled_example_simple.xml M test/functional/tools/samples_tool_conf.xml A test/integration/dockerized_job_conf.xml A test/integration/test_dockerized_jobs.py

Log Message: ----------- Dockerized job integration testing.

Add simple test cases for both mulled containers and explicit Docker image resolution.

Commit: cd04ff30665ae6f15a6af35152af54d630458e60 https://github.com/galaxyproject/galaxy/commit/cd04ff30665ae6f15a6af35152af5... Author: John Chilton jmchilton@gmail.com Date: 2018-01-09 (Tue, 09 Jan 2018)

Changed paths: M lib/galaxy/tools/deps/docker_util.py

Log Message: ----------- Set group id in addition to user id when running docker containers by default.

Noticed cwltool does this and it is a good idea. It wasn't the bug I was tracking but it is a solid improvement.

Commit: 4bf47d08770c14010bf925c73ccba2262dfe78ee https://github.com/galaxyproject/galaxy/commit/4bf47d08770c14010bf925c73ccba... Author: John Chilton jmchilton@gmail.com Date: 2018-01-09 (Tue, 09 Jan 2018)

Changed paths: M config/job_conf.xml.sample_advanced M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/jobs/rule_helper.py M lib/galaxy/jobs/runners/__init__.py M lib/galaxy/jobs/runners/pulsar.py M lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh M lib/galaxy/jobs/runners/util/job_script/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/deps/containers.py M lib/galaxy/tools/deps/docker_util.py M lib/galaxy/tools/evaluation.py M lib/galaxy/tools/parser/interface.py M lib/galaxy/tools/parser/xml.py M lib/galaxy/tools/xsd/galaxy.xsd M test/base/integration_util.py A test/functional/tools/job_environment_default.xml A test/functional/tools/job_environment_default_legacy.xml A test/functional/tools/job_environment_explicit_shared_home.xml M test/functional/tools/samples_tool_conf.xml A test/integration/sets_tmp_dir_expression_job_conf.xml A test/integration/sets_tmp_dir_to_true_job_conf.xml A test/integration/simple_job_conf.xml M test/integration/test_dockerized_jobs.py A test/integration/test_job_environments.py M test/unit/jobs/test_runner_local.py M test/unit/tools/test_evaluation.py

Log Message: ----------- Improved home and temp directory handling.

Home Directory Handling -----------------------

- For profile < 18.01 tools: - If use_shared_home="false" is set on the command block - the tool will be given a clean $HOME directory. - If ``shared_home_dir`` is set in Galaxy's config or the job destination configuration, Galaxy will set $HOME in the tool's job environment to point at this directory. - For profile >= 18.01 tools, jobs will be given a clean home directory by default unless ``use_shared_home="true"`` is set. If that is set, the profile < 18.01 behavior is used.

In addition to these changes for Galaxy tools, the tool framework itself has been updated to allow other potential behaviors including the CWL defaults.

Integration tests for each of these cases has been added.

Upgrading to 18.01 - we recommend dropping overriding ``HOME`` in any Galaxy environment configuration (e.g. env directives in job_conf.xml). If any such configuration is found, we recommend instead setting "shared_home_dir" for destinations to the that cluster destination's shared HOME directory. If all destinations share a single HOME directory, this can be set in galaxy.ini instead of job_conf.xml.

Temp Directory Handling -----------------------

There were serious disagreements with how to proceed here. I felt we should aggressively isolate tool TMP directories and provide deployers options to tweak these through structured arguments. Nicola felt we should defer to the environment variables already being set job_conf.xml - but tweak their meanings by default. Nate fell somewhere in between. I think all sides have merit and could make sense depending on what you want to improve (easing cognative load, reducing out-of-box errors, easing advanced deployer configs, structured reasonsing of paths by Galaxy, etc...).

I've navigated a path here that doesn't particularly reduce out of the box errors as I wanted by setting up a per-job temp directory by default but does make it possible and shouldn't break any existing configurations and doesn't make it anything more difficult to manage these things via environment variables.

The new approach doesn't change any thing by default for Galaxy tools (regardless of tool profile version). The only way to achieve new behaviors is for the deployer to set a job_conf parameter called "tmp_dir". If this is set, it can be set to "True" (to default to a new, clean directory below the job directory) or to a shell expression to allow setting this dynamically at runtime to paths or paths beneath directories as needed on a per-destination basis. If a temp directory is set this way all three variables TMP, TMPDIR, and TEMP are set.

Docker Environment Handling ---------------------------

- Mirror the CWL behavior of mounting an external /tmp by default. - For newer tools, pass through the HOME, TMP, TMPDIR, and TEMP environment variables into the Docker container in addition to GALAXY_SLOTS. - Fix the pass through of environment variables into Docker containers.

Commit: db0633eb8965306f4a7a8cca36e882cffc73ad55 https://github.com/galaxyproject/galaxy/commit/db0633eb8965306f4a7a8cca36e88... Author: John Chilton jmchilton@gmail.com Date: 2018-01-09 (Tue, 09 Jan 2018)

Changed paths: M lib/galaxy/jobs/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/deps/docker_util.py M lib/galaxy/tools/evaluation.py M lib/galaxy/tools/parser/interface.py M test/integration/test_dockerized_jobs.py M test/integration/test_job_environments.py M test/unit/tools/test_evaluation.py

Log Message: ----------- Follow up on comments for #5193.

Commit: 080148a16190988a18df6709ba1bd9f650ea10de https://github.com/galaxyproject/galaxy/commit/080148a16190988a18df6709ba1bd... Author: Nate Coraor nate@bx.psu.edu Date: 2018-01-19 (Fri, 19 Jan 2018)

Changed paths: M config/job_conf.xml.sample_advanced M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/jobs/rule_helper.py M lib/galaxy/jobs/runners/__init__.py M lib/galaxy/jobs/runners/pulsar.py M lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh M lib/galaxy/jobs/runners/util/job_script/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/deps/containers.py M lib/galaxy/tools/deps/docker_util.py M lib/galaxy/tools/evaluation.py M lib/galaxy/tools/parser/interface.py M lib/galaxy/tools/parser/xml.py M lib/galaxy/tools/xsd/galaxy.xsd M test/base/driver_util.py M test/base/integration_util.py A test/functional/tools/job_environment_default.xml A test/functional/tools/job_environment_default_legacy.xml A test/functional/tools/job_environment_explicit_shared_home.xml A test/functional/tools/mulled_example_explicit.xml A test/functional/tools/mulled_example_simple.xml M test/functional/tools/samples_tool_conf.xml A test/integration/dockerized_job_conf.xml A test/integration/sets_tmp_dir_expression_job_conf.xml A test/integration/sets_tmp_dir_to_true_job_conf.xml A test/integration/simple_job_conf.xml A test/integration/test_dockerized_jobs.py A test/integration/test_job_environments.py M test/unit/jobs/test_runner_local.py M test/unit/tools/test_evaluation.py

Log Message: ----------- Merge pull request #5193 from jmchilton/structured_job_environment_1

Improved home and temp directory handling.

Compare: https://github.com/galaxyproject/galaxy/compare/1cac9ad54fea...080148a16190