Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: eca238d7837928c7ee962f10435fd667d61ccc4c https://github.com/galaxyproject/galaxy/commit/eca238d7837928c7ee962f10435fd... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M config/job_conf.xml.sample_advanced M lib/galaxy/dependencies/__init__.py M lib/galaxy/dependencies/conditional-requirements.txt Log Message: ----------- Make pykube a conditional requirement Commit: c82cd04ca820d65c9279872c029d712358e4c65f https://github.com/galaxyproject/galaxy/commit/c82cd04ca820d65c9279872c029d7... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Use standard os.environ Commit: 879d7c22b63733dfd5f9e1c6cbe368c97e06495d https://github.com/galaxyproject/galaxy/commit/879d7c22b63733dfd5f9e1c6cbe36... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/base/integration_util.py A test/integration/test_kubernetes_runner.py Log Message: ----------- Add basic kubernetes integration test Currently times out becasue jobs aren't monitored succesfully. Commit: 4eed32dc4c86da17a525419ba985d4d06d7a768b https://github.com/galaxyproject/galaxy/commit/4eed32dc4c86da17a525419ba985d... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/cli.py M lib/galaxy/jobs/runners/condor.py M lib/galaxy/jobs/runners/drmaa.py M lib/galaxy/jobs/runners/local.py M lib/galaxy/jobs/runners/pbs.py Log Message: ----------- Propagate selected shell to job_script Commit: 912fc61592a92fbed329501b38b35a591e556b2d https://github.com/galaxyproject/galaxy/commit/912fc61592a92fbed329501b38b35... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Allow skipping pod_retrial Commit: e7b89122489083f80b0c183ff0a958a58d823118 https://github.com/galaxyproject/galaxy/commit/e7b89122489083f80b0c183ff0a95... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Use galaxy's exisiting job_file abstraction This allows setting environment variables, job instrumentation, exit code checking and so on. Commit: 3498f87d8734ce5bc3341e546842dabe29b93d40 https://github.com/galaxyproject/galaxy/commit/3498f87d8734ce5bc3341e546842d... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/integration/test_kubernetes_runner.py Log Message: ----------- Use sh as job shell for kubernetes test, bash is not available in default container Commit: 5dd10c4f86b07d9483b76c59511148bdce8949d3 https://github.com/galaxyproject/galaxy/commit/5dd10c4f86b07d9483b76c5951114... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/tools/deps/dependencies.py Log Message: ----------- Drop mutable defaults Commit: 1a007682737af1b466a6a8545922e17c7f42fb19 https://github.com/galaxyproject/galaxy/commit/1a007682737af1b466a6a8545922e... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Skip broken string encoding logic Commit: d2ed543ecd39fee1876fcbad063c1aa4c4c3ce33 https://github.com/galaxyproject/galaxy/commit/d2ed543ecd39fee1876fcbad063c1... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/chronos.py M lib/galaxy/jobs/runners/godocker.py M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Remove return statements from queue_job, this is never consumed anywhere Commit: 1103ce1dccd0ac3dda0dca715da4cb0feafde4ee https://github.com/galaxyproject/galaxy/commit/1103ce1dccd0ac3dda0dca715da4c... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Remove commented out non-functional code in kubernetes runner. Commit: a23063b2dc3fdf82c645383fd36c3b6439b68f52 https://github.com/galaxyproject/galaxy/commit/a23063b2dc3fdf82c645383fd36c3... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Cleanup of kubernetes exception logging Commit: 4e39c0fc8488a4794870752f8895f61dc8d704cf https://github.com/galaxyproject/galaxy/commit/4e39c0fc8488a4794870752f8895f... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py M test/integration/test_containerized_jobs.py M test/integration/test_job_recovery.py M test/integration/test_kubernetes_runner.py Log Message: ----------- Test cancelling jobs, running mulled containers on kubernetes Commit: 9019916ae2180afe6f35b6ead45f1ae294c2bfe9 https://github.com/galaxyproject/galaxy/commit/9019916ae2180afe6f35b6ead45f1... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/integration/test_kubernetes_runner.py Log Message: ----------- Fix test skipping Commit: 34c0a2940b69c59b20bbdd8df0944332cb59e64c https://github.com/galaxyproject/galaxy/commit/34c0a2940b69c59b20bbdd8df0944... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Don't write fake stderr file Commit: 5c5f199f812ff68e7d061c047956aeaa3627632f https://github.com/galaxyproject/galaxy/commit/5c5f199f812ff68e7d061c047956a... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/integration/test_kubernetes_runner.py Log Message: ----------- Add job properties test Commit: b43f1985d71fadf3f907feeab2b529408dc05588 https://github.com/galaxyproject/galaxy/commit/b43f1985d71fadf3f907feeab2b52... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/command_factory.py M lib/galaxy/jobs/runners/__init__.py M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Capture stdout and stderr for kubernetes runner There is currently no way to discriminate stdout and stderr from kubernetes logs (https://github.com/kubernetes/kubernetes/issues/28167), but Galaxy really needs to know about the stderr. To work around this we capture the stdout and stderr into files using the approach outined in https://stackoverflow.com/a/692407. Commit: d0e749375a2e703f9f59e8794825867760ee7b44 https://github.com/galaxyproject/galaxy/commit/d0e749375a2e703f9f59e87948258... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Drop custom fail_job and stderr logging methods We now log stdout and stderr to files, so this isn't needed. Commit: 98b571e19aec1b182dbb636d23ab8a3ff290f30d https://github.com/galaxyproject/galaxy/commit/98b571e19aec1b182dbb636d23ab8... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Remove user-cancelled jobs from monitor queue Commit: dc184567376ec25eba3d317c9a70d2a27d4aefed https://github.com/galaxyproject/galaxy/commit/dc184567376ec25eba3d317c9a70d... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: A lib/galaxy/util/bytesize.py Log Message: ----------- Add utility class for manipulating byte sizes Commit: c61c790fccc84f06991d48dcbea70ef8181fe452 https://github.com/galaxyproject/galaxy/commit/c61c790fccc84f06991d48dcbea70... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py M lib/galaxy/jobs/runners/util/job_script/CLUSTER_SLOTS_STATEMENT.sh M test/functional/tools/samples_tool_conf.xml M test/integration/test_kubernetes_runner.py Log Message: ----------- Make kubernetes pass GALAXY_SLOTS and GALAXY_MEMORY_MB Also drops __transform_cpu_value, incoming values are always strings, so this wasn't functional in any case. Commit: fe02dbbf8107b8ded12cb9c76fd1686515a47dd4 https://github.com/galaxyproject/galaxy/commit/fe02dbbf8107b8ded12cb9c76fd16... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M config/job_conf.xml.sample_advanced Log Message: ----------- Fix typo in config/job_conf.xml.sample_advanced Commit: 54c964011a29eca1c72360f17f5114022ad626bf https://github.com/galaxyproject/galaxy/commit/54c964011a29eca1c72360f17f511... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py M test/integration/test_kubernetes_runner.py Log Message: ----------- Take ceiling of kubernetes cpu value Commit: 936679ea817151438ff76d2e14ac2557c4caa3a0 https://github.com/galaxyproject/galaxy/commit/936679ea817151438ff76d2e14ac2... Author: Pablo Moreno <pablo.a.moreno@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Inject env vars for k8s job to avoid changing tool environment inside container. Commit: e52bb5daa78eb040304919299dc624491573c53f https://github.com/galaxyproject/galaxy/commit/e52bb5daa78eb040304919299dc62... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/integration/test_kubernetes_runner.py Log Message: ----------- Import from correct module Commit: 626657e3c2b8dc3325a4c2dee35777ac45ac8f0c https://github.com/galaxyproject/galaxy/commit/626657e3c2b8dc3325a4c2dee3577... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/command_factory.py M lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh M lib/galaxy/jobs/runners/util/job_script/__init__.py M test/unit/jobs/test_command_factory.py Log Message: ----------- Revert "job script returns tool exit code in any case" This breaks exit code reporting and leads to pod re-scheduling. This reverts commit 6beeec836f6c6ab479700e6a1e1accbe5069dacf. Commit: 7cf161ba0155abf02582cbbf4636d1ca8681efbe https://github.com/galaxyproject/galaxy/commit/7cf161ba0155abf02582cbbf4636d... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M config/job_conf.xml.sample_advanced M lib/galaxy/jobs/runners/kubernetes.py M test/integration/test_kubernetes_runner.py Log Message: ----------- Allow adding multiple volume claims Commit: e293a74e66002a28adc1045fda0e3415c6f2e08b https://github.com/galaxyproject/galaxy/commit/e293a74e66002a28adc1045fda0e3... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/integration/test_kubernetes_runner.py Log Message: ----------- Add test for tool using python dependencies Commit: 32cad5977b102c33088111abe80887ab1a567a19 https://github.com/galaxyproject/galaxy/commit/32cad5977b102c33088111abe8088... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Set galaxy_virtual_env to None for kubernetes runner This prevents attempts of activating Galaxy's virtualenv inside the container. Commit: b931576ed1ebcdfc0f9e0d5cbab74d199e8bbb2a https://github.com/galaxyproject/galaxy/commit/b931576ed1ebcdfc0f9e0d5cbab74... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M lib/galaxy/jobs/runners/kubernetes.py Log Message: ----------- Drop GALAXY_VIRTUAL_ENV env hack GALAXY_VIRTUAL_ENV is injected through job_wrapper.galaxy_virtual_env, which does `os.environ.get(VIRTUAL_ENV, None)`, so this didn't work. The previous commit addresses that. Commit: db387ab898cf47076279ac9d97407ba0fdc45e49 https://github.com/galaxyproject/galaxy/commit/db387ab898cf47076279ac9d97407... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2019-02-19 (Tue, 19 Feb 2019) Changed paths: M test/unit/jobs/test_command_factory.py Log Message: ----------- Fix typo fixed by reverted commit 6beeec8 Commit: c53fb623d8cf3af99732a3fe9d1f513f9b538c2b https://github.com/galaxyproject/galaxy/commit/c53fb623d8cf3af99732a3fe9d1f5... Author: John Chilton <jmchilton@gmail.com> Date: 2019-02-22 (Fri, 22 Feb 2019) Changed paths: M config/job_conf.xml.sample_advanced M lib/galaxy/dependencies/__init__.py M lib/galaxy/dependencies/conditional-requirements.txt M lib/galaxy/jobs/command_factory.py M lib/galaxy/jobs/runners/__init__.py M lib/galaxy/jobs/runners/chronos.py M lib/galaxy/jobs/runners/cli.py M lib/galaxy/jobs/runners/condor.py M lib/galaxy/jobs/runners/drmaa.py M lib/galaxy/jobs/runners/godocker.py M lib/galaxy/jobs/runners/kubernetes.py M lib/galaxy/jobs/runners/local.py M lib/galaxy/jobs/runners/pbs.py M lib/galaxy/jobs/runners/util/job_script/CLUSTER_SLOTS_STATEMENT.sh M lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh M lib/galaxy/jobs/runners/util/job_script/__init__.py M lib/galaxy/tools/deps/dependencies.py A lib/galaxy/util/bytesize.py M test/base/integration_util.py M test/functional/tools/samples_tool_conf.xml M test/integration/test_containerized_jobs.py M test/integration/test_job_recovery.py A test/integration/test_kubernetes_runner.py M test/unit/jobs/test_command_factory.py Log Message: ----------- Merge pull request #6958 from mvdbeek/kubernetes_integration_test Kubernetes job runner integration test and enhancements Compare: https://github.com/galaxyproject/galaxy/compare/65f0c2ac8da8...c53fb623d8cf