Branch: refs/heads/dev
Home: https://github.com/galaxyproject/galaxy
Commit: 306b114191a1d326afcbdca68f489b015ba07660
https://github.com/galaxyproject/galaxy/commit/306b114191a1d326afcbdca68f48…
Author: Nicola Soranzo <nicola.soranzo(a)earlham.ac.uk>
Date: 2018-11-17 (Sat, 17 Nov 2018)
Changed paths:
M lib/galaxy/jobs/handler.py
M lib/galaxy/jobs/runners/__init__.py
M lib/galaxy/jobs/runners/chronos.py
M lib/galaxy/jobs/runners/cli.py
M lib/galaxy/jobs/runners/condor.py
M lib/galaxy/jobs/runners/drmaa.py
M lib/galaxy/jobs/runners/godocker.py
M lib/galaxy/jobs/runners/kubernetes.py
M lib/galaxy/jobs/runners/local.py
M lib/galaxy/jobs/runners/pbs.py
M lib/galaxy/jobs/runners/pulsar.py
M lib/galaxy/jobs/runners/tasks.py
M test/unit/jobs/test_runner_local.py
Log Message:
-----------
Pass a ``JobWrapper`` instead of a ``Job`` to ``stop_job()`` methods
Commit: 0ddc5aeba1827833991cd782a9d806fa9ccd414e
https://github.com/galaxyproject/galaxy/commit/0ddc5aeba1827833991cd782a9d8…
Author: Nicola Soranzo <nicola.soranzo(a)earlham.ac.uk>
Date: 2018-11-17 (Sat, 17 Nov 2018)
Changed paths:
M lib/galaxy/jobs/handler.py
M lib/galaxy/jobs/runners/drmaa.py
M lib/galaxy/jobs/runners/local.py
M lib/galaxy/jobs/runners/pulsar.py
M lib/galaxy/jobs/runners/tasks.py
M lib/galaxy/model/__init__.py
Log Message:
-----------
Drop ``get_id()`` method
Commit: 183d84e5dc8ce4530835852292f3d20318dc9980
https://github.com/galaxyproject/galaxy/commit/183d84e5dc8ce4530835852292f3…
Author: Nicola Soranzo <nicola.soranzo(a)earlham.ac.uk>
Date: 2018-11-17 (Sat, 17 Nov 2018)
Changed paths:
M lib/galaxy/jobs/__init__.py
Log Message:
-----------
2 small simplifications in ``JobWrapper``
After commit eeb1dae5b71200ffe9f2056b2a7fd8ed560033c1,
`JobWrapper.job_destination()` does not call
`self._job_dataset_path_rewriter()` any more.
Commit: 6778fa47ed899eefe1418139f43832f225760e0f
https://github.com/galaxyproject/galaxy/commit/6778fa47ed899eefe1418139f438…
Author: John Chilton <jmchilton(a)gmail.com>
Date: 2018-11-20 (Tue, 20 Nov 2018)
Changed paths:
M lib/galaxy/jobs/__init__.py
M lib/galaxy/jobs/handler.py
M lib/galaxy/jobs/runners/__init__.py
M lib/galaxy/jobs/runners/chronos.py
M lib/galaxy/jobs/runners/cli.py
M lib/galaxy/jobs/runners/condor.py
M lib/galaxy/jobs/runners/drmaa.py
M lib/galaxy/jobs/runners/godocker.py
M lib/galaxy/jobs/runners/kubernetes.py
M lib/galaxy/jobs/runners/local.py
M lib/galaxy/jobs/runners/pbs.py
M lib/galaxy/jobs/runners/pulsar.py
M lib/galaxy/jobs/runners/tasks.py
M lib/galaxy/model/__init__.py
M test/unit/jobs/test_runner_local.py
Log Message:
-----------
Merge pull request #7023 from nsoranzo/stop_job_wrapper
Pass a ``JobWrapper`` instead of a ``Job`` to ``stop_job()`` methods
Compare: https://github.com/galaxyproject/galaxy/compare/cd0b7a9347c8...6778fa47ed89
**NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Branch: refs/heads/dev
Home: https://github.com/galaxyproject/galaxy
Commit: eb3664316ce9d7ab3f6d5d2c637b6901138c19a3
https://github.com/galaxyproject/galaxy/commit/eb3664316ce9d7ab3f6d5d2c637b…
Author: Nicola Soranzo <nicola.soranzo(a)earlham.ac.uk>
Date: 2018-11-20 (Tue, 20 Nov 2018)
Changed paths:
M lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Log Message:
-----------
Only migrate distinct (input_step_id, input_name) pairs
otherwise the UNIQUE constraint on the `workflow_step_input` table is
violated.
Follow-up on #6850.
Reported by @mvdbeek.
Commit: 797c2c5b270bb544773025e0d0d85855aba62d6b
https://github.com/galaxyproject/galaxy/commit/797c2c5b270bb544773025e0d0d8…
Author: Nicola Soranzo <nicola.soranzo(a)earlham.ac.uk>
Date: 2018-11-20 (Tue, 20 Nov 2018)
Changed paths:
M lib/galaxy/model/mapping.py
M lib/galaxy/model/migrate/versions/0006_change_qual_datatype.py
Log Message:
-----------
Simplification of SQLAlchemy session handling
Commit: d13792d64605bc3480eae5e877e933b1c111e28f
https://github.com/galaxyproject/galaxy/commit/d13792d64605bc3480eae5e877e9…
Author: John Chilton <jmchilton(a)gmail.com>
Date: 2018-11-20 (Tue, 20 Nov 2018)
Changed paths:
M lib/galaxy/model/mapping.py
M lib/galaxy/model/migrate/versions/0006_change_qual_datatype.py
M lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Log Message:
-----------
Merge pull request #7029 from nsoranzo/followup_on_6850
Follow-up on #6850
Compare: https://github.com/galaxyproject/galaxy/compare/72b2781d4b22...d13792d64605
**NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Branch: refs/heads/dev
Home: https://github.com/galaxyproject/galaxy
Commit: 55f5235ab8b444a2ef20b03b9a03183cb5266ebb
https://github.com/galaxyproject/galaxy/commit/55f5235ab8b444a2ef20b03b9a03…
Author: M Bernt <m.bernt(a)ufz.de>
Date: 2018-11-14 (Wed, 14 Nov 2018)
Changed paths:
M lib/galaxy/dependencies/__init__.py
M lib/galaxy/jobs/runners/drmaa.py
A lib/galaxy/jobs/runners/univa.py
M lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh
Log Message:
-----------
A new runner for DRMAA (currently UNIVA)
Reimplementation of the DRMAA runner inspired by the SLURM runner.
Currently tested only for the UNIVA grid engine (but I'm optimistic
that it should work as well for other drmaa systems).
This solves the problem that the current DRMAAJobRunner does
not work when jobs are submitted as real user (because jobs that are
started in a different drmaa session can not be accessed from the
session that is open in galaxy):
- this is done by resorting to command line tools qstat and qacct if
the drmaa library can not be used to check the job status and to get run
time information.
- this has the additional advantage that if the drmaa library
functions are not working (DRMAAJobRunner had implemented a repeated
checking to handle this problem) the runner can still use the command
line tools.
Furthermore (in contrast to the original drmaa runner) the new one
tests for run time and memory violations:
- memory violations are determined by comparing the used and the
requested memory
- run time violations are determined by checking the signal that
killed the job and by comparing the used and the requested run time
Where the used memory and time are determined with drmaa.wait() or
qacct
Open (or better perspective):
- adaptions to other grid engines. the current implementation (the
command line calls and result parsing) might be specific for the
Univa grid engine. to include other GEs one could determine the
GE (+ version) and make the calls and result parsing depending
on this.
Implementation note:
The changes in drmaa.py do not change the functionality at all,
but only reorganize the code. In particular part of
the function `check_watched_items` was put into a new function
`check_watched_item` in order to make subclassing more convenient.
Replaces #6931 (which replaced #4275), since I did mess up with git
again (there were some duplicated commits).
Commit: 72b2781d4b22cf1c6a312912ffc5333fe5db10b7
https://github.com/galaxyproject/galaxy/commit/72b2781d4b22cf1c6a312912ffc5…
Author: John Chilton <jmchilton(a)gmail.com>
Date: 2018-11-19 (Mon, 19 Nov 2018)
Changed paths:
M lib/galaxy/dependencies/__init__.py
M lib/galaxy/jobs/runners/drmaa.py
A lib/galaxy/jobs/runners/univa.py
M lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh
Log Message:
-----------
Merge pull request #7004 from bernt-matthias/topic/univa3
A new runner for DRMAA (currently UNIVA)
Compare: https://github.com/galaxyproject/galaxy/compare/f9685459a45f...72b2781d4b22
**NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Branch: refs/heads/dev
Home: https://github.com/galaxyproject/galaxy
Commit: 67efd072ae98d6a34cbbc7f2270743524126bf8a
https://github.com/galaxyproject/galaxy/commit/67efd072ae98d6a34cbbc7f22707…
Author: John Chilton <jmchilton(a)gmail.com>
Date: 2018-11-15 (Thu, 15 Nov 2018)
Changed paths:
M lib/galaxy/dependencies/pipfiles/default/pinned-requirements.txt
M lib/galaxy/managers/workflows.py
M lib/galaxy/model/__init__.py
M lib/galaxy/model/mapping.py
M lib/galaxy/model/migrate/versions/0136_collection_and_workflow_state.py
A lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
M lib/galaxy/webapps/tool_shed/model/__init__.py
M lib/galaxy/workflow/extract.py
M lib/galaxy/workflow/modules.py
M lib/tool_shed/util/workflow_util.py
M test/api/test_workflows.py
M test/unit/test_galaxy_mapping.py
M test/unit/workflows/test_modules.py
M test/unit/workflows/test_render.py
M test/unit/workflows/test_workflow_progress.py
M test/unit/workflows/workflow_support.py
Log Message:
-----------
Track workflow step input definitions in our model.
We don't track workflow step inputs in any formal way in our model currently. This has resulted in some current hacks and prevents future enhancements. This commit splits WorkflowStepConnection into two models WorkflowStepInput and WorkflowStepConnection - normalizing the previous table workflow_step_connection on input step and input name.
In terms of current hacks forced on it by restricting all of tool state to be confined to a big JSON blob in the database - we have problems distinguishing keys and values when walking tool state. As we store more and more JSON blobs inside of the giant tool state blob - the worse this problem gets. Take for instance checking for runtime parameters or the rules parameter values - these both use JSON blobs that aren't simple values, so it is hard to tell looking at the tool state blob in the database or the workflow export to tell what is a key or what is a value. Tracking state as normalized inputs with default values and explicit attributes runtime values should allow much more percise state definition and construction.
This variant of the models would also potentially allow defining runtime values with non-tool default values (so default values defined for the workflow but still explicitly settable at runtime). The combinations of overriding defaults and defining runtime values were not representable before.
In terms of future enhancements, there is a lot we cannot track with the current models - such as map/reduce options for collection operations (https://github.com/galaxyproject/galaxy/issues/4623#issuecomment-389544980). This should enable a lot of that. Obviously there are a lot of attributes defined here that are not yet utilized, but I'm using most (all?) of them downstream in the CWL branch. I'd rather populate this table fully realized and fill in the implementation around it as work continues to stream in from the CWL branch - to keep things simple and avoid extra database migrations. But I understand if this feels like speculative complexity we want to avoid despite the implementation being readily available for inspection downstream.
Commit: 15c43e777b7c66614fd9d157ab2fcabb54d7f759
https://github.com/galaxyproject/galaxy/commit/15c43e777b7c66614fd9d157ab2f…
Author: Nicola Soranzo <nicola.soranzo(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M lib/galaxy/model/mapping.py
Log Message:
-----------
Update lib/galaxy/model/mapping.py
Co-Authored-By: jmchilton <jmchilton(a)gmail.com>
Commit: 0a18ed62977b8a81932841361f7fa126f8dca2dc
https://github.com/galaxyproject/galaxy/commit/0a18ed62977b8a81932841361f7f…
Author: Nicola Soranzo <nicola.soranzo(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Log Message:
-----------
Update lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Co-Authored-By: jmchilton <jmchilton(a)gmail.com>
Commit: 00af4e21dee168fd6c54ac925414bcb95b8e7930
https://github.com/galaxyproject/galaxy/commit/00af4e21dee168fd6c54ac925414…
Author: Nicola Soranzo <nicola.soranzo(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Log Message:
-----------
Update lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Co-Authored-By: jmchilton <jmchilton(a)gmail.com>
Commit: f2c17e589b734afedcb807d038d0f1b8caf27f52
https://github.com/galaxyproject/galaxy/commit/f2c17e589b734afedcb807d038d0…
Author: Nicola Soranzo <nicola.soranzo(a)earlham.ac.uk>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M lib/galaxy/model/mapping.py
M lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
Log Message:
-----------
Preserve updated content in migration 145 downgrade
Also add UNIQUE constraint to `("workflow_step_id", "name")` to the
new `workflow_step_input` table.
Commit: f9685459a45f2e2b3395077cbfa11e9a45245021
https://github.com/galaxyproject/galaxy/commit/f9685459a45f2e2b3395077cbfa1…
Author: Nicola Soranzo <nicola.soranzo(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M lib/galaxy/dependencies/pipfiles/default/pinned-requirements.txt
M lib/galaxy/managers/workflows.py
M lib/galaxy/model/__init__.py
M lib/galaxy/model/mapping.py
M lib/galaxy/model/migrate/versions/0136_collection_and_workflow_state.py
A lib/galaxy/model/migrate/versions/0145_add_workflow_step_input.py
M lib/galaxy/webapps/tool_shed/model/__init__.py
M lib/galaxy/workflow/extract.py
M lib/galaxy/workflow/modules.py
M lib/tool_shed/util/workflow_util.py
M test/api/test_workflows.py
M test/unit/test_galaxy_mapping.py
M test/unit/workflows/test_modules.py
M test/unit/workflows/test_render.py
M test/unit/workflows/test_workflow_progress.py
M test/unit/workflows/workflow_support.py
Log Message:
-----------
Merge pull request #6850 from jmchilton/workflows_track_step_inputs
Track workflow step input definitions in our model.
Compare: https://github.com/galaxyproject/galaxy/compare/7695569b50b7...f9685459a45f
**NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Branch: refs/heads/dev
Home: https://github.com/galaxyproject/galaxy
Commit: b8c97f108681e0e0963efc95a16af13be84642f9
https://github.com/galaxyproject/galaxy/commit/b8c97f108681e0e0963efc95a16a…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-15 (Thu, 15 Nov 2018)
Changed paths:
M lib/galaxy/jobs/mapper.py
M test/unit/jobs/test_mapper.py
M test/unit/jobs/test_rules/10_site.py
Log Message:
-----------
Added support for chaining dynamic destinations
Commit: f46ab5a5dc21c1e8dd115ec3d9a37e0e3ee26a56
https://github.com/galaxyproject/galaxy/commit/f46ab5a5dc21c1e8dd115ec3d9a3…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-15 (Thu, 15 Nov 2018)
Changed paths:
M lib/galaxy/jobs/mapper.py
Log Message:
-----------
Fix debug logging
Commit: f6c0e6ff9fdbf66a2af6d6538ad816274ecb93b3
https://github.com/galaxyproject/galaxy/commit/f6c0e6ff9fdbf66a2af6d6538ad8…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-15 (Thu, 15 Nov 2018)
Changed paths:
M lib/galaxy/jobs/mapper.py
Log Message:
-----------
Fixed issue with job_destination caching and added documentation
Commit: c114ba4f43cb4e73fbff470f1283aed103e91fc2
https://github.com/galaxyproject/galaxy/commit/c114ba4f43cb4e73fbff470f1283…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-15 (Thu, 15 Nov 2018)
Changed paths:
M lib/galaxy/jobs/mapper.py
M test/unit/jobs/test_mapper.py
Log Message:
-----------
Added test for cached_job_destination to prevent regression
Commit: 766039adbd17490bf6e758311329b03917fefa18
https://github.com/galaxyproject/galaxy/commit/766039adbd17490bf6e758311329…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-15 (Thu, 15 Nov 2018)
Changed paths:
M config/job_conf.xml.sample_advanced
Log Message:
-----------
Added usage example of dynamic rule chaining
Commit: ca7e2b9e68900e26a3be444f7269e21399176b10
https://github.com/galaxyproject/galaxy/commit/ca7e2b9e68900e26a3be444f7269…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
A test/integration/chained_dyndest_job_conf.xml
A test/integration/chained_dyndest_rules/__init__.py
A test/integration/chained_dyndest_rules/rules.py
A test/integration/test_chained_dynamic_destinations.py
Log Message:
-----------
Added integration test for chained destinations
Commit: 5ba1c24a1475c13f977fc2d1a6e02b5dc8158bb6
https://github.com/galaxyproject/galaxy/commit/5ba1c24a1475c13f977fc2d1a6e0…
Author: Nuwan Goonasekera <nuwan.ag(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M test/integration/chained_dyndest_job_conf.xml
A test/integration/chained_dyndest_rules/module1/__init__.py
A test/integration/chained_dyndest_rules/module1/rules.py
A test/integration/chained_dyndest_rules/module2/__init__.py
A test/integration/chained_dyndest_rules/module2/rules.py
A test/integration/chained_dyndest_rules/module3/__init__.py
A test/integration/chained_dyndest_rules/module3/rules.py
R test/integration/chained_dyndest_rules/rules.py
Log Message:
-----------
Added integration test for rules_module overrides
Commit: ae7869170762b08425c264b0d9777df0efd977ea
https://github.com/galaxyproject/galaxy/commit/ae7869170762b08425c264b0d977…
Author: Marius van den Beek <m.vandenbeek(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M test/integration/test_chained_dynamic_destinations.py
Log Message:
-----------
Update test/integration/test_chained_dynamic_destinations.py
Co-Authored-By: nuwang <nuwan.goonasekera(a)gmail.com>
Commit: 7695569b50b7bb1ae249a248743612debd519f10
https://github.com/galaxyproject/galaxy/commit/7695569b50b7bb1ae249a2487436…
Author: Marius van den Beek <m.vandenbeek(a)gmail.com>
Date: 2018-11-16 (Fri, 16 Nov 2018)
Changed paths:
M config/job_conf.xml.sample_advanced
M lib/galaxy/jobs/mapper.py
A test/integration/chained_dyndest_job_conf.xml
A test/integration/chained_dyndest_rules/__init__.py
A test/integration/chained_dyndest_rules/module1/__init__.py
A test/integration/chained_dyndest_rules/module1/rules.py
A test/integration/chained_dyndest_rules/module2/__init__.py
A test/integration/chained_dyndest_rules/module2/rules.py
A test/integration/chained_dyndest_rules/module3/__init__.py
A test/integration/chained_dyndest_rules/module3/rules.py
A test/integration/test_chained_dynamic_destinations.py
M test/unit/jobs/test_mapper.py
M test/unit/jobs/test_rules/10_site.py
Log Message:
-----------
Merge pull request #7006 from nuwang/chained_dynamic_destinations
Added support for chaining dynamic destinations
Compare: https://github.com/galaxyproject/galaxy/compare/952847aad1eb...7695569b50b7
**NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.