Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 0ff639a29c6179e530c977b358cebf229c76139d https://github.com/galaxyproject/galaxy/commit/0ff639a29c6179e530c977b358ceb... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_workflows.py
Log Message: ----------- More debugging in workflows API tests.
Commit: 4e9ee22ebcd5cc358d1b86f60e957846f35765a5 https://github.com/galaxyproject/galaxy/commit/4e9ee22ebcd5cc358d1b86f60e957... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_workflow_extraction.py M test/base/populators.py
Log Message: ----------- More verbose workflow extraction test cases.
Commit: 0e0894bbba8ee1d8b7c6946a5a42695fa6bd0858 https://github.com/galaxyproject/galaxy/commit/0e0894bbba8ee1d8b7c6946a5a426... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/webapps/galaxy/api/history_contents.py
Log Message: ----------- Refactoring history contents API.
Commit: 8c1658f61a12b9c1e2efdb04ee3530c1272b0bde https://github.com/galaxyproject/galaxy/commit/8c1658f61a12b9c1e2efdb04ee353... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/jobs/__init__.py M lib/galaxy/model/__init__.py
Log Message: ----------- Mixin for uses create and update time.
Commit: c076b3b6c5bf5ac9d374d56d42c9a91c352688ce https://github.com/galaxyproject/galaxy/commit/c076b3b6c5bf5ac9d374d56d42c9a... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_workflows.py M test/base/populators.py
Log Message: ----------- Refactor workflow testing toward reuse via "populators".
Should allow more functionality to be shared between vanilla API tests and integration tests for various workflow scheduling options.
Commit: 55aa7bce5540469fad266f1166c9a2e7365030d4 https://github.com/galaxyproject/galaxy/commit/55aa7bce5540469fad266f1166c9a... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_tools.py A test/functional/tools/output_filter_with_input.xml M test/functional/tools/samples_tool_conf.xml
Log Message: ----------- Test cases for mapping for outputs with filter statements.
Checked through tools-iuc and I couldn't find any tools that use the actual input data parameters in filter statements so I think it is a fairly good approximation to assume that all mapped jobs will either filter or not at least with current execution model.
Commit: d5f98f49045bca351e982f17cac9386308089eeb https://github.com/galaxyproject/galaxy/commit/d5f98f49045bca351e982f17cac93... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/jobs/actions/post.py M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py A lib/galaxy/model/migrate/versions/0136_record_workflow_outputs.py M lib/galaxy/tools/execute.py M lib/galaxy/workflow/modules.py M lib/galaxy/workflow/run.py M test/api/test_workflows.py M test/base/populators.py M test/unit/workflows/test_workflow_progress.py
Log Message: ----------- Formalize workflow invocation and invocation step outputs.
Workflow Invocations --------------------
The workflow invocation outputs half of this is relatively straight forward. It is modelled somewhat on job outputs, output datasets and output dataset collections are now tracked for each workflow invocation and exposed via the workflow invocation API. This required adding new tables (linked to WorkflowInvocations and WorkflowOutputs) that track these output associations.
Previously one could imagine backtracking this information for simple tool steps via the WorkflowInvocationStep -> Job table, but for steps that have many jobs (i.e. mapping over a collection) or for non-tool steps such information was more difficult to recover (and simply couldn't be recovered from the API at all or even internally without significant knowledge of the underlying workflow).
Workflow Invocation Steps -------------------------
Tracking the outputs of WorkflowInvocationSteps was not previously done at all, one would have to follow the Job table as well. A signficant downside to this is that one cannot map over empty collections in a workflow - since no such job would exist. Tracking job outputs for WorkflowInvocationSteps is not a simple matter of just attaching outputs to an existing table because we had no concept of a workflow step tracked - since there could be many WorklfowInvocationSteps corresponding to the same combination of WorkflowInvocation and WorkflowStep. That should feel wrong and that is because it is - when collections were added the possiblity of having many jobs for the same combination of WorkflowInvocation and WorkflowStep was added. I should have split WorkflowInvocationSteps into WorkflowInvocationSteps and WorkflowInvocationStepJobAssociations at that time but didn't. This commit now does it - effectively normalizing the ``workflow_invocation_step`` table by introducing the new ``wo rkflow_invocation_step_job_association`` table.
Splitting up the WorkflowInvocationStep table this way allows recovering the mapped over output (e.g. the implicitly created collection from all the jobs) as well the outputs from the individual jobs (by walking WorkflowInvocationStep -> WorkflowInvocationStepJobAssociation -> Job -> JobToOutput*Association).
This split up involves failrly substantial changes to the workflow module interface. Any place a list of WorkflowInvocationSteps was assumed, I reworked it to just expect a single WorkflowInvocationStep. I vastly simplified recover_mapping to just use the persisted outputs (this was needed in order to also implment empty collection mapping in workflows). This also fixes a bug (or implements a missing feature) where Subworkflow moudles had no recover_mapping methods - so for instance if a tool that produces dynamic collections appeared anywhere in a workflow after a subworkflow step - that workflow would not complete scheduling properly.
Now that we have a way to reference the set of jobs corresponding to a workflow step within an invocation, we can start to track partial scheduling of such steps. This is outlined in https://github.com/galaxyproject/galaxy/issues/3883 and refactoring toward this goal is included here - including adding a state to WorkflowInvocationStep so Galaxy can determine if it has started scheduling this step and an index when scheduling jobs so it can tell how far into a scheduling things have gone as well as augmenting the tool executor to take a maximum number of jobs to execute and allow recovery of existing jobs for collection building purposes.
*Applications*
These changes will enable:
- A simple, consistent API for finding workflow outputs that can be consumed by Planemo for testing workflows. - Mapping over empty collections in workflows. - Re-scheduling workflow invocations that include subworkflow steps. - Partial scheduling within steps requiring a large number of jobs when scheduling workflow invocations.
Commit: 64835c1e72eea545269260256c7a238a4e5ea7fe https://github.com/galaxyproject/galaxy/commit/64835c1e72eea545269260256c7a2... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_workflows.py
Log Message: ----------- Three new worklfow test cases to verify mapping over workflows is possible.
- Simple mapping over an input dataset. - Mapping that produces nested collections. - Mapping over subworkflows.
Commit: 0185e04a5beaf951bb20d670d9421d988d70bb29 https://github.com/galaxyproject/galaxy/commit/0185e04a5beaf951bb20d670d9421... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_workflows.py
Log Message: ----------- Test recover_mapping in conjunction with subworkflow executions.
Commit: 9e471d3e97a6d6b6893ed243b6a48108076899c4 https://github.com/galaxyproject/galaxy/commit/9e471d3e97a6d6b6893ed243b6a48... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/dataset_collections/matching.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/execute.py M lib/galaxy/workflow/modules.py M test/api/test_tools.py M test/api/test_workflows.py M test/base/populators.py A test/functional/tools/for_workflows/count_list.xml A test/functional/tools/for_workflows/count_multi_file.xml A test/functional/tools/for_workflows/empty_list.xml M test/functional/tools/samples_tool_conf.xml
Log Message: ----------- Allow mapping over empty collections.
If one maps a tool over an empty collection, one should get an empty output collection for each of the tool's outputs.
This should have always been a valid operation I think, the code just wasn't quite structured right when the collection work was merged - there were too many places I was assuming the existence of at least one actual output or input that was mapped over to continue going. This reworks all of that and seems to handle empty collections appropriately.
Fixes #4025 (not being informing the user of the operation, but actually by allowing the operation - which it should be allowed I think).
Commit: 53cba4aa74b0d2543445824a7636f92249baa150 https://github.com/galaxyproject/galaxy/commit/53cba4aa74b0d2543445824a7636f... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/managers/collections.py
Log Message: ----------- Refactor collection manager into smaller, more focused methods.
Stronger assertions to verify arguments are correct.
Commit: 78babab6455f4cd1c51aed0b1ddf6b4b29ad6d03 https://github.com/galaxyproject/galaxy/commit/78babab6455f4cd1c51aed0b1ddf6... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/dataset_collections/matching.py M lib/galaxy/dataset_collections/structure.py M lib/galaxy/dataset_collections/type_description.py M lib/galaxy/managers/collections.py M lib/galaxy/managers/collections_util.py M lib/galaxy/model/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/actions/model_operations.py M lib/galaxy/tools/execute.py M lib/galaxy/tools/parser/output_objects.py M lib/galaxy/workflow/modules.py M lib/galaxy/workflow/run.py M test/api/test_tools.py M test/api/test_workflow_extraction.py M test/api/test_workflows.py M test/base/populators.py M test/unit/dataset_collections/test_matching.py
Log Message: ----------- Pre-build collections during mapping to be more recoverable.
Create mapped over collections ahead of time instead of at the end only if they are valid.
- This way if there is a problem that problem can be reported in the state of the mapped collection. - This way datasets can just be hidden instead of displayed and then hidden (fixes #1790). - I think this should improve performance and prevent locks because we no longer have conflicting threads trying to write dataset state. - We populate the collection as we go so we can recover and continue scheduling the mapped step (needed for #3883)
Commit: 40fd72a1980bd2df5ea116aae1016c8bad6c3ca1 https://github.com/galaxyproject/galaxy/commit/40fd72a1980bd2df5ea116aae1016... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/tools/execute.py M lib/galaxy/workflow/modules.py M lib/galaxy/workflow/run.py R test/integration/test_maximum_worklfow_invocation_duration.py A test/integration/test_workflow_scheduling_options.py
Log Message: ----------- Allow rescheduling of single workflow steps.
Implement maximum_workflow_jobs_per_scheduling_iteration to test this and mitigate memory issues associated with scheduling large workflow steps (maybe).
Fixes #3883.
Commit: d69962e7f4de38963401a032ada5131e6190b5e5 https://github.com/galaxyproject/galaxy/commit/d69962e7f4de38963401a032ada51... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/managers/collections.py M lib/galaxy/managers/hdcas.py M lib/galaxy/managers/jobs.py M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py M lib/galaxy/model/migrate/versions/0136_record_workflow_outputs.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/execute.py M lib/galaxy/webapps/galaxy/api/history_contents.py M lib/galaxy/webapps/galaxy/buildapp.py M scripts/db_shell.py M test/api/test_history_contents.py M test/api/test_tools.py M test/base/populators.py
Log Message: ----------- Implement model abstraction for ``ImplicitCollectionJobs``.
These aggregate all the jobs contributing to an implicitly created HistoryDatasetCollectionAssociation.
- Model enhancements that allow this. - API endpoint for jobs summary of history content (somewhat uniform across HDAs and HDCAs just for consistency). - Test cases. - Rework history contents API a bit to eliminate more costly state summary checks by default.
Commit: e82012bde154c456f24743f3b61c0ed9ffc7eca2 https://github.com/galaxyproject/galaxy/commit/e82012bde154c456f24743f3b61c0... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/jobs/actions/post.py M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py M lib/galaxy/model/migrate/versions/0136_record_workflow_outputs.py M lib/galaxy/tools/execute.py
Log Message: ----------- Undo splitting WorkflowInvocationStep...
... this concept of ImplicitCollectionJobs makes more sense for tracking collections of implicitly created jobs.
Commit: 0dceb5a553d589324d4a9a97ec5f7629bc379b48 https://github.com/galaxyproject/galaxy/commit/0dceb5a553d589324d4a9a97ec5f7... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M lib/galaxy/dataset_collections/builder.py M lib/galaxy/managers/collections.py M lib/galaxy/managers/collections_util.py M lib/galaxy/managers/hdcas.py M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py M lib/galaxy/model/migrate/versions/0136_record_workflow_outputs.py M test/api/test_dataset_collections.py
Log Message: ----------- Add element count to DatasetCollection objects.
There are no places where this count is not set atomically in one thread so this should not have any performance downsides versus the older approach of calculating it on the fly (which was very expensive).
Commit: ca09e803aeaf856556433e69bf5a48f8bab60c5d https://github.com/galaxyproject/galaxy/commit/ca09e803aeaf856556433e69bf5a4... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/api/test_tools.py M test/base/populators.py M test/functional/tools/collection_creates_dynamic_nested.xml M test/functional/tools/collection_creates_dynamic_nested_fail.xml M test/galaxy_selenium/navigates_galaxy.py M test/galaxy_selenium/navigation.yml M test/selenium_tests/framework.py A test/selenium_tests/test_history_copy_elements.py A test/selenium_tests/test_history_multi_view.py A test/selenium_tests/test_history_panel_collections.py
Log Message: ----------- Dataset collection state UX test case.
Commit: 5a9f0dd11eb0ba0abfed29ae9da1d0a5a5a01360 https://github.com/galaxyproject/galaxy/commit/5a9f0dd11eb0ba0abfed29ae9da1d... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M client/galaxy/scripts/mvc/history/hdca-li.js M client/galaxy/scripts/mvc/history/history-contents.js M client/galaxy/scripts/mvc/history/history-model.js M client/galaxy/scripts/mvc/history/history-view.js A client/galaxy/scripts/mvc/history/job-states-model.js M client/galaxy/scripts/mvc/history/multi-panel.js M client/galaxy/style/less/base.less M client/galaxy/style/less/list-item.less M lib/galaxy/managers/jobs.py M lib/galaxy/webapps/galaxy/api/history_contents.py M lib/galaxy/webapps/galaxy/buildapp.py
Log Message: ----------- Add job state tracking to the GUI.
- ``populated`` in the API response is gone - it was very expensive on the backend to calculate. - Eliminate GUI "thread" fetching ``element_count`` for collections after the panel is loaded, this value is now available at load time thanks to recent state change optimizations on the backend. - Track job state and implicit collection job states over time in a new "thread". - New different colors of job states, add progress bar for multiple jobs.
Commit: e7e659cabcf2a5928ab58e73715c0959c78cc997 https://github.com/galaxyproject/galaxy/commit/e7e659cabcf2a5928ab58e73715c0... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M client/galaxy/scripts/mvc/history/hdca-li.js M client/galaxy/style/less/dataset.less
Log Message: ----------- Better dataset loading description.
Commit: 075d8c1d0ab7dca9c8573fe69a5ef7ec18ab2202 https://github.com/galaxyproject/galaxy/commit/075d8c1d0ab7dca9c8573fe69a5ef... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: A lib/galaxy/model/migrate/versions/0136_collection_and_workflow_state.py R lib/galaxy/model/migrate/versions/0136_record_workflow_outputs.py
Log Message: ----------- Rename huge database migration to reflect new PR topic.
Commit: 8fa5496ba59e39ce4d16bd7c8da1d92989c7f5da https://github.com/galaxyproject/galaxy/commit/8fa5496ba59e39ce4d16bd7c8da1d... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M test/selenium_tests/test_history_panel_collections.py
Log Message: ----------- Restructure collection state GUI tests to separate flakey tests of running state out.
Paste just sometimes randomly hangs when it is not supposed to.
Commit: c47d8992ba52e5af8eae635a4ce50cca8a6d02c9 https://github.com/galaxyproject/galaxy/commit/c47d8992ba52e5af8eae635a4ce50... Author: John Chilton jmchilton@gmail.com Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M static/scripts/bundled/admin.bundled.js M static/scripts/bundled/adminToolshed.bundled.js M static/scripts/bundled/analysis.bundled.js M static/scripts/bundled/library.bundled.js M static/scripts/bundled/libs.bundled.js M static/scripts/bundled/login.bundled.js M static/scripts/bundled/masthead.bundled.js M static/scripts/bundled/viz.bundled.js M static/scripts/bundled/workflow.bundled.js M static/scripts/mvc/history/hdca-li.js M static/scripts/mvc/history/history-contents.js M static/scripts/mvc/history/history-model.js M static/scripts/mvc/history/history-view.js A static/scripts/mvc/history/job-states-model.js M static/scripts/mvc/history/multi-panel.js M static/style/blue/base.css
Log Message: ----------- Repack client.
Commit: 9ef4e3bd70d3abe7b36388e9d9e52c609749002d https://github.com/galaxyproject/galaxy/commit/9ef4e3bd70d3abe7b36388e9d9e52... Author: Martin Cech marten@bx.psu.edu Date: 2017-11-30 (Thu, 30 Nov 2017)
Changed paths: M client/galaxy/scripts/mvc/history/hdca-li.js M client/galaxy/scripts/mvc/history/history-contents.js M client/galaxy/scripts/mvc/history/history-model.js M client/galaxy/scripts/mvc/history/history-view.js A client/galaxy/scripts/mvc/history/job-states-model.js M client/galaxy/scripts/mvc/history/multi-panel.js M client/galaxy/style/less/base.less M client/galaxy/style/less/dataset.less M client/galaxy/style/less/list-item.less M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/dataset_collections/builder.py M lib/galaxy/dataset_collections/matching.py M lib/galaxy/dataset_collections/structure.py M lib/galaxy/dataset_collections/type_description.py M lib/galaxy/jobs/__init__.py M lib/galaxy/managers/collections.py M lib/galaxy/managers/collections_util.py M lib/galaxy/managers/hdcas.py M lib/galaxy/managers/jobs.py M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py A lib/galaxy/model/migrate/versions/0136_collection_and_workflow_state.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/actions/model_operations.py M lib/galaxy/tools/execute.py M lib/galaxy/tools/parser/output_objects.py M lib/galaxy/webapps/galaxy/api/history_contents.py M lib/galaxy/webapps/galaxy/buildapp.py M lib/galaxy/workflow/modules.py M lib/galaxy/workflow/run.py M scripts/db_shell.py M static/scripts/bundled/admin.bundled.js M static/scripts/bundled/adminToolshed.bundled.js M static/scripts/bundled/analysis.bundled.js M static/scripts/bundled/library.bundled.js M static/scripts/bundled/libs.bundled.js M static/scripts/bundled/login.bundled.js M static/scripts/bundled/masthead.bundled.js M static/scripts/bundled/viz.bundled.js M static/scripts/bundled/workflow.bundled.js M static/scripts/mvc/history/hdca-li.js M static/scripts/mvc/history/history-contents.js M static/scripts/mvc/history/history-model.js M static/scripts/mvc/history/history-view.js A static/scripts/mvc/history/job-states-model.js M static/scripts/mvc/history/multi-panel.js M static/style/blue/base.css M test/api/test_dataset_collections.py M test/api/test_history_contents.py M test/api/test_tools.py M test/api/test_workflow_extraction.py M test/api/test_workflows.py M test/base/populators.py M test/functional/tools/collection_creates_dynamic_nested.xml M test/functional/tools/collection_creates_dynamic_nested_fail.xml A test/functional/tools/for_workflows/count_list.xml A test/functional/tools/for_workflows/count_multi_file.xml A test/functional/tools/for_workflows/empty_list.xml A test/functional/tools/output_filter_with_input.xml M test/functional/tools/samples_tool_conf.xml M test/galaxy_selenium/navigates_galaxy.py M test/galaxy_selenium/navigation.yml R test/integration/test_maximum_worklfow_invocation_duration.py A test/integration/test_workflow_scheduling_options.py M test/selenium_tests/framework.py A test/selenium_tests/test_history_copy_elements.py A test/selenium_tests/test_history_multi_view.py A test/selenium_tests/test_history_panel_collections.py M test/unit/dataset_collections/test_matching.py M test/unit/workflows/test_workflow_progress.py
Log Message: ----------- Merge pull request #5013 from jmchilton/workflow_and_collection_state
Improved Collection and Workflow State with Applications
Compare: https://github.com/galaxyproject/galaxy/compare/711107473fa5...9ef4e3bd70d3