- galaxy-commits - lists.galaxyproject.org

[galaxyproject/galaxy] 5dcc0e: Clean up old TODO into history tests.
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 5dcc0ed04376d08deb960f309e21fc14a6d29ce8 https://github.com/galaxyproject/galaxy/commit/5dcc0ed04376d08deb960f309e21… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M test/api/test_users.py Log Message: ----------- Clean up old TODO into history tests. We upgraded the module a long time ago now that was referenced in the Trello card. Commit: 6af5334f125461fba1ea1735bf06eed516bd6754 https://github.com/galaxyproject/galaxy/commit/6af5334f125461fba1ea1735bf06… Author: Martin Cech <marten(a)bx.psu.edu> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M test/api/test_users.py Log Message: ----------- Merge pull request #3855 from jmchilton/user_api_tests Clean up old TODO into history tests. Compare: https://github.com/galaxyproject/galaxy/compare/ad2e6473bc07...6af5334f1254

1 0

[galaxyproject/galaxy] c51adf: Insert update caller/helper for options
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/release_17.01 Home: https://github.com/galaxyproject/galaxy Commit: c51adf781a1a711c4e54d89fbdca7ea9d3331f08 https://github.com/galaxyproject/galaxy/commit/c51adf781a1a711c4e54d89fbdca… Author: guerler <aysam.guerler(a)gmail.com> Date: 2017-03-30 (Thu, 30 Mar 2017) Changed paths: M client/galaxy/scripts/mvc/ui/ui-options.js M static/maps/mvc/ui/ui-options.js.map M static/scripts/bundled/analysis.bundled.js M static/scripts/bundled/analysis.bundled.js.map M static/scripts/bundled/libs.bundled.js M static/scripts/bundled/libs.bundled.js.map M static/scripts/mvc/ui/ui-options.js Log Message: ----------- Insert update caller/helper for options Commit: c8eb2c186def4b505143a5a6b0d971549b15d19a https://github.com/galaxyproject/galaxy/commit/c8eb2c186def4b505143a5a6b0d9… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M client/galaxy/scripts/mvc/ui/ui-options.js M static/maps/mvc/ui/ui-options.js.map M static/scripts/bundled/analysis.bundled.js M static/scripts/bundled/analysis.bundled.js.map M static/scripts/bundled/libs.bundled.js M static/scripts/bundled/libs.bundled.js.map M static/scripts/mvc/ui/ui-options.js Log Message: ----------- Merge pull request #3851 from guerler/add_lost_update_caller [17.01] Fix update caller/helper for options Compare: https://github.com/galaxyproject/galaxy/compare/bf6b65f00596...c8eb2c186def

1 0

[galaxyproject/galaxy] 5d3250: [17.01] Order processing evaluation of workflow in...
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/master Home: https://github.com/galaxyproject/galaxy Commit: 5d3250fac03b32f4d14213cd6039aac884a1307e https://github.com/galaxyproject/galaxy/commit/5d3250fac03b32f4d14213cd6039… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-28 (Tue, 28 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- [17.01] Order processing evaluation of workflow invocations when scheduling. xref https://github.com/galaxyproject/galaxy/pull/3820#issuecomment-289725795 Also optimize the query to just pull out workflow invocation IDs per the suggestion of @nsoranzo [here](https://github.com/galaxyproject/galaxy/pull/3830#pullrequestreview-2…. Commit: bf6b65f005965eff9d01f432bf52f5d1104294b7 https://github.com/galaxyproject/galaxy/commit/bf6b65f005965eff9d01f432bf52… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- Merge pull request #3830 from jmchilton/1701_order_collection_invocation_processing [17.01] Order processing evaluation of workflow invocations when scheduling. Commit: 7e64fa395c6c56a0973f56ecc1aa615f15d9fe80 https://github.com/galaxyproject/galaxy/commit/7e64fa395c6c56a0973f56ecc1aa… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- Merge remote-tracking branch 'upstream/release_17.01' Compare: https://github.com/galaxyproject/galaxy/compare/dd439530509f...7e64fa395c6c

1 0

[galaxyproject/galaxy] 5d3250: [17.01] Order processing evaluation of workflow in...
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 5d3250fac03b32f4d14213cd6039aac884a1307e https://github.com/galaxyproject/galaxy/commit/5d3250fac03b32f4d14213cd6039… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-28 (Tue, 28 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- [17.01] Order processing evaluation of workflow invocations when scheduling. xref https://github.com/galaxyproject/galaxy/pull/3820#issuecomment-289725795 Also optimize the query to just pull out workflow invocation IDs per the suggestion of @nsoranzo [here](https://github.com/galaxyproject/galaxy/pull/3830#pullrequestreview-2…. Commit: bf6b65f005965eff9d01f432bf52f5d1104294b7 https://github.com/galaxyproject/galaxy/commit/bf6b65f005965eff9d01f432bf52… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- Merge pull request #3830 from jmchilton/1701_order_collection_invocation_processing [17.01] Order processing evaluation of workflow invocations when scheduling. Commit: ad2e6473bc07a5b229d50f85c28cad0363a40f11 https://github.com/galaxyproject/galaxy/commit/ad2e6473bc07a5b229d50f85c28c… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- Merge remote-tracking branch 'upstream/release_17.01' into dev Compare: https://github.com/galaxyproject/galaxy/compare/190d1901a43a...ad2e6473bc07

1 0

[galaxyproject/galaxy] 5d3250: [17.01] Order processing evaluation of workflow in...
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/release_17.01 Home: https://github.com/galaxyproject/galaxy Commit: 5d3250fac03b32f4d14213cd6039aac884a1307e https://github.com/galaxyproject/galaxy/commit/5d3250fac03b32f4d14213cd6039… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-28 (Tue, 28 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- [17.01] Order processing evaluation of workflow invocations when scheduling. xref https://github.com/galaxyproject/galaxy/pull/3820#issuecomment-289725795 Also optimize the query to just pull out workflow invocation IDs per the suggestion of @nsoranzo [here](https://github.com/galaxyproject/galaxy/pull/3830#pullrequestreview-2…. Commit: bf6b65f005965eff9d01f432bf52f5d1104294b7 https://github.com/galaxyproject/galaxy/commit/bf6b65f005965eff9d01f432bf52… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M lib/galaxy/model/__init__.py M test/api/test_workflows.py Log Message: ----------- Merge pull request #3830 from jmchilton/1701_order_collection_invocation_processing [17.01] Order processing evaluation of workflow invocations when scheduling. Compare: https://github.com/galaxyproject/galaxy/compare/07887be3a162...bf6b65f00596

1 0

[galaxyproject/galaxy] aba31b: [17.01] Restrict workflow scheduling within a hist...
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: aba31bc0d484f3729b46a10b2b357b92fa826fae https://github.com/galaxyproject/galaxy/commit/aba31bc0d484f3729b46a10b2b35… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-29 (Wed, 29 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- [17.01] Restrict workflow scheduling within a history to a fixed, random handler. Lets revisit the problem that background scheduling workflows (as is the default UI behavior as of 16.10) makes it easier for histories to contain datasets interleaved from different workflow invocations under certain reasonable conditions (https://github.com/galaxyproject/galaxy/issues/3474) Considering only a four year old workflow and tool feature set (no collection operations, no dynamic dataset discovery, only tool and input workflow modules), all workflows can and will fully schedule on the first scheduling iteration. Under those circumstances, this solution is functionally equivalent to history_local_serial_workflow_scheduling introduced #3520 - but should be more performant because all such workflows fully schedule in the first iteration and the double loop introduced here https://github.com/galaxyproject/galaxy/pull/3520/files#diff-d7e80a366f3965… is avoided for each workflow invocation for each iteration. This addresses both concerns I outlined [here](https://github.com/galaxyproject/galaxy/issues/3816#issuecomment-2893…. For workflows that use certain classes of newer tools or newer workflow features - I'd argue this approach will not degrade as harshly as enabling history_local_serial_workflow_scheduling. For instance, imagine a workflow with a dynamic dataset collection output step (such as used by IUC tools Deseq2, Trinity, Stacks, and various Mothur tools) half way through that takes 24 hour of queue time to reach. Now imagine a user running 5 such workflows at once. - Without this and without history_local_serial_workflow_scheduling, the 5 workflows will each run as fast as possible and the UI will show as much of each workflow as can be scheduled but the order of the datsets may be shuffled. The workflows will be complete for the users in 48 hours. - With history_local_serial_workflow_scheduling enabled, only 1 workflow will be scheduled only half way for the first 24 hours and the user will be given no visual indication for why the other workflows are not running for 1 day. The final workflow output will take nearly a week to be complete for the users. - With this enabled - the new default in this commit - each workflow will be scheduled in two chunks but these chunks will be contingious and it should be fairly clear to the user what tool caused the discontinuity of the datasets in the history. So things are still mostly ordered, but the draw backs of history_local_serial_workflow_scheduling are avoided entirely. Namely, the other four workflows aren't hidden from the user without a UI indication and the workflows will still only take 48 hours to be complete and outputs ready for the user. The only drawback of this new default behavior is that you could potentially see some performance improvements by scheduling multiple workflow invocations within one history - but this was never a design goal in my mind when implementing background scheduling and under typical Galaxy use cases I don't think this would be worth the UI problems. So, the older behavior can be re-enabled by setting parallelize_workflow_scheduling_within_histories to True in galaxy.ini but it won't be on by default or really recommended if the Galaxy UI is being used. Commit: 07887be3a162678859e32c063dae024f0a8da25c https://github.com/galaxyproject/galaxy/commit/07887be3a162678859e32c063dae… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- Merge pull request #3820 from jmchilton/fixed_history_handler [17.01] Restrict workflow scheduling within a history to a fixed, random handler. Commit: 190d1901a43a284c478876d2ec3567a1cc1af5a5 https://github.com/galaxyproject/galaxy/commit/190d1901a43a284c478876d2ec35… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- Merge remote-tracking branch 'upstream/release_17.01' into dev Compare: https://github.com/galaxyproject/galaxy/compare/dfbb80dd8b21...190d1901a43a

1 0

[galaxyproject/galaxy] aba31b: [17.01] Restrict workflow scheduling within a hist...
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/master Home: https://github.com/galaxyproject/galaxy Commit: aba31bc0d484f3729b46a10b2b357b92fa826fae https://github.com/galaxyproject/galaxy/commit/aba31bc0d484f3729b46a10b2b35… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-29 (Wed, 29 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- [17.01] Restrict workflow scheduling within a history to a fixed, random handler. Lets revisit the problem that background scheduling workflows (as is the default UI behavior as of 16.10) makes it easier for histories to contain datasets interleaved from different workflow invocations under certain reasonable conditions (https://github.com/galaxyproject/galaxy/issues/3474) Considering only a four year old workflow and tool feature set (no collection operations, no dynamic dataset discovery, only tool and input workflow modules), all workflows can and will fully schedule on the first scheduling iteration. Under those circumstances, this solution is functionally equivalent to history_local_serial_workflow_scheduling introduced #3520 - but should be more performant because all such workflows fully schedule in the first iteration and the double loop introduced here https://github.com/galaxyproject/galaxy/pull/3520/files#diff-d7e80a366f3965… is avoided for each workflow invocation for each iteration. This addresses both concerns I outlined [here](https://github.com/galaxyproject/galaxy/issues/3816#issuecomment-2893…. For workflows that use certain classes of newer tools or newer workflow features - I'd argue this approach will not degrade as harshly as enabling history_local_serial_workflow_scheduling. For instance, imagine a workflow with a dynamic dataset collection output step (such as used by IUC tools Deseq2, Trinity, Stacks, and various Mothur tools) half way through that takes 24 hour of queue time to reach. Now imagine a user running 5 such workflows at once. - Without this and without history_local_serial_workflow_scheduling, the 5 workflows will each run as fast as possible and the UI will show as much of each workflow as can be scheduled but the order of the datsets may be shuffled. The workflows will be complete for the users in 48 hours. - With history_local_serial_workflow_scheduling enabled, only 1 workflow will be scheduled only half way for the first 24 hours and the user will be given no visual indication for why the other workflows are not running for 1 day. The final workflow output will take nearly a week to be complete for the users. - With this enabled - the new default in this commit - each workflow will be scheduled in two chunks but these chunks will be contingious and it should be fairly clear to the user what tool caused the discontinuity of the datasets in the history. So things are still mostly ordered, but the draw backs of history_local_serial_workflow_scheduling are avoided entirely. Namely, the other four workflows aren't hidden from the user without a UI indication and the workflows will still only take 48 hours to be complete and outputs ready for the user. The only drawback of this new default behavior is that you could potentially see some performance improvements by scheduling multiple workflow invocations within one history - but this was never a design goal in my mind when implementing background scheduling and under typical Galaxy use cases I don't think this would be worth the UI problems. So, the older behavior can be re-enabled by setting parallelize_workflow_scheduling_within_histories to True in galaxy.ini but it won't be on by default or really recommended if the Galaxy UI is being used. Commit: 07887be3a162678859e32c063dae024f0a8da25c https://github.com/galaxyproject/galaxy/commit/07887be3a162678859e32c063dae… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- Merge pull request #3820 from jmchilton/fixed_history_handler [17.01] Restrict workflow scheduling within a history to a fixed, random handler. Commit: dd439530509f12ede24f8e9763f2bdb5314e2c13 https://github.com/galaxyproject/galaxy/commit/dd439530509f12ede24f8e9763f2… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- Merge remote-tracking branch 'upstream/release_17.01' Compare: https://github.com/galaxyproject/galaxy/compare/52b5e62de498...dd439530509f

1 0

[galaxyproject/galaxy] aba31b: [17.01] Restrict workflow scheduling within a hist...
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/release_17.01 Home: https://github.com/galaxyproject/galaxy Commit: aba31bc0d484f3729b46a10b2b357b92fa826fae https://github.com/galaxyproject/galaxy/commit/aba31bc0d484f3729b46a10b2b35… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-29 (Wed, 29 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- [17.01] Restrict workflow scheduling within a history to a fixed, random handler. Lets revisit the problem that background scheduling workflows (as is the default UI behavior as of 16.10) makes it easier for histories to contain datasets interleaved from different workflow invocations under certain reasonable conditions (https://github.com/galaxyproject/galaxy/issues/3474) Considering only a four year old workflow and tool feature set (no collection operations, no dynamic dataset discovery, only tool and input workflow modules), all workflows can and will fully schedule on the first scheduling iteration. Under those circumstances, this solution is functionally equivalent to history_local_serial_workflow_scheduling introduced #3520 - but should be more performant because all such workflows fully schedule in the first iteration and the double loop introduced here https://github.com/galaxyproject/galaxy/pull/3520/files#diff-d7e80a366f3965… is avoided for each workflow invocation for each iteration. This addresses both concerns I outlined [here](https://github.com/galaxyproject/galaxy/issues/3816#issuecomment-2893…. For workflows that use certain classes of newer tools or newer workflow features - I'd argue this approach will not degrade as harshly as enabling history_local_serial_workflow_scheduling. For instance, imagine a workflow with a dynamic dataset collection output step (such as used by IUC tools Deseq2, Trinity, Stacks, and various Mothur tools) half way through that takes 24 hour of queue time to reach. Now imagine a user running 5 such workflows at once. - Without this and without history_local_serial_workflow_scheduling, the 5 workflows will each run as fast as possible and the UI will show as much of each workflow as can be scheduled but the order of the datsets may be shuffled. The workflows will be complete for the users in 48 hours. - With history_local_serial_workflow_scheduling enabled, only 1 workflow will be scheduled only half way for the first 24 hours and the user will be given no visual indication for why the other workflows are not running for 1 day. The final workflow output will take nearly a week to be complete for the users. - With this enabled - the new default in this commit - each workflow will be scheduled in two chunks but these chunks will be contingious and it should be fairly clear to the user what tool caused the discontinuity of the datasets in the history. So things are still mostly ordered, but the draw backs of history_local_serial_workflow_scheduling are avoided entirely. Namely, the other four workflows aren't hidden from the user without a UI indication and the workflows will still only take 48 hours to be complete and outputs ready for the user. The only drawback of this new default behavior is that you could potentially see some performance improvements by scheduling multiple workflow invocations within one history - but this was never a design goal in my mind when implementing background scheduling and under typical Galaxy use cases I don't think this would be worth the UI problems. So, the older behavior can be re-enabled by setting parallelize_workflow_scheduling_within_histories to True in galaxy.ini but it won't be on by default or really recommended if the Galaxy UI is being used. Commit: 07887be3a162678859e32c063dae024f0a8da25c https://github.com/galaxyproject/galaxy/commit/07887be3a162678859e32c063dae… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/workflow/scheduling_manager.py A test/integration/test_workflow_handler_configuration.py A test/integration/workflow_handler_configuration_job_conf.xml Log Message: ----------- Merge pull request #3820 from jmchilton/fixed_history_handler [17.01] Restrict workflow scheduling within a history to a fixed, random handler. Compare: https://github.com/galaxyproject/galaxy/compare/8f8526bf307d...07887be3a162

1 0

[galaxyproject/galaxy] c9f602: Update datatypes_conf.xml.sample
by GitHub 31 Mar '17

31 Mar '17

Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: c9f602d434104423078d2f0661e6acd8aa5c359b https://github.com/galaxyproject/galaxy/commit/c9f602d434104423078d2f0661e6… Author: Gildas Le Corguillé <lecorguille(a)users.noreply.github.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/datatypes_conf.xml.sample Log Message: ----------- Update datatypes_conf.xml.sample Oups, I did it again :) Just add the sniffer for the mzData datatype Commit: dfbb80dd8b21f87af217fce49d14df396517c6d9 https://github.com/galaxyproject/galaxy/commit/dfbb80dd8b21f87af217fce49d14… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-31 (Fri, 31 Mar 2017) Changed paths: M config/datatypes_conf.xml.sample Log Message: ----------- Merge pull request #3852 from lecorguille/patch-1 Update datatypes_conf.xml.sample Compare: https://github.com/galaxyproject/galaxy/compare/200e475bc7cd...dfbb80dd8b21

1 0

[galaxyproject/galaxy] 185aa3: Log when workflow scheduling is delayed about why.
by GitHub 30 Mar '17

30 Mar '17

Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 185aa3968bd38cfe021f7bcec9bae02c12ae4c0b https://github.com/galaxyproject/galaxy/commit/185aa3968bd38cfe021f7bcec9ba… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-30 (Thu, 30 Mar 2017) Changed paths: M lib/galaxy/workflow/modules.py M lib/galaxy/workflow/run.py M test/api/test_workflows.py Log Message: ----------- Log when workflow scheduling is delayed about why. Currently the logs contain fairly uninformative step delayed messages that don't allow admins to understand why parts of the workflow are being delayed during scheduling. Here are some examples of before and after. When running the test: ``` ./run_tests.sh -api test/api/test_workflows.py:WorkflowsApiTestCase.test_workflow_pause ``` Before these lines would show up: ``` galaxy.workflow.run DEBUG 2017-03-30 09:23:47,327 Workflow step 2 of invocation 1 invoked (174.149 ms) galaxy.workflow.run DEBUG 2017-03-30 09:23:47,347 Workflow step 3 of invocation 1 invoked (19.355 ms) galaxy.workflow.run DEBUG 2017-03-30 09:23:47,364 Workflow step 4 of invocation 1 delayed (16.262 ms) ``` Now these same lines are as follows: ``` galaxy.workflow.run DEBUG 2017-03-30 09:19:28,601 Workflow step 2 of invocation 1 invoked (172.849 ms) galaxy.workflow.run DEBUG 2017-03-30 09:19:28,619 Marking step 3 outputs delayed (executing pause step) galaxy.workflow.run DEBUG 2017-03-30 09:19:28,620 Workflow step 3 of invocation 1 invoked (17.999 ms) galaxy.workflow.run DEBUG 2017-03-30 09:19:28,633 Workflow step 4 of invocation 1 delayed (dependent step [3] delayed, so this step must be delayed) (13.398 ms) ``` Also, when running the test: ``` ./run_tests.sh -api test/api/test_workflows.py:WorkflowsApiTestCase.test_workflow_run_dynamic_output_collections_3 ``` Before these lines would be printed: ``` galaxy.workflow.run DEBUG 2017-03-30 09:25:35,910 Workflow step 4 of invocation 1 invoked (281.479 ms) galaxy.workflow.run DEBUG 2017-03-30 09:25:35,937 Workflow step 5 of invocation 1 delayed (25.904 ms) galaxy.workflow.run DEBUG 2017-03-30 09:25:35,970 Workflow step 6 of invocation 1 delayed (32.597 ms) ``` Now these same lines are as follows: ``` galaxy.workflow.run DEBUG 2017-03-30 09:27:54,270 Workflow step 4 of invocation 1 invoked (295.826 ms) galaxy.workflow.run DEBUG 2017-03-30 09:27:54,315 Workflow step 5 of invocation 1 delayed (dependent collection [1] not yet populated with datasets) (44.581 ms) galaxy.workflow.run DEBUG 2017-03-30 09:27:54,326 Workflow step 6 of invocation 1 delayed (dependent step [5] delayed, so this step must be delayed) (10.804 ms) ``` When running workflows with steps that depend explicitly on other steps (instead of implicitly between dataset connections), lines such as: ``` galaxy.workflow.run DEBUG 2017-03-30 09:33:52,966 Marking step 3 outputs delayed (workflow paused at this step waiting for review) galaxy.workflow.run DEBUG 2017-03-30 09:33:53,001 Workflow step 4 of invocation 1 delayed (dependent step [3] delayed, so this step must be delayed) (0.157 ms) galaxy.workflow.run DEBUG 2017-03-30 09:33:53,001 Workflow step 5 of invocation 1 delayed (depends on step [4] but that step has not been invoked yet) (0.084 ms) ``` and ``` galaxy.workflow.run DEBUG 2017-03-30 09:33:57,090 Workflow step 5 of invocation 1 delayed (depends on step [4] but one or more jobs created from that step have not finished yet) (0.144 ms) ``` now may appear. Commit: b8692309484c3a671145684277201ed7aaf444dd https://github.com/galaxyproject/galaxy/commit/b8692309484c3a67114568427720… Author: John Chilton <jmchilton(a)gmail.com> Date: 2017-03-30 (Thu, 30 Mar 2017) Changed paths: M lib/galaxy/workflow/run.py Log Message: ----------- Re-arrange workflow scheduling logic according to @nsoranzo's suggestions. Commit: 200e475bc7cd340401f46028fc234a1e7c2ad87c https://github.com/galaxyproject/galaxy/commit/200e475bc7cd340401f46028fc23… Author: Dannon Baker <dannon.baker(a)gmail.com> Date: 2017-03-30 (Thu, 30 Mar 2017) Changed paths: M lib/galaxy/workflow/modules.py M lib/galaxy/workflow/run.py M test/api/test_workflows.py Log Message: ----------- Merge pull request #3849 from jmchilton/improved_scheduling_delay_logging Log reason when workflow scheduling of a step is delayed. Compare: https://github.com/galaxyproject/galaxy/compare/90ac2f435662...200e475bc7cd

1 0