Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 563e363478258cb1d08802e16586c42efe511e6a https://github.com/galaxyproject/galaxy/commit/563e363478258cb1d08802e16586c... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/managers/histories.py M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Tool optimization - don't set state to queued twice. Just keep the dataset in the correct NEW state until it has actually been queued. Addresses FIXME comment added by James 7 years ago in https://github.com/galaxyproject/galaxy/commit/4c3db1af95fb0520960046ae54946.... This behavior feels correct to me, but it does have ramifications in the GUI. I had previously never actually seen a dataset in the "NEW" state. Saves an extra flush per dataset, on sqlite this translates to 50ms per dataset for me. Commit: 0ac7c33522cfeb51b89ec8d14fbb2d716f956450 https://github.com/galaxyproject/galaxy/commit/0ac7c33522cfeb51b89ec8d14fbb2... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Tool optimization and comment improvement. Commit: 2b8a1423f7e9ae6bdf09571b8992493dc30e035c https://github.com/galaxyproject/galaxy/commit/2b8a1423f7e9ae6bdf09571b89924... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Tool optimization - remove another seemingly needless flush. Saves around a teenth of a second per job. Before: - galaxy.tools.execute DEBUG 2015-12-01 13:00:56,984 Tool [cat1] created job [24] (435.468 ms) - galaxy.tools.execute DEBUG 2015-12-01 13:01:17,296 Tool [cat1] created job [25] (341.687 ms) - galaxy.tools.execute DEBUG 2015-12-01 13:01:32,896 Tool [cat1] created job [26] (354.138 ms) After - - galaxy.tools.execute DEBUG 2015-12-01 12:58:51,154 Tool [cat1] created job [21] (261.932 ms) - galaxy.tools.execute DEBUG 2015-12-01 12:59:11,354 Tool [cat1] created job [22] (278.699 ms) - galaxy.tools.execute DEBUG 2015-12-01 12:59:43,105 Tool [cat1] created job [23] (296.438 ms) Commit: 81799bd3a72b3e704be6c12e89b85eb5bd077428 https://github.com/galaxyproject/galaxy/commit/81799bd3a72b3e704be6c12e89b85... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/model/__init__.py Log Message: ----------- Micro-optimizations for adding many datasets to history... ... during tool execution. Build method once instead of in each function call, use imap instead of map since we don't need a list, remove some duplicated checks. Frankly this is all stuff Python is probably doing anyway - but in case it doesn't and just so the eye doesn't jump to these optimizations again. Timings before and after for a section of tool action execute that includes this additon show that this might have a small effect. Before: - galaxy.tools.actions INFO 2015-12-01 13:11:39,446 Add outputs to history (127.379 ms) - galaxy.tools.actions INFO 2015-12-01 13:12:06,598 Add outputs to history (137.029 ms) - galaxy.tools.actions INFO 2015-12-01 13:12:23,931 Add outputs to history (118.489 ms) After: - galaxy.tools.actions INFO 2015-12-01 13:13:09,999 Add outputs to history (99.456 ms) - galaxy.tools.actions INFO 2015-12-01 13:13:38,573 Add outputs to history (126.131 ms) - galaxy.tools.actions INFO 2015-12-01 13:13:54,538 Add outputs to history (137.643 ms) - galaxy.tools.actions INFO 2015-12-01 13:14:12,516 Add outputs to history (101.451 ms) Commit: bd6a34e279f8d612185fe9d8aa27084efe246e2d https://github.com/galaxyproject/galaxy/commit/bd6a34e279f8d612185fe9d8aa270... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Tool optimization comments for future reference. Including replacing a log statement with a comment. Commit: 59c3acfbf56e14c8f691946932be71b81774de0e https://github.com/galaxyproject/galaxy/commit/59c3acfbf56e14c8f691946932be7... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Tool execution optimization - remove another unneeded flush. Flush the job and its output datasets all at the same time in one transaction. Inuitively it makes sense that this should work and the timings seem to improve, triming another quater or third of the submission time per job. Before: - galaxy.tools.execute DEBUG 2015-12-01 12:58:51,154 Tool [cat1] created job [21] (261.932 ms) - galaxy.tools.execute DEBUG 2015-12-01 12:59:11,354 Tool [cat1] created job [22] (278.699 ms) - galaxy.tools.execute DEBUG 2015-12-01 12:59:43,105 Tool [cat1] created job [23] (296.438 ms) After: - galaxy.tools.execute DEBUG 2015-12-01 13:22:00,649 Tool [cat1] created job [39] (265.547 ms) - galaxy.tools.execute DEBUG 2015-12-01 13:22:27,720 Tool [cat1] created job [40] (214.905 ms) - galaxy.tools.execute DEBUG 2015-12-01 13:21:40,225 Tool [cat1] created job [38] (198.936 ms) - galaxy.tools.execute DEBUG 2015-12-01 13:22:44,076 Tool [cat1] created job [41] (213.096 ms) Commit: 3bf466c8f5d554f583dfc2fe8f7b95397278b2d9 https://github.com/galaxyproject/galaxy/commit/3bf466c8f5d554f583dfc2fe8f7b9... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Micro optimization in tool execute - eliminate repeated trans.app fetch. Commit: e72604be7c0c0a88c6267140b6433fcde88f1d01 https://github.com/galaxyproject/galaxy/commit/e72604be7c0c0a88c6267140b6433... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/upload.py Log Message: ----------- Instrument upload action for timing. - Checked and cleanup datasets ~500ms - Create job ~200ms - Persisting ~20 ms Commit: 2c6819cd87520fb43f4ebea96eea5fb151787edb https://github.com/galaxyproject/galaxy/commit/2c6819cd87520fb43f4ebea96eea5... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Tool optimization - eliminate repeated role generation. For cat1 with 5 inputs - this shaves about 100 ms off of job execution time. Before: galaxy.tools.execute DEBUG 2015-12-01 15:08:56,745 Tool [cat1] created job [171] (545.776 ms) galaxy.tools.execute DEBUG 2015-12-01 15:08:31,871 Tool [cat1] created job [170] (478.813 ms) After : galaxy.tools.execute DEBUG 2015-12-01 15:07:11,401 Tool [cat1] created job [168] (371.722 ms) galaxy.tools.execute DEBUG 2015-12-01 15:07:42,060 Tool [cat1] created job [169] (333.296 ms) Commit: 836b0dfc1504357b738375b109b22243dd58efe7 https://github.com/galaxyproject/galaxy/commit/836b0dfc1504357b738375b109b22... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/execute.py Log Message: ----------- Fix location of timer generation for fairer comparison in next commit. Commit: c14c45be46a4a3b08b3cb0b5132cbbdf151d5100 https://github.com/galaxyproject/galaxy/commit/c14c45be46a4a3b08b3cb0b5132cb... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/execute.py M test/unit/tools/test_execution.py Log Message: ----------- Further cache user roles across batched tool executions. Commit: 53354c7e361cc769f42c7eae42de0eca1769e9a4 https://github.com/galaxyproject/galaxy/commit/53354c7e361cc769f42c7eae42de0... Author: John Chilton <jmchilton@gmail.com> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/tools/actions/__init__.py Log Message: ----------- Fix collection output for less flushing in tool action handle_output. Commit: 4fe36f8d6a151536267e3aeace2414cc148e18e8 https://github.com/galaxyproject/galaxy/commit/4fe36f8d6a151536267e3aeace241... Author: Björn Grüning <bjoern@gruenings.eu> Date: 2015-12-06 (Sun, 06 Dec 2015) Changed paths: M lib/galaxy/managers/histories.py M lib/galaxy/model/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/actions/upload.py M lib/galaxy/tools/execute.py M test/unit/tools/test_execution.py Log Message: ----------- Merge pull request #1199 from jmchilton/tool_opts_4 Yet More Tool Execution Optimizations Compare: https://github.com/galaxyproject/galaxy/compare/beb4a67fb0a4...4fe36f8d6a15