Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: dd7bddcaa682d6d7cdaba8661c2c7cc80997ade0 https://github.com/galaxyproject/galaxy/commit/dd7bddcaa682d6d7cdaba8661c2c7... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M lib/galaxy/jobs/__init__.py M lib/galaxy/jobs/runners/__init__.py M lib/galaxy/model/__init__.py

Log Message: ----------- Optimize initial queueing of jobs.

- Don't refresh job when this is the only thread fetching it. - Avoid a bunch of unnecessary flushes, just flush once essentially during process.

Brings this process from taking over 1 second per job on average on sqlite for cat1 on my laptop to around 400 ms on average.

Commit: 299fbe99d7e157b9c01bbf0cdfcb7057dd59791a https://github.com/galaxyproject/galaxy/commit/299fbe99d7e157b9c01bbf0cdfcb7... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M lib/galaxy/tools/__init__.py

Log Message: ----------- Micro optimization, only fetch tool validation once per multi-run.

Commit: 876c28f88d81a8599a3278d9deac730adbac10d9 https://github.com/galaxyproject/galaxy/commit/876c28f88d81a8599a3278d9deac7... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M lib/galaxy/tools/__init__.py M lib/galaxy/tools/execute.py

Log Message: ----------- Add more job timers.

Commit: 41d3dc62f7e967ea3b409be410310563d0737bc4 https://github.com/galaxyproject/galaxy/commit/41d3dc62f7e967ea3b409be410310... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/managers/collections.py M lib/galaxy/model/__init__.py M lib/galaxy/security/__init__.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/execute.py

Log Message: ----------- Parallelize executing tool parameter combinations.

Should work when mapping over collections or for big muli-run tool submissions.

Because of database tension with sqlalchemy it is not strictly a linear increase, but the end user walltime experience for a 24 dataset collection being submitted using 4 threads instead 1 drops execution time from 68 seconds to 35.

Rebased with fixes thanks to @nsoranzo - https://github.com/jmchilton/galaxy/commit/7f6514a21222ea787a0676764e4a4fe82....

Commit: 64b7cc9fef21caac5f9d21a8d42b60262fd28162 https://github.com/galaxyproject/galaxy/commit/64b7cc9fef21caac5f9d21a8d42b6... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: A test/functional/tools/create_10.xml M test/functional/tools/samples_tool_conf.xml

Log Message: ----------- Add sample tool that creates 10 outputs.

Commit: a190b69e33a41717ddcc9eaa82826a82cc3d69b1 https://github.com/galaxyproject/galaxy/commit/a190b69e33a41717ddcc9eaa82826... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py M lib/galaxy/tools/actions/__init__.py

Log Message: ----------- Optimize tools that produce multiple outputs.

Fetch HIDs for these datasets all at once and flush once for all assignments. Creating jobs from 4 threads each job producing 10 outputs - this resulted in reducing the runtime of this portion of the code by 90% and saving around 20 seconds per job.

Before:

galaxy.tools.actions INFO 2015-11-30 15:53:46,235 Add outputs to history (22494.829 ms) galaxy.tools.actions INFO 2015-11-30 15:53:47,295 Add outputs to history (22239.062 ms) galaxy.tools.actions INFO 2015-11-30 15:53:48,320 Add outputs to history (23781.079 ms) galaxy.tools.actions INFO 2015-11-30 15:53:51,490 Add outputs to history (21820.567 ms) galaxy.tools.actions INFO 2015-11-30 15:54:24,506 Add outputs to history (25353.837 ms)

After:

galaxy.tools.actions INFO 2015-11-30 16:08:47,640 Add outputs to history (2781.675 ms) galaxy.tools.actions INFO 2015-11-30 16:08:47,860 Add outputs to history (3177.738 ms) galaxy.tools.actions INFO 2015-11-30 16:08:48,776 Add outputs to history (2425.528 ms) galaxy.tools.actions INFO 2015-11-30 16:09:02,942 Add outputs to history (2579.022 ms)

Commit: ab2c7c2c079b51ffd4ed7ef00724ef3f8d2636ec https://github.com/galaxyproject/galaxy/commit/ab2c7c2c079b51ffd4ed7ef00724e... Author: John Chilton jmchilton@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M lib/galaxy/tools/actions/__init__.py

Log Message: ----------- Tool execution micro optimization...

... Don't create an extra list to reverse, just iterate over in reversed order.

Commit: 9a1fc647a38007fef2a7a179648ea8f925ab5a8f https://github.com/galaxyproject/galaxy/commit/9a1fc647a38007fef2a7a179648ea... Author: Martin Cech cech.marten@gmail.com Date: 2015-12-03 (Thu, 03 Dec 2015)

Changed paths: M config/galaxy.ini.sample M lib/galaxy/config.py M lib/galaxy/jobs/__init__.py M lib/galaxy/jobs/runners/__init__.py M lib/galaxy/managers/collections.py M lib/galaxy/model/__init__.py M lib/galaxy/model/mapping.py M lib/galaxy/security/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/__init__.py M lib/galaxy/tools/execute.py A test/functional/tools/create_10.xml M test/functional/tools/samples_tool_conf.xml

Log Message: ----------- Merge pull request #1212 from jmchilton/tool_opts_6

Smaller Tool Optimization PR

Compare: https://github.com/galaxyproject/galaxy/compare/14ac3665ee37...9a1fc647a380