Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: 93d5c54946e992112841df3461512bbeb6f5c7e9 https://github.com/galaxyproject/galaxy/commit/93d5c54946e992112841df3461512... Author: John Chilton jmchilton@gmail.com Date: 2018-02-21 (Wed, 21 Feb 2018)

Changed paths: M lib/galaxy/web/framework/helpers/grids.py

Log Message: ----------- Optimize count query in grids (part 1).

Many grid operations execute a complex query twice - once to produce the total number of rows and once to iterate through the selected parts of the data. This can be more optimal in many ways - this starts us down the path to addressing that by doing two things.

The first thing it does is eliminates the query all together if paging is enabled but the user has opted to see "all" the rows. There was never a reason to execute the query in that case I don't think.

The second thing it does is move the calculation of the count query until after the data has been generated - this isn't more optimal by itself but it allows us to potentially calculate the count as part of the main query. I've implemented this in a broken way @ https://github.com/galaxyproject/galaxy/commit/ea404721585b5f78da1b91e57e31b... but I realized it was too broad an approach and doesn't work if certain kinds of joins are used in the initial query - but a more targetted approach on a per-grid basis could work I think and moving this code after allow us to potentially do that more easily in subsequent commits.

Additionally I added a note - that I think it would be best to just return None as the number of pages and issue a second query to the same endpoint with the same arguments but with count_only=true added as a query request parameter so we can render the main grid faster and then render the paging elements after the second query finishes. This would feel more responsive I think.

Commit: 1656c7b429854323efda82accc4c190a965be8a3 https://github.com/galaxyproject/galaxy/commit/1656c7b429854323efda82accc4c1... Author: John Chilton jmchilton@gmail.com Date: 2018-02-21 (Wed, 21 Feb 2018)

Changed paths: M lib/galaxy/tools/parameters/output_collect.py

Log Message: ----------- Fix repeated fetching dataset permissions in dynamic output collection.

Commit: c00e772159f6a121bc4f7759fa57f48e79c48f32 https://github.com/galaxyproject/galaxy/commit/c00e772159f6a121bc4f7759fa57f... Author: John Chilton jmchilton@gmail.com Date: 2018-02-21 (Wed, 21 Feb 2018)

Changed paths: M client/galaxy/scripts/mvc/grid/grid-shared.js M client/galaxy/scripts/mvc/grid/grid-view.js M client/galaxy/scripts/mvc/history/history-list.js M client/galaxy/scripts/mvc/list/list-view.js M client/galaxy/scripts/mvc/workflow/workflow.js M client/galaxy/scripts/ui/loading-indicator.js

Log Message: ----------- Fix lack of feedback when slow grids are loading data from the server.

Fixes #5473 (after numerous other backend optimizations also).

Commit: b861b9ad4d693000ba8cf492ce6b1ff057db3d34 https://github.com/galaxyproject/galaxy/commit/b861b9ad4d693000ba8cf492ce6b1... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-21 (Wed, 21 Feb 2018)

Changed paths: M lib/galaxy/tools/toolbox/base.py

Log Message: ----------- implement basic cache on toolbox to_dict objects

base the cache on absence of filters

allow cache for users that don't opt in for filters

Commit: 8806030f53c2b9d0002505bf5193be88d7cff8f9 https://github.com/galaxyproject/galaxy/commit/8806030f53c2b9d0002505bf5193b... Author: Nicola Soranzo nicola.soranzo@earlham.ac.uk Date: 2018-02-22 (Thu, 22 Feb 2018)

Changed paths: M lib/galaxy/datatypes/converters/sam_to_bam.py M lib/galaxy/dependencies/pinned-requirements.txt M lib/galaxy/dependencies/requirements.txt M lib/galaxy/jobs/runners/pulsar.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/deps/conda_util.py M lib/galaxy/tools/deps/mulled/util.py M lib/galaxy/tools/toolbox/lineages/interface.py M lib/galaxy/visualization/data_providers/genome.py

Log Message: ----------- Replace distutils.version with packaging.version

distutils under Python3 has this bug:

https://bugs.python.org/issue14894

which causes this traceback:

``` Traceback (most recent call last): File "lib/galaxy/webapps/galaxy/buildapp.py", line 49, in app_factory app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs) File "lib/galaxy/app.py", line 125, in __init__ self._configure_toolbox() File "lib/galaxy/config.py", line 930, in _configure_toolbox self.toolbox = tools.ToolBox(tool_configs, self.config.tool_path, self) File "lib/galaxy/tools/__init__.py", line 232, in __init__ app=app, File "lib/galaxy/tools/toolbox/base.py", line 1056, in __init__ super(BaseGalaxyToolBox, self).__init__(config_filenames, tool_root_dir, app) File "lib/galaxy/tools/toolbox/base.py", line 84, in __init__ self._load_tool_panel() File "lib/galaxy/tools/toolbox/base.py", line 358, in _load_tool_panel self.__add_tool_to_tool_panel(section_val, section, section=True) File "lib/galaxy/tools/toolbox/base.py", line 278, in __add_tool_to_tool_panel related_tool = self._lineage_in_panel(panel_dict, tool=tool) File "lib/galaxy/tools/toolbox/base.py", line 961, in _lineage_in_panel lineage_tool_versions = reversed(tool_lineage.get_versions()) File "lib/galaxy/tools/toolbox/lineages/interface.py", line 77, in get_versions return [ToolLineageVersion(tool_id, tool_version) for tool_id, tool_version in zip(self.tool_ids, self.tool_versions)] File "lib/galaxy/tools/toolbox/lineages/interface.py", line 55, in tool_ids return ["%s/%s" % (tool_id, version) for version in self.tool_versions] File "lib/galaxy/tools/toolbox/lineages/interface.py", line 49, in tool_versions return sorted(self._tool_versions, key=LooseVersion) File "/usr/lib/python3.4/distutils/version.py", line 58, in __lt__ c = self._cmp(other) File "/usr/lib/python3.4/distutils/version.py", line 343, in _cmp if self.version < other.version: TypeError: unorderable types: str() < int() ```

when a tool version contains a letter in the "wrong" place (e.g. in the first position).

Fix https://github.com/galaxyproject/galaxy/issues/5408 .

Commit: ce3b40cfc10e134d5acc477757404eba40ea6553 https://github.com/galaxyproject/galaxy/commit/ce3b40cfc10e134d5acc477757404... Author: Marius van den Beek m.vandenbeek@gmail.com Date: 2018-02-22 (Thu, 22 Feb 2018)

Changed paths: M lib/galaxy/tools/parameters/output_collect.py

Log Message: ----------- Merge pull request #5578 from jmchilton/opt_dyn_collect

[18.01] Fix repeated fetching dataset permissions in dynamic output collection.

Commit: fc52fe5c62700b4f8527e4edead0fba4e7cfe33f https://github.com/galaxyproject/galaxy/commit/fc52fe5c62700b4f8527e4edead0f... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-22 (Thu, 22 Feb 2018)

Changed paths: M lib/galaxy/datatypes/converters/sam_to_bam.py M lib/galaxy/dependencies/pinned-requirements.txt M lib/galaxy/dependencies/requirements.txt M lib/galaxy/jobs/runners/pulsar.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/deps/conda_util.py M lib/galaxy/tools/deps/mulled/util.py M lib/galaxy/tools/toolbox/lineages/interface.py M lib/galaxy/visualization/data_providers/genome.py

Log Message: ----------- Merge pull request #5588 from jmchilton/backport_packaging

[18.01] Replace distutils.version with packaging.version

Commit: 8a88cb9827b68e6795cf00496008e1aa189185ad https://github.com/galaxyproject/galaxy/commit/8a88cb9827b68e6795cf00496008e... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-26 (Mon, 26 Feb 2018)

Changed paths: M lib/galaxy/tools/toolbox/base.py

Log Message: ----------- move the cache from toolbox level to tool level

the cache remains toolbox-specific so it is stored within the toolbox object

includes refactor thanks to eagle eyes

Commit: e3ced2c669485f76eae7fcfb3b32aa048730964d https://github.com/galaxyproject/galaxy/commit/e3ced2c669485f76eae7fcfb3b32a... Author: Dannon Baker dannon.baker@gmail.com Date: 2018-02-26 (Mon, 26 Feb 2018)

Changed paths: M lib/galaxy/model/migrate/versions/0137_add_copied_from_job_id_column.py

Log Message: ----------- Fix migration 0137's downgrade -- previously failed to select the correct column to drop

Commit: c8407786f25c98c87f441f15222f72eeea4edc72 https://github.com/galaxyproject/galaxy/commit/c8407786f25c98c87f441f15222f7... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-26 (Mon, 26 Feb 2018)

Changed paths: M lib/galaxy/model/migrate/versions/0137_add_copied_from_job_id_column.py

Log Message: ----------- Merge pull request #5605 from dannon/migrate_fix

[18.01] Fix migration 0137's downgrade

Commit: d8585f6a6f87735d072635495bd49047da746624 https://github.com/galaxyproject/galaxy/commit/d8585f6a6f87735d072635495bd49... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-27 (Tue, 27 Feb 2018)

Changed paths: M client/galaxy/scripts/mvc/grid/grid-shared.js M client/galaxy/scripts/mvc/grid/grid-view.js M client/galaxy/scripts/mvc/history/history-list.js M client/galaxy/scripts/mvc/list/list-view.js M client/galaxy/scripts/mvc/workflow/workflow.js M client/galaxy/scripts/ui/loading-indicator.js

Log Message: ----------- Merge pull request #5582 from jmchilton/indicators

[18.01] Fix lack of feedback when slow grids are loading data from the server.

Commit: 50cd159d4c212a36b8d3861295125ef055a296d9 https://github.com/galaxyproject/galaxy/commit/50cd159d4c212a36b8d3861295125... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-27 (Tue, 27 Feb 2018)

Changed paths: M lib/galaxy/web/framework/helpers/grids.py

Log Message: ----------- Merge pull request #5574 from jmchilton/optimize_count_query_1

[18.01] Optimize count query in grids (Part 1).

Commit: a90c3a1d3f2c2a12f285f3c21e91ca364f81853e https://github.com/galaxyproject/galaxy/commit/a90c3a1d3f2c2a12f285f3c21e91c... Author: John Chilton jmchilton@gmail.com Date: 2018-02-27 (Tue, 27 Feb 2018)

Changed paths: M lib/galaxy/tools/toolbox/base.py

Log Message: ----------- Merge pull request #5583 from martenson/backport-toolbox-cache

[18.01] Backport toolbox cache

Commit: 4caaa70006cead1c286c1bbf5f036860a3fd8a85 https://github.com/galaxyproject/galaxy/commit/4caaa70006cead1c286c1bbf5f036... Author: Martin Cech marten@bx.psu.edu Date: 2018-02-27 (Tue, 27 Feb 2018)

Changed paths: M client/galaxy/scripts/mvc/grid/grid-shared.js M client/galaxy/scripts/mvc/grid/grid-view.js M client/galaxy/scripts/mvc/history/history-list.js M client/galaxy/scripts/mvc/list/list-view.js M client/galaxy/scripts/mvc/workflow/workflow.js M client/galaxy/scripts/ui/loading-indicator.js M lib/galaxy/model/migrate/versions/0137_add_copied_from_job_id_column.py M lib/galaxy/tools/parameters/output_collect.py M lib/galaxy/web/framework/helpers/grids.py

Log Message: ----------- Merge branch 'release_18.01' into dev

Compare: https://github.com/galaxyproject/galaxy/compare/73a0d72e0c98...4caaa70006ce