Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: bb19c7e44844bd724ba2ad3920abd078aae083a0 https://github.com/galaxyproject/galaxy/commit/bb19c7e44844bd724ba2ad3920abd... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-02-22 (Thu, 22 Feb 2018) Changed paths: M config/datatypes_conf.xml.sample M lib/galaxy/datatypes/binary.py R lib/galaxy/datatypes/converters/bam_native_to_bam_converter.xml M lib/galaxy/datatypes/converters/sam_to_bam_native.xml A lib/galaxy/datatypes/converters/to_coordinate_sorted_bam.xml A lib/galaxy/datatypes/converters/to_qname_sorted_bam.xml A test-data/qname_sorted.bam M test/functional/tools/sam_to_bam_native.xml M test/functional/tools/sample_datatypes_conf.xml Log Message: ----------- Add queryname sorted and input sorted datatypes The queryname sorted datatype (BamQuerynameSorted) ensures that the file is queryname ordered. The BamInputSorted datatype can be used to describe the output of aligners, which usually keep mate pairs adjacent. The BamInputSorted datatype allows using tools that require mates to be adjacent, without requiring an explicit sorting step. This can reduce the time and space required for duplicate marking tools, HiC tools and structural variant detection tools that require mates to be grouped together in an alignment file. The BamQuerynameSorted has a converter that works for all datatypes that can be consumed by samtools. There is no converter for BamInputSorted, since BamQuerynameSorted is valid input for tools requiring BamInputSorted input. I'm targeting 18.01 since I think it would be very unfortunate if we start annotating aligners with the very loose BamNative datatype, since this datatype does not make any promises about sort order at all. This closes https://github.com/galaxyproject/galaxy/issues/5497, goes a long way towards the problems mentioned in https://github.com/galaxyproject/galaxy/issues/5496 and would make https://github.com/galaxyproject/tools-iuc/pull/1732 and https://github.com/galaxyproject/tools-iuc/pull/1591 better Commit: d648e22080ac97b3940799d35e3be304d1a0aed7 https://github.com/galaxyproject/galaxy/commit/d648e22080ac97b3940799d35e3be... Author: Martin Cech <marten@bx.psu.edu> Date: 2018-03-01 (Thu, 01 Mar 2018) Changed paths: M lib/galaxy/tools/toolbox/base.py M lib/galaxy/tools/toolbox/panel.py Log Message: ----------- fix for cases where panel elements have to_dict with different signature we want to only cache Tools anyways, so this fixes other possible bug where section/label etc would have same id as a Tool change the detection of Tool object to avoid circular import errors Also use cache for dictifying sections, we should gain more speed from that. Commit: 5406a7153bd6a51174145d04eda26b6796211154 https://github.com/galaxyproject/galaxy/commit/5406a7153bd6a51174145d04eda26... Author: Martin Cech <marten@bx.psu.edu> Date: 2018-03-01 (Thu, 01 Mar 2018) Changed paths: M lib/galaxy/tools/toolbox/base.py M lib/galaxy/tools/toolbox/panel.py Log Message: ----------- pass the toolbox obj to make sure cache is available Commit: b96fd2c74010009a7dc85c09e77f87a9d4eff271 https://github.com/galaxyproject/galaxy/commit/b96fd2c74010009a7dc85c09e77f8... Author: Martin Cech <marten@bx.psu.edu> Date: 2018-03-01 (Thu, 01 Mar 2018) Changed paths: M lib/galaxy/tools/toolbox/base.py M lib/galaxy/tools/toolbox/panel.py Log Message: ----------- enable cache for tools that inherit from Tool Commit: 77d085a40f5d5d2e131dc2fc8073e0c030599988 https://github.com/galaxyproject/galaxy/commit/77d085a40f5d5d2e131dc2fc8073e... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-01 (Thu, 01 Mar 2018) Changed paths: M config/datatypes_conf.xml.sample M lib/galaxy/datatypes/binary.py R lib/galaxy/datatypes/converters/bam_native_to_bam_converter.xml M lib/galaxy/datatypes/converters/sam_to_bam_native.xml A lib/galaxy/datatypes/converters/to_coordinate_sorted_bam.xml A lib/galaxy/datatypes/converters/to_qname_sorted_bam.xml A test-data/qname_sorted.bam M test/functional/tools/sam_to_bam_native.xml M test/functional/tools/sample_datatypes_conf.xml Log Message: ----------- Merge pull request #5589 from mvdbeek/samtools_queryname_and_input_order_datatypes [18.01] Add queryname sorted and input sorted datatypes Commit: 41910f824ea311e8ce7d93e4dc03a44c75c8132b https://github.com/galaxyproject/galaxy/commit/41910f824ea311e8ce7d93e4dc03a... Author: Martin Cech <marten@bx.psu.edu> Date: 2018-03-01 (Thu, 01 Mar 2018) Changed paths: M lib/galaxy/tools/toolbox/base.py M lib/galaxy/tools/toolbox/panel.py Log Message: ----------- Merge pull request #5615 from martenson/distinct-elts [18.01] fix for cases where panel elements have to_dict with different signature Commit: 7254d55e8b192ff4387ef6a276e202db5821ca60 https://github.com/galaxyproject/galaxy/commit/7254d55e8b192ff4387ef6a276e20... Author: guerler <aysam.guerler@gmail.com> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M client/galaxy/scripts/mvc/grid/grid-template.js M lib/galaxy/web/framework/helpers/grids.py Log Message: ----------- Remove href from grid sorting links Commit: 7ffabb1df3f6feb11a3746c2f27bc762b1af29c7 https://github.com/galaxyproject/galaxy/commit/7ffabb1df3f6feb11a3746c2f27bc... Author: guerler <aysam.guerler@gmail.com> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M lib/galaxy/web/framework/helpers/grids.py Log Message: ----------- Remove unused variable Commit: bd69278f0bcc67afc1eb827a8a7bc80ae7766072 https://github.com/galaxyproject/galaxy/commit/bd69278f0bcc67afc1eb827a8a7bc... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M test/functional/tools/multi_output_recurse.xml Log Message: ----------- Add a testcase for optional output discovery This test correctly fails with: ``` galaxy.jobs.runners.local ERROR 2018-03-02 17:55:32,396 Job wrapper finish method failed Traceback (most recent call last): File "/Users/mvandenb/src/galaxy/lib/galaxy/jobs/runners/local.py", line 152, in queue_job self._finish_or_resubmit_job(job_state, stdout, stderr, exit_code) File "/Users/mvandenb/src/galaxy/lib/galaxy/jobs/runners/__init__.py", line 432, in _finish_or_resubmit_job job_state.job_wrapper.finish(stdout, stderr, exit_code, check_output_detected_state=check_output_detected_state) File "/Users/mvandenb/src/galaxy/lib/galaxy/jobs/__init__.py", line 1381, in finish 'primary': self.tool.collect_primary_datasets(out_data, self.get_tool_provided_job_metadata(), tool_working_directory, input_ext, input_dbkey) File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/__init__.py", line 1614, in collect_primary_datasets return output_collect.collect_primary_datasets(self, output, tool_provided_metadata, job_working_directory, input_ext, input_dbkey=input_dbkey) File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 372, in collect_primary_datasets for discovered_file in discover_files(name, tool_provided_metadata, dataset_collectors, job_working_directory, outdata): File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 490, in discover_files for (match, collector) in walk_over_file_collectors(extra_file_collectors, job_working_directory, matchable): File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 507, in walk_over_file_collectors for match in walk_over_extra_files(extra_file_collector.directory, extra_file_collector, job_working_directory, matchable): File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 519, in walk_over_extra_files for filename in os.listdir(directory): OSError: [Errno 2] No such file or directory: '/private/var/folders/df/6xqpqpcd7h73b6jpx9t6cwhw0000gn/T/tmp5Tz0m9/tmpfUHHl6/tmpVBcnfI/database/job_working_directory_LJioiW/000/2/working/i_do_not_exist' ``` Commit: d21d7d9d2e10d9c09bdbadbf328fd3eef84630e6 https://github.com/galaxyproject/galaxy/commit/d21d7d9d2e10d9c09bdbadbf328fd... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M lib/galaxy/tools/parameters/output_collect.py Log Message: ----------- Fix output discovery when directory doesn't exist This broke in 8b76a5a698fd3bc13d4acc38194221def9ed5163. Checking if `directory` is a directory is what we had done previously. Commit: e01f9fa1b89e6a5fd12f9ef98ddb005959e1b761 https://github.com/galaxyproject/galaxy/commit/e01f9fa1b89e6a5fd12f9ef98ddb0... Author: Martin Cech <marten@bx.psu.edu> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M client/galaxy/scripts/mvc/grid/grid-template.js M lib/galaxy/web/framework/helpers/grids.py Log Message: ----------- Merge pull request #5634 from guerler/fix_async_grid_links [18.01] Remove href from grid sorting links Commit: 7ed81e18101c33a1e83bb5e72851698910a118e9 https://github.com/galaxyproject/galaxy/commit/7ed81e18101c33a1e83bb5e728516... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M test/functional/tools/multi_output_recurse.xml Log Message: ----------- Move `optional` tag to the output data element Thx @nsoranzo! Commit: da867852ae6437c02c64287521dde0b8aa1e8109 https://github.com/galaxyproject/galaxy/commit/da867852ae6437c02c64287521dde... Author: mvdbeek <m.vandenbeek@gmail.com> Date: 2018-03-02 (Fri, 02 Mar 2018) Changed paths: M test/functional/tools/multi_output_recurse.xml Log Message: ----------- Drop optional="true" altogther Commit: d77845268a9243535e2b77383bc4ed3d433e2e7a https://github.com/galaxyproject/galaxy/commit/d77845268a9243535e2b77383bc4e... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-04 (Sun, 04 Mar 2018) Changed paths: M lib/galaxy/tools/parameters/output_collect.py M test/functional/tools/multi_output_recurse.xml Log Message: ----------- Merge pull request #5635 from mvdbeek/fix_optional_output_discovery [18.01] Fix output discovery if dir doesn't exist Commit: a6593d4cafe966062460e33eaac1c5a28566c109 https://github.com/galaxyproject/galaxy/commit/a6593d4cafe966062460e33eaac1c... Author: Matthias Bernt <m.bernt@ufz.de> Date: 2018-03-05 (Mon, 05 Mar 2018) Changed paths: M tools/stats/gsummary.py Log Message: ----------- python3 fix strings can not be written to byte mode file handlers. writing resulted in: TypeError: a bytes-like object is required, not 'str' Alternative would be to make string byte. Commit: 28724ab795ac5dfcc5fa5b9a9a88fb0b64fc7752 https://github.com/galaxyproject/galaxy/commit/28724ab795ac5dfcc5fa5b9a9a88f... Author: Dannon Baker <dannon.baker@gmail.com> Date: 2018-03-05 (Mon, 05 Mar 2018) Changed paths: M tools/stats/gsummary.py Log Message: ----------- Merge pull request #5639 from nsoranzo/release_17.09_backport_5043 [17.09] Backport of #5043 Commit: ff9bc843fab96ab95b7a5ae69c71a99185977b96 https://github.com/galaxyproject/galaxy/commit/ff9bc843fab96ab95b7a5ae69c71a... Author: Nicola Soranzo <nicola.soranzo@earlham.ac.uk> Date: 2018-03-05 (Mon, 05 Mar 2018) Log Message: ----------- Merge branch 'release_17.09' into release_18.01 Commit: 1118e6f1be87701eecce2ce99163a3ff9551f9d6 https://github.com/galaxyproject/galaxy/commit/1118e6f1be87701eecce2ce99163a... Author: Nicola Soranzo <nicola.soranzo@earlham.ac.uk> Date: 2018-03-05 (Mon, 05 Mar 2018) Changed paths: M client/galaxy/scripts/mvc/grid/grid-template.js M config/datatypes_conf.xml.sample M lib/galaxy/datatypes/binary.py R lib/galaxy/datatypes/converters/bam_native_to_bam_converter.xml M lib/galaxy/datatypes/converters/sam_to_bam_native.xml A lib/galaxy/datatypes/converters/to_coordinate_sorted_bam.xml A lib/galaxy/datatypes/converters/to_qname_sorted_bam.xml M lib/galaxy/tools/parameters/output_collect.py M lib/galaxy/web/framework/helpers/grids.py A test-data/qname_sorted.bam M test/functional/tools/multi_output_recurse.xml M test/functional/tools/sam_to_bam_native.xml M test/functional/tools/sample_datatypes_conf.xml Log Message: ----------- Merge branch 'release_18.01' into dev Conflicts: lib/galaxy/datatypes/binary.py Compare: https://github.com/galaxyproject/galaxy/compare/3aa7af6372e7...1118e6f1be87