Branch: refs/heads/dev Home: https://github.com/galaxyproject/galaxy Commit: ebac0fd9e576f5d1cac797d71d470aedb911ec7d https://github.com/galaxyproject/galaxy/commit/ebac0fd9e576f5d1cac797d71d470... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M tools/data_source/upload.py Log Message: ----------- Upload simplification - just base this check on link_only. Commit: 94cc4dfef690edbe0163eadca55d6371ef1c9968 https://github.com/galaxyproject/galaxy/commit/94cc4dfef690edbe0163eadca55d6... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/util/compression_utils.py M lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py Log Message: ----------- Refactor shed's CompressedFile abstraction into galaxy.util.compression_util. I need to use this from upload code. Commit: 306dded24862d27a3cd2ef6bb5380bd4a267de4e https://github.com/galaxyproject/galaxy/commit/306dded24862d27a3cd2ef6bb5380... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/webapps/galaxy/api/library_contents.py Log Message: ----------- Improved doc for library API. Commit: a45cbfeb8d7d7cc3df5e2561b422023986e805d0 https://github.com/galaxyproject/galaxy/commit/a45cbfeb8d7d7cc3df5e2561b4220... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M tools/data_source/upload.py Log Message: ----------- Stronger assertion about link_data_only option in upload.py. Commit: 1c6cc02415f03b999767d97b049acfd1f39388de https://github.com/galaxyproject/galaxy/commit/1c6cc02415f03b999767d97b049ac... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/actions/library.py M lib/galaxy/app.py M lib/galaxy/datatypes/sniff.py A lib/galaxy/datatypes/upload_util.py M lib/galaxy/dependencies/pinned-requirements.txt M lib/galaxy/jobs/__init__.py M lib/galaxy/managers/collections.py M lib/galaxy/model/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/upload.py M lib/galaxy/tools/actions/upload_common.py A lib/galaxy/tools/data_fetch.py A lib/galaxy/tools/data_fetch.xml M lib/galaxy/tools/execute.py M lib/galaxy/tools/parameters/output_collect.py M lib/galaxy/tools/special_tools.py A lib/galaxy/webapps/galaxy/api/_fetch_util.py M lib/galaxy/webapps/galaxy/api/tools.py M lib/galaxy/webapps/galaxy/buildapp.py A scripts/api/fetch_to_library.py A scripts/api/fetch_to_library_example.yml A test-data/example-bag.zip A test-data/testdir1.zip M test/api/test_dataset_collections.py M test/api/test_libraries.py M test/base/integration_util.py M test/base/populators.py M test/integration/test_upload_configuration_options.py M tools/data_source/upload.py Log Message: ----------- Hierarchical upload API optimized for folders & collections. Allows describing hierarchical data in JSON or inferring structure from archives or directories. Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories. Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads). The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render. This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_... with one that converts a simple YAML file into an API call and allows many new options for free. In future PRs I'll add filtering options to this and it will serve as the backend to 4733. Commit: ce2170a56ede27c9caa2a3c881fa5c582c25b196 https://github.com/galaxyproject/galaxy/commit/ce2170a56ede27c9caa2a3c881fa5... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M test/base/populators.py M test/integration/test_upload_configuration_options.py Log Message: ----------- Test case for link_to_files during upload. Commit: b6f4bffbbb89d3f96f7a292eca38e631cd950a44 https://github.com/galaxyproject/galaxy/commit/b6f4bffbbb89d3f96f7a292eca38e... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/workflow/modules.py Log Message: ----------- Do not allow workflows to run tools that are not workflow-compatible. In the case of data-fetch there is extra validation that is done so this is somewhat important. Commit: 347f1e123bb4956361384935c28714e0734ac916 https://github.com/galaxyproject/galaxy/commit/347f1e123bb4956361384935c2871... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/actions/library.py M test/base/driver_util.py M test/integration/test_upload_configuration_options.py Log Message: ----------- Don't purge library path pastes and such in upload.py during testing. Concerning that they sometimes will get deleted in production with default settings - see 5361. Commit: 3bf5990155faea13ceaa7e43b76d61cfb7ec3cd2 https://github.com/galaxyproject/galaxy/commit/3bf5990155faea13ceaa7e43b76d6... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/parameters/output_collect.py M lib/galaxy/webapps/galaxy/api/_fetch_util.py M test/integration/test_upload_configuration_options.py Log Message: ----------- Allow uploading individual HDAs via fetch API. Commit: 1720354d86c29bcc5f50664139cbf935ebc51e5b https://github.com/galaxyproject/galaxy/commit/1720354d86c29bcc5f50664139cbf... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M test/integration/test_upload_configuration_options.py Log Message: ----------- More upload testing, some input is getting deleted that shouldn't. Commit: 5c5dbd2b6587dee35fef329b9536cc627f2bf6a4 https://github.com/galaxyproject/galaxy/commit/5c5dbd2b6587dee35fef329b9536c... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/data_fetch.py M test/functional/tools/sample_datatypes_conf.xml M test/integration/test_upload_configuration_options.py Log Message: ----------- Handle compressed datatypes appropriately in data fetch API. Commit: 54ee573952215214a15cf21f20c79b735a394342 https://github.com/galaxyproject/galaxy/commit/54ee573952215214a15cf21f20c79... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/actions/upload.py Log Message: ----------- Simplify and reduce duplication of upload actions. Commit: 60f632bb242c0c73831dbe0af204218f59b38d77 https://github.com/galaxyproject/galaxy/commit/60f632bb242c0c73831dbe0af2042... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/actions/upload.py M lib/galaxy/tools/actions/upload_common.py M lib/galaxy/tools/data_fetch.py M lib/galaxy/tools/parameters/output_collect.py M lib/galaxy/webapps/galaxy/api/_fetch_util.py M test/integration/test_upload_configuration_options.py Log Message: ----------- Precreate certain outputs for upload 2.0 API. Trying to improve the user experience of this rule based uploader by placing HDAs and HDCAs in the history at the outset that the history panel can poll and that we can turn red if the upload fails.

From Marius' PR review:

I can see that a job launched in my logs, but it failed and there were no visual indications of this in the UI

Not every HDA for instance can be created, for example if reading them from a zip file for instance that happens on the backend still. Likewise if HDCAs don't define a collection type up front they cannot be pre-created (if for instance that is inferred from a folder structure). Library things aren't precreated at all in this commit. There is room to pre-create more but I think this is an atomic commit as it is now and it will hopefully improve the user experience for the rule based uploader considerably. Commit: d783fc33091e2952c2775f13a3453517bcffb9bb https://github.com/galaxyproject/galaxy/commit/d783fc33091e2952c2775f13a3453... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M test/integration/test_upload_configuration_options.py Log Message: ----------- Avoid symlinks in upload FTP tests. Commit: 33141835c9700c2fd93fc10ce6e68c6a41026377 https://github.com/galaxyproject/galaxy/commit/33141835c9700c2fd93fc10ce6e68... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/actions/upload_common.py M lib/galaxy/tools/data_fetch.py M lib/galaxy/tools/parameters/output_collect.py Log Message: ----------- Cleanup hierarchical upload commit based on PR comments from @bgruening. - Remove seemingly unneeded hack in upload_common. - Remove stray debug statement. - Add more comments in the output collection code related to different destination types. - Restructure if/else in data_fetch to avoid assertion with constant. Commit: ca28000f8b74738e50bb33aab67eb933a2df6f7a https://github.com/galaxyproject/galaxy/commit/ca28000f8b74738e50bb33aab67eb... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/parameters/output_collect.py M test/integration/test_upload_configuration_options.py Log Message: ----------- Fixes for pre-creating HDAs using data fetch API. Commit: 495d1258fd50780ea1dffd05eeef558161dc041d https://github.com/galaxyproject/galaxy/commit/495d1258fd50780ea1dffd05eeef5... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M tools/data_source/upload.py Log Message: ----------- Consistent sniffing regardless of in_place. Previously sniffing would happen on the original file (before carriage returns and tabular spaces were converted) if in_place was false and on the converted file if it was true. Commit: d651348da663966c1c04585f557820af53c5f651 https://github.com/galaxyproject/galaxy/commit/d651348da663966c1c04585f55782... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/datatypes/sniff.py M lib/galaxy/tools/data_fetch.py A test-data/1.csv M test/api/test_tools_upload.py M test/functional/tools/sample_datatypes_conf.xml Log Message: ----------- More upload tests and fixes. Commit: bca2c3cf173604910cc06d53afa9e0acbb02df49 https://github.com/galaxyproject/galaxy/commit/bca2c3cf173604910cc06d53afa9e... Author: John Chilton <jmchilton@gmail.com> Date: 2018-03-08 (Thu, 08 Mar 2018) Changed paths: M lib/galaxy/tools/data_fetch.py Log Message: ----------- Fix for data_fetch sniffing. Commit: 21959113e7a02d9846d5ab311a9d412cd8eba18e https://github.com/galaxyproject/galaxy/commit/21959113e7a02d9846d5ab311a9d4... Author: Björn Grüning <bjoern@gruenings.eu> Date: 2018-03-09 (Fri, 09 Mar 2018) Changed paths: M lib/galaxy/actions/library.py M lib/galaxy/app.py M lib/galaxy/datatypes/sniff.py A lib/galaxy/datatypes/upload_util.py M lib/galaxy/dependencies/pinned-requirements.txt M lib/galaxy/jobs/__init__.py M lib/galaxy/managers/collections.py M lib/galaxy/model/__init__.py M lib/galaxy/tools/__init__.py M lib/galaxy/tools/actions/upload.py M lib/galaxy/tools/actions/upload_common.py A lib/galaxy/tools/data_fetch.py A lib/galaxy/tools/data_fetch.xml M lib/galaxy/tools/execute.py M lib/galaxy/tools/parameters/output_collect.py M lib/galaxy/tools/special_tools.py M lib/galaxy/util/compression_utils.py A lib/galaxy/webapps/galaxy/api/_fetch_util.py M lib/galaxy/webapps/galaxy/api/library_contents.py M lib/galaxy/webapps/galaxy/api/tools.py M lib/galaxy/webapps/galaxy/buildapp.py M lib/galaxy/workflow/modules.py M lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py A scripts/api/fetch_to_library.py A scripts/api/fetch_to_library_example.yml A test-data/1.csv A test-data/example-bag.zip A test-data/testdir1.zip M test/api/test_dataset_collections.py M test/api/test_libraries.py M test/api/test_tools_upload.py M test/base/driver_util.py M test/base/integration_util.py M test/base/populators.py M test/functional/tools/sample_datatypes_conf.xml M test/integration/test_upload_configuration_options.py M tools/data_source/upload.py Log Message: ----------- Merge pull request #5220 from jmchilton/upload_2.0 Hierarchical upload API optimized for folders & collections. Compare: https://github.com/galaxyproject/galaxy/compare/300fffe73ee0...21959113e7a0