- galaxy-commits - lists.galaxyproject.org

commit/galaxy-central: 2 new changesets
by commits-noreply＠bitbucket.org 18 Mar '14

18 Mar '14

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/a17810e4f3ce/ Changeset: a17810e4f3ce User: jmchilton Date: 2014-03-14 16:12:41 Summary: Check ftype attribute (if defined) on test output datasets. Only works for newer API driven tests. Affected #: 2 files diff -r 3fb927653301a0c06a0bf94f2b6bd71b3595ec0d -r a17810e4f3ce1fce18565b33c1306885dfb3ef38 test/base/interactor.py --- a/test/base/interactor.py +++ b/test/base/interactor.py @@ -45,13 +45,20 @@ fetcher = self.__dataset_fetcher( history_id ) ## TODO: Twill version verifys dataset is 'ok' in here. self.twill_test_case.verify_hid( outfile, hda_id=hid, attributes=attributes, dataset_fetcher=fetcher, shed_tool_id=shed_tool_id ) - metadata = attributes.get( 'metadata', {} ) + metadata = attributes.get( 'metadata', {} ).copy() + for key, value in metadata.copy().iteritems(): + new_key = "metadata_%s" % key + metadata[ new_key ] = metadata[ key ] + del metadata[ key ] + expected_file_type = attributes.get( 'ftype', None ) + if expected_file_type: + metadata[ "file_ext" ] = expected_file_type + if metadata: dataset = self._get( "histories/%s/contents/%s" % ( history_id, hid ) ).json() for key, value in metadata.iteritems(): - dataset_key = "metadata_%s" % key try: - dataset_value = dataset.get( dataset_key, None ) + dataset_value = dataset.get( key, None ) if dataset_value != value: msg = "Dataset metadata verification for [%s] failed, expected [%s] but found [%s]." msg_params = ( key, value, dataset_value ) diff -r 3fb927653301a0c06a0bf94f2b6bd71b3595ec0d -r a17810e4f3ce1fce18565b33c1306885dfb3ef38 test/functional/tools/metadata.xml --- a/test/functional/tools/metadata.xml +++ b/test/functional/tools/metadata.xml @@ -16,7 +16,7 @@ <metadata name="base_name" value="Example Metadata" /></param> - <output name="output_of_input_metadata"> + <output name="output_of_input_metadata" ftype="txt"><assert_contents><has_line line="Example Metadata" /></assert_contents> https://bitbucket.org/galaxy/galaxy-central/commits/327583f6b058/ Changeset: 327583f6b058 User: jmchilton Date: 2014-03-18 18:42:23 Summary: Merged in jmchilton/galaxy-central-fork-1 (pull request #347) Check ftype attribute (if defined) on test output datasets. Affected #: 2 files diff -r 0a9263af45ec678bea82a6d128f49cf5372d370d -r 327583f6b0581c2bdf22b933b2659298af2ed699 test/base/interactor.py --- a/test/base/interactor.py +++ b/test/base/interactor.py @@ -45,13 +45,20 @@ fetcher = self.__dataset_fetcher( history_id ) ## TODO: Twill version verifys dataset is 'ok' in here. self.twill_test_case.verify_hid( outfile, hda_id=hid, attributes=attributes, dataset_fetcher=fetcher, shed_tool_id=shed_tool_id ) - metadata = attributes.get( 'metadata', {} ) + metadata = attributes.get( 'metadata', {} ).copy() + for key, value in metadata.copy().iteritems(): + new_key = "metadata_%s" % key + metadata[ new_key ] = metadata[ key ] + del metadata[ key ] + expected_file_type = attributes.get( 'ftype', None ) + if expected_file_type: + metadata[ "file_ext" ] = expected_file_type + if metadata: dataset = self._get( "histories/%s/contents/%s" % ( history_id, hid ) ).json() for key, value in metadata.iteritems(): - dataset_key = "metadata_%s" % key try: - dataset_value = dataset.get( dataset_key, None ) + dataset_value = dataset.get( key, None ) if dataset_value != value: msg = "Dataset metadata verification for [%s] failed, expected [%s] but found [%s]." msg_params = ( key, value, dataset_value ) diff -r 0a9263af45ec678bea82a6d128f49cf5372d370d -r 327583f6b0581c2bdf22b933b2659298af2ed699 test/functional/tools/metadata.xml --- a/test/functional/tools/metadata.xml +++ b/test/functional/tools/metadata.xml @@ -16,7 +16,7 @@ <metadata name="base_name" value="Example Metadata" /></param> - <output name="output_of_input_metadata"> + <output name="output_of_input_metadata" ftype="txt"><assert_contents><has_line line="Example Metadata" /></assert_contents> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: dannon: Update boto to 2.27.0 -- thirequired update to support VPC support that worked incorrectly in previous versions.
by commits-noreply＠bitbucket.org 18 Mar '14

18 Mar '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/0a9263af45ec/ Changeset: 0a9263af45ec User: dannon Date: 2014-03-18 16:04:55 Summary: Update boto to 2.27.0 -- thirequired update to support VPC support that worked incorrectly in previous versions. Affected #: 1 file diff -r 91aa1db362c9f5c76160b09bab52b8a77b0f30b0 -r 0a9263af45ec678bea82a6d128f49cf5372d370d eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -35,7 +35,7 @@ amqp = 1.4.3 anyjson = 0.3.3 Beaker = 1.4 -boto = 2.25.0 +boto = 2.27.0 decorator = 3.1.2 docutils = 0.7 drmaa = 0.6 Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: carlfeberhard: Saved visualizations: fix to display by slug to render built-in visualizations using original display code
by commits-noreply＠bitbucket.org 18 Mar '14

18 Mar '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/91aa1db362c9/ Changeset: 91aa1db362c9 User: carlfeberhard Date: 2014-03-18 14:48:47 Summary: Saved visualizations: fix to display by slug to render built-in visualizations using original display code Affected #: 1 file diff -r dea106a05af8597b66846bde6d28bfe97a8ea338 -r 91aa1db362c9f5c76160b09bab52b8a77b0f30b0 lib/galaxy/webapps/galaxy/controllers/visualization.py --- a/lib/galaxy/webapps/galaxy/controllers/visualization.py +++ b/lib/galaxy/webapps/galaxy/controllers/visualization.py @@ -584,7 +584,8 @@ ave_item_rating, num_ratings = self.get_ave_item_rating_data( trans.sa_session, visualization ) # Display. - if trans.app.visualizations_registry and visualization.type in trans.app.visualizations_registry.plugins: + if( ( trans.app.visualizations_registry and visualization.type in trans.app.visualizations_registry.plugins ) + and ( visualization.type not in trans.app.visualizations_registry.BUILT_IN_VISUALIZATIONS ) ): # if a registry visualization, load a version of display.mako that will load the vis into an iframe :( #TODO: simplest path from A to B but not optimal - will be difficult to do reg visualizations any other way #TODO: this will load the visualization twice (once above, once when the iframe src calls 'saved') Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by commits-noreply＠bitbucket.org 18 Mar '14

18 Mar '14

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/6b4139f8f40c/ Changeset: 6b4139f8f40c User: mike8115 Date: 2014-02-22 02:32:01 Summary: Trello Card #1437: Optional Input Datasets Not Compatible with Parallelism Tag multi.py edited to check for input.data is None. Affected #: 1 file diff -r 2403d6f67500ec35b16ce604142d98766286b65a -r 6b4139f8f40cda2bc21230fd4b32444474bfea8d lib/galaxy/jobs/splitters/multi.py --- a/lib/galaxy/jobs/splitters/multi.py +++ b/lib/galaxy/jobs/splitters/multi.py @@ -44,18 +44,26 @@ # For things like paired end alignment, we need two inputs to be split. Since all inputs to all # derived subtasks need to be correlated, allow only one input type to be split + # If shared_inputs are not specified, we assume all non-split inputs are shared inputs. + # For any input we must consider if each input is None. With optional arguments, a data input could be set to optional type_to_input_map = {} for input in parent_job.input_datasets: - if input.name in split_inputs: - type_to_input_map.setdefault(input.dataset.datatype, []).append(input.name) - elif input.name in shared_inputs: - pass # pass original file name - elif auto_shared_inputs: - shared_inputs.append(input.name) + if input.dataset is None: + if input.name in shared_inputs: + shared_inputs.remove(input.name) + else: + pass else: - log_error = "The input '%s' does not define a method for implementing parallelism" % str(input.name) - log.exception(log_error) - raise Exception(log_error) + if input.name in split_inputs: + type_to_input_map.setdefault(input.dataset.datatype, []).append(input.name) + elif input.name in shared_inputs: + pass # pass original file name + elif auto_shared_inputs: + shared_inputs.append(input.name) + else: + log_error = "The input '%s' does not define a method for implementing parallelism" % str(input.name) + log.exception(log_error) + raise Exception(log_error) if len(type_to_input_map) > 1: log_error = "The multi splitter does not support splitting inputs of more than one type" https://bitbucket.org/galaxy/galaxy-central/commits/dea106a05af8/ Changeset: dea106a05af8 User: dannon Date: 2014-03-18 13:57:36 Summary: Merged in mike8115/galaxy-central (pull request #334) Trello Card #1437: Optional Input Datasets Not Compatible with Parallelism Tag Affected #: 1 file diff -r c53fbb2655e2756c8977782df2381d883c5aa8af -r dea106a05af8597b66846bde6d28bfe97a8ea338 lib/galaxy/jobs/splitters/multi.py --- a/lib/galaxy/jobs/splitters/multi.py +++ b/lib/galaxy/jobs/splitters/multi.py @@ -44,18 +44,26 @@ # For things like paired end alignment, we need two inputs to be split. Since all inputs to all # derived subtasks need to be correlated, allow only one input type to be split + # If shared_inputs are not specified, we assume all non-split inputs are shared inputs. + # For any input we must consider if each input is None. With optional arguments, a data input could be set to optional type_to_input_map = {} for input in parent_job.input_datasets: - if input.name in split_inputs: - type_to_input_map.setdefault(input.dataset.datatype, []).append(input.name) - elif input.name in shared_inputs: - pass # pass original file name - elif auto_shared_inputs: - shared_inputs.append(input.name) + if input.dataset is None: + if input.name in shared_inputs: + shared_inputs.remove(input.name) + else: + pass else: - log_error = "The input '%s' does not define a method for implementing parallelism" % str(input.name) - log.exception(log_error) - raise Exception(log_error) + if input.name in split_inputs: + type_to_input_map.setdefault(input.dataset.datatype, []).append(input.name) + elif input.name in shared_inputs: + pass # pass original file name + elif auto_shared_inputs: + shared_inputs.append(input.name) + else: + log_error = "The input '%s' does not define a method for implementing parallelism" % str(input.name) + log.exception(log_error) + raise Exception(log_error) if len(type_to_input_map) > 1: log_error = "The multi splitter does not support splitting inputs of more than one type" Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: Add tool functional test for parallelism with optional inputs.
by commits-noreply＠bitbucket.org 17 Mar '14

17 Mar '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/c53fbb2655e2/ Changeset: c53fbb2655e2 User: jmchilton Date: 2014-03-17 02:41:42 Summary: Add tool functional test for parallelism with optional inputs. This functionality is broken without patch from pull request #334. To run this broken test: % # First two exports are optional but speed up testing and force newest test framework. % export GALAXY_TEST_DEFAULT_INTERACTOR=api; % export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-bootstrap/raw/m… % ./run_functional_tests.sh -framework -id parallelism_optional Affected #: 2 files diff -r 11e83ed03dab7f658e3b08e29be25768f3546c72 -r c53fbb2655e2756c8977782df2381d883c5aa8af test/functional/tools/parallelism_optional.xml --- /dev/null +++ b/test/functional/tools/parallelism_optional.xml @@ -0,0 +1,21 @@ +<tool id="parallelism_optional" name="Split file line-by-line and rebuild dataset (with optional dataset)"> + <parallelism method="multi" split_inputs="input1" split_mode="to_size" split_size="1" merge_outputs="out_file1" /> + <command> + cat $input1 > $out_file1 + </command> + <inputs> + <param name="input1" type="data" label="Dataset"/> + <param name="input2" type="data" label="Optional Dataset" optional="true" /> + </inputs> + <outputs> + <data name="out_file1" format="txt" /> + </outputs> + <tests> + <test> + <param name="input1" value="simple_line_x2.txt"/> + <output name="out_file1" file="simple_line_x2.txt" /> + </test> + </tests> + <help> + </help> +</tool> diff -r 11e83ed03dab7f658e3b08e29be25768f3546c72 -r c53fbb2655e2756c8977782df2381d883c5aa8af test/functional/tools/samples_tool_conf.xml --- a/test/functional/tools/samples_tool_conf.xml +++ b/test/functional/tools/samples_tool_conf.xml @@ -14,6 +14,7 @@ <tool file="disambiguate_repeats.xml" /><tool file="min_repeat.xml" /><tool file="parallelism.xml" /> + <tool file="parallelism_optional.xml" /><tool file="implicit_default_conds.xml" /><tool file="multi_data_param.xml" /></toolbox> \ No newline at end of file Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by commits-noreply＠bitbucket.org 17 Mar '14

17 Mar '14

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/6007992cbcd3/ Changeset: 6007992cbcd3 Branch: stable User: nsoranzo Date: 2014-03-14 15:21:25 Summary: Fix for https://trello.com/c/8NpgYM3U/1430-tool-panel-label-bug-introduced-with-201… Bug was introduced in commit 5f6c670dca9f42d52480f66afeb91b63710e7ffb . Affected #: 1 file diff -r ac4e18e81ed5d9256cb1f07692da7928dcd7969f -r 6007992cbcd3626cbed38b0e883f516b016f98be lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -276,7 +276,7 @@ section = ToolSection( elem ) log.debug( "Loading section: %s" % elem.get( 'name' ) ) for section_key, section_val in val.elems.items(): - if section_key.startswith( 'tool_' ): + if isinstance( section_val, Tool ): tool_id = section_key.replace( 'tool_', '', 1 ) if tool_id in self.tools_by_id: self.__add_tool_to_tool_panel( tool_id, section, section=True ) @@ -286,7 +286,7 @@ workflow = self.workflows_by_id[ workflow_id ] section.elems[ section_key ] = workflow log.debug( "Loaded workflow: %s %s" % ( workflow_id, workflow.name ) ) - elif isinstance( val, ToolSectionLabel ): + elif isinstance( section_val, ToolSectionLabel ): if section_val: section.elems[ section_key ] = section_val log.debug( "Loaded label: %s" % ( section_val.text ) ) https://bitbucket.org/galaxy/galaxy-central/commits/11e83ed03dab/ Changeset: 11e83ed03dab User: jmchilton Date: 2014-03-17 02:12:20 Summary: Merge latest stable. Affected #: 1 file Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by commits-noreply＠bitbucket.org 17 Mar '14

17 Mar '14

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/d3427fa33560/ Changeset: d3427fa33560 User: nsoranzo Date: 2014-03-14 15:21:25 Summary: Fix for https://trello.com/c/8NpgYM3U/1430-tool-panel-label-bug-introduced-with-201… Bug was introduced in commit 5f6c670dca9f42d52480f66afeb91b63710e7ffb . Affected #: 1 file diff -r 3fb927653301a0c06a0bf94f2b6bd71b3595ec0d -r d3427fa335602629611c7aedbc52196b3a0eea64 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -285,7 +285,7 @@ section = ToolSection( elem ) log.debug( "Loading section: %s" % elem.get( 'name' ) ) for section_key, section_val in val.elems.items(): - if section_key.startswith( 'tool_' ): + if isinstance( section_val, Tool ): tool_id = section_key.replace( 'tool_', '', 1 ) if tool_id in self.tools_by_id: self.__add_tool_to_tool_panel( tool_id, section, section=True ) @@ -295,7 +295,7 @@ workflow = self.workflows_by_id[ workflow_id ] section.elems[ section_key ] = workflow log.debug( "Loaded workflow: %s %s" % ( workflow_id, workflow.name ) ) - elif isinstance( val, ToolSectionLabel ): + elif isinstance( section_val, ToolSectionLabel ): if section_val: section.elems[ section_key ] = section_val log.debug( "Loaded label: %s" % ( section_val.text ) ) https://bitbucket.org/galaxy/galaxy-central/commits/b58efefe1060/ Changeset: b58efefe1060 User: jmchilton Date: 2014-03-17 02:10:01 Summary: Merged in nsoranzo/galaxy-central (pull request #346) Fix for https://trello.com/c/8NpgYM3U/1430-tool-panel-label-bug-introduced-with-201… Affected #: 1 file diff -r 35e90d31c3c78e4710a0f73a0126ddbd9ca8397a -r b58efefe106088d30e234373363e3cb0d6434954 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -285,7 +285,7 @@ section = ToolSection( elem ) log.debug( "Loading section: %s" % elem.get( 'name' ) ) for section_key, section_val in val.elems.items(): - if section_key.startswith( 'tool_' ): + if isinstance( section_val, Tool ): tool_id = section_key.replace( 'tool_', '', 1 ) if tool_id in self.tools_by_id: self.__add_tool_to_tool_panel( tool_id, section, section=True ) @@ -295,7 +295,7 @@ workflow = self.workflows_by_id[ workflow_id ] section.elems[ section_key ] = workflow log.debug( "Loaded workflow: %s %s" % ( workflow_id, workflow.name ) ) - elif isinstance( val, ToolSectionLabel ): + elif isinstance( section_val, ToolSectionLabel ): if section_val: section.elems[ section_key ] = section_val log.debug( "Loaded label: %s" % ( section_val.text ) ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: PEP-8 fixes, var name fixes, and unit tests for SecurityHelper.
by commits-noreply＠bitbucket.org 17 Mar '14

17 Mar '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/35e90d31c3c7/ Changeset: 35e90d31c3c7 User: jmchilton Date: 2014-03-17 01:55:47 Summary: PEP-8 fixes, var name fixes, and unit tests for SecurityHelper. Tests cover nearly all functionality - in particular newly added per-kind key generation. Affected #: 2 files diff -r 2727f16171610e0e27420d780e9a5ed56f1908f5 -r 35e90d31c3c78e4710a0f73a0126ddbd9ca8397a lib/galaxy/web/security/__init__.py --- a/lib/galaxy/web/security/__init__.py +++ b/lib/galaxy/web/security/__init__.py @@ -1,5 +1,7 @@ import collections -import os, os.path, logging +import os +import os.path +import logging import pkg_resources pkg_resources.require( "pycrypto" ) @@ -13,6 +15,7 @@ if os.path.exists( "/dev/urandom" ): # We have urandom, use it as the source of random data random_fd = os.open( "/dev/urandom", os.O_RDONLY ) + def get_random_bytes( nbytes ): value = os.read( random_fd, nbytes ) # Normally we should get as much as we need @@ -39,7 +42,7 @@ self.id_cipher = Blowfish.new( self.id_secret ) per_kind_id_secret_base = config.get( 'per_kind_id_secret_base', self.id_secret ) - self.id_ciphers_for_key = _cipher_cache( per_kind_id_secret_base ) + self.id_ciphers_for_kind = _cipher_cache( per_kind_id_secret_base ) def encode_id( self, obj_id, kind=None ): id_cipher = self.__id_cipher( kind ) @@ -84,7 +87,7 @@ if not kind: id_cipher = self.id_cipher else: - id_cipher = self.id_ciphers_for_key[ kind ] + id_cipher = self.id_ciphers_for_kind[ kind ] return id_cipher diff -r 2727f16171610e0e27420d780e9a5ed56f1908f5 -r 35e90d31c3c78e4710a0f73a0126ddbd9ca8397a test/unit/test_security_helper.py --- /dev/null +++ b/test/unit/test_security_helper.py @@ -0,0 +1,53 @@ +from galaxy.web import security + + +test_helper_1 = security.SecurityHelper( id_secret="sec1" ) +test_helper_2 = security.SecurityHelper( id_secret="sec2" ) + + +def test_encode_decode(): + # Different ids are encoded differently + assert test_helper_1.encode_id( 1 ) != test_helper_1.encode_id( 2 ) + # But decoding and encoded id brings back to original id + assert 1 == test_helper_1.decode_id( test_helper_1.encode_id( 1 ) ) + + +def test_per_kind_encode_deocde(): + # Different ids are encoded differently + assert test_helper_1.encode_id( 1, kind="k1" ) != test_helper_1.encode_id( 2, kind="k1" ) + # But decoding and encoded id brings back to original id + assert 1 == test_helper_1.decode_id( test_helper_1.encode_id( 1, kind="k1" ), kind="k1" ) + + +def test_different_secrets_encode_differently(): + assert test_helper_1.encode_id( 1 ) != test_helper_2.encode_id( 1 ) + + +def test_per_kind_encodes_id_differently(): + assert test_helper_1.encode_id( 1 ) != test_helper_2.encode_id( 1, kind="new_kind" ) + + +def test_encode_dict(): + test_dict = dict( + id=1, + other=2, + history_id=3, + ) + encoded_dict = test_helper_1.encode_dict_ids( test_dict ) + assert encoded_dict[ "id" ] == test_helper_1.encode_id( 1 ) + assert encoded_dict[ "other" ] == 2 + assert encoded_dict[ "history_id" ] == test_helper_1.encode_id( 3 ) + + +def test_guid_generation(): + guids = set() + for i in range( 100 ): + guids.add( test_helper_1.get_new_guid() ) + assert len( guids ) == 100 # Not duplicate guids generated. + + +def test_encode_decode_guid(): + session_key = test_helper_1.get_new_guid() + encoded_key = test_helper_1.encode_guid( session_key ) + decoded_key = test_helper_1.decode_guid( encoded_key ).encode( "utf-8" ) + assert session_key == decoded_key, "%s != %s" % ( session_key, decoded_key ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 3 new changesets
by commits-noreply＠bitbucket.org 17 Mar '14

17 Mar '14

3 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/d847b6985330/ Changeset: d847b6985330 User: jmchilton Date: 2014-02-20 16:37:17 Summary: Extend SecurityHelper to allow generation of different "kind"s of keys. There has been talk about including the object's class into the hash when generating keys so the ids are not overlapping between model types. This change could allow that, but my immediate desire is to just to create single-purpose per job keys to allow external job running code to access files on behalf of a given job. Affected #: 1 file diff -r 9c673e0ea8919fea6941f24fdf6936686840bb71 -r d847b69853302d9a127c31c612dd19cc7ffa9f61 lib/galaxy/web/security/__init__.py --- a/lib/galaxy/web/security/__init__.py +++ b/lib/galaxy/web/security/__init__.py @@ -1,3 +1,4 @@ +import collections import os, os.path, logging import pkg_resources @@ -37,27 +38,32 @@ self.id_secret = config['id_secret'] self.id_cipher = Blowfish.new( self.id_secret ) - def encode_id( self, obj_id ): + per_kind_id_secret_base = config.get( 'per_kind_id_secret_base', self.id_secret ) + self.id_ciphers_for_key = _cipher_cache( per_kind_id_secret_base ) + + def encode_id( self, obj_id, kind=None ): + id_cipher = self.__id_cipher( kind ) # Convert to string s = str( obj_id ) # Pad to a multiple of 8 with leading "!" s = ( "!" * ( 8 - len(s) % 8 ) ) + s # Encrypt - return self.id_cipher.encrypt( s ).encode( 'hex' ) + return id_cipher.encrypt( s ).encode( 'hex' ) - def encode_dict_ids( self, a_dict ): + def encode_dict_ids( self, a_dict, kind=None ): """ Encode all ids in dictionary. Ids are identified by (a) an 'id' key or (b) a key that ends with '_id' """ for key, val in a_dict.items(): if key == 'id' or key.endswith('_id'): - a_dict[ key ] = self.encode_id( val ) + a_dict[ key ] = self.encode_id( val, kind=kind ) return a_dict - def decode_id( self, obj_id ): - return int( self.id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( "!" ) ) + def decode_id( self, obj_id, kind=None ): + id_cipher = self.__id_cipher( kind ) + return int( id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( "!" ) ) def encode_guid( self, session_key ): # Session keys are strings @@ -73,3 +79,19 @@ def get_new_guid( self ): # Generate a unique, high entropy 128 bit random number return get_random_bytes( 16 ) + + def __id_cipher( self, kind ): + if not kind: + id_cipher = self.id_cipher + else: + id_cipher = self.id_ciphers_for_key[ kind ] + return id_cipher + + +class _cipher_cache( collections.defaultdict ): + + def __init__( self, secret_base ): + self.secret_base = secret_base + + def __missing__( self, key ): + return Blowfish.new( self.secret_base + "__" + key ) https://bitbucket.org/galaxy/galaxy-central/commits/58acdb670e4b/ Changeset: 58acdb670e4b User: jmchilton Date: 2014-02-20 16:37:17 Summary: Implement API allowing access to a job's files. Provides a mechanism for remote job execution mechanisms to read and write files on behalf of an "active" jobs. For a variety of reasons simply providing access to datasets is insuccifient for this - there are working directory files needed for metadata calculation, inputs and outputs may be of intermediate form (task split files for instance), tool files, location files, etc.... Therefore this API endpoint provides access to a jobs view of these files. Some attempt is made to verify the paths that are written to are valid for the supplied job (they either correspond to the output datasets of the job or the working directory of the job). Authorizing suchs paths for reading is much more difficult and left undone due to the unstructure and arbitrary nature of .loc files. To implement this securely and with minimal configuration required - single-purpose "job_key"s are generated to authenticate these API calls. Each key allow usage of the API only for a single, active job. Update LWR job runner to be able to leverage this API to stage files remotely via HTTP for "remote_transfer" actions (added recently to the LWR client). Affected #: 6 files diff -r d847b69853302d9a127c31c612dd19cc7ffa9f61 -r 58acdb670e4b8ea9a35d3b19aaab1e66c5b75009 lib/galaxy/jobs/runners/lwr.py --- a/lib/galaxy/jobs/runners/lwr.py +++ b/lib/galaxy/jobs/runners/lwr.py @@ -28,6 +28,10 @@ NO_REMOTE_GALAXY_FOR_METADATA_MESSAGE = "LWR misconfiguration - LWR client configured to set metadata remotely, but remote LWR isn't properly configured with a galaxy_home directory." NO_REMOTE_DATATYPES_CONFIG = "LWR client is configured to use remote datatypes configuration when setting metadata externally, but LWR is not configured with this information. Defaulting to datatypes_conf.xml." +# Is there a good way to infer some default for this? Can only use +# url_for from web threads. https://gist.github.com/jmchilton/9098762 +DEFAULT_GALAXY_URL = "http://localhost:8080" + class LwrJobRunner( AsynchronousJobRunner ): """ @@ -35,13 +39,14 @@ """ runner_name = "LWRRunner" - def __init__( self, app, nworkers, transport=None, cache=None, url=None ): + def __init__( self, app, nworkers, transport=None, cache=None, url=None, galaxy_url=DEFAULT_GALAXY_URL ): """Start the job runner """ super( LwrJobRunner, self ).__init__( app, nworkers ) self.async_status_updates = dict() self._init_monitor_thread() self._init_worker_threads() client_manager_kwargs = {'transport_type': transport, 'cache': string_as_bool_or_none(cache), "url": url} + self.galaxy_url = galaxy_url self.client_manager = build_client_manager(**client_manager_kwargs) def url_to_destination( self, url ): @@ -224,7 +229,21 @@ return self.get_client( job_destination_params, job_id ) def get_client( self, job_destination_params, job_id ): - return self.client_manager.get_client( job_destination_params, str( job_id ) ) + # Cannot use url_for outside of web thread. + #files_endpoint = url_for( controller="job_files", job_id=encoded_job_id ) + + encoded_job_id = self.app.security.encode_id(job_id) + job_key = self.app.security.encode_id( job_id, kind="jobs_files" ) + files_endpoint = "%s/api/jobs/%s/files?job_key=%s" % ( + self.galaxy_url, + encoded_job_id, + job_key + ) + get_client_kwds = dict( + job_id=str( job_id ), + files_endpoint=files_endpoint, + ) + return self.client_manager.get_client( job_destination_params, **get_client_kwds ) def finish_job( self, job_state ): stderr = stdout = '' diff -r d847b69853302d9a127c31c612dd19cc7ffa9f61 -r 58acdb670e4b8ea9a35d3b19aaab1e66c5b75009 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py +++ b/lib/galaxy/model/__init__.py @@ -266,6 +266,16 @@ self.handler = None self.exit_code = None + @property + def finished( self ): + states = self.states + return self.state in [ + states.OK, + states.ERROR, + states.DELETED, + states.DELETED_NEW, + ] + # TODO: Add accessors for members defined in SQL Alchemy for the Job table and # for the mapper defined to the Job table. def get_external_output_metadata( self ): diff -r d847b69853302d9a127c31c612dd19cc7ffa9f61 -r 58acdb670e4b8ea9a35d3b19aaab1e66c5b75009 lib/galaxy/web/__init__.py --- a/lib/galaxy/web/__init__.py +++ b/lib/galaxy/web/__init__.py @@ -20,3 +20,4 @@ from framework import _future_expose_api from framework import _future_expose_api_anonymous from framework import _future_expose_api_raw +from framework import _future_expose_api_raw_anonymous diff -r d847b69853302d9a127c31c612dd19cc7ffa9f61 -r 58acdb670e4b8ea9a35d3b19aaab1e66c5b75009 lib/galaxy/web/framework/__init__.py --- a/lib/galaxy/web/framework/__init__.py +++ b/lib/galaxy/web/framework/__init__.py @@ -278,6 +278,10 @@ return _future_expose_api( func, to_json=False, user_required=True ) +def _future_expose_api_raw_anonymous( func ): + return _future_expose_api( func, to_json=False, user_required=False ) + + # TODO: rename as expose_api and make default. def _future_expose_api( func, to_json=True, user_required=True ): """ diff -r d847b69853302d9a127c31c612dd19cc7ffa9f61 -r 58acdb670e4b8ea9a35d3b19aaab1e66c5b75009 lib/galaxy/webapps/galaxy/api/job_files.py --- /dev/null +++ b/lib/galaxy/webapps/galaxy/api/job_files.py @@ -0,0 +1,142 @@ +""" API for asynchronous job running mechanisms can use to fetch or put files +related to running and queued jobs. +""" +import os +import shutil + +from galaxy import exceptions +from galaxy import util +from galaxy import model +from galaxy.web.base.controller import BaseAPIController +from galaxy.web import _future_expose_api_anonymous as expose_api_anonymous +from galaxy.web import _future_expose_api_raw_anonymous as expose_api_raw_anonymous + + +import logging +log = logging.getLogger( __name__ ) + + +class JobFilesAPIController( BaseAPIController ): + """ This job files controller allows remote job running mechanisms to + read and modify the current state of files for queued and running jobs. + It is certainly not meant to represent part of Galaxy's stable, user + facing API. + + Furthermore, even if a user key corresponds to the user running the job, + it should not be accepted for authorization - this API allows access to + low-level unfiltered files and such authorization would break Galaxy's + security model for tool execution. + """ + + @expose_api_raw_anonymous + def index( self, trans, job_id, **kwargs ): + """ + index( self, trans, job_id, **kwargs ) + * GET /api/jobs/{job_id}/files + Get a file required to staging a job (proper datasets, extra inputs, + task-split inputs, working directory files). + + :type job_id: str + :param job_id: encoded id string of the job + :type path: str + :param path: Path to file. + :type job_key: str + :param job_key: A key used to authenticate this request as acting on + behalf or a job runner for the specified job. + ..note: + This API method is intended only for consumption by job runners, + not end users. + + :rtype: binary + :returns: contents of file + """ + self.__authorize_job_access( trans, job_id, **kwargs ) + path = kwargs.get("path", None) + return open(path, 'rb') + + @expose_api_anonymous + def create( self, trans, job_id, payload, **kwargs ): + """ + create( self, trans, job_id, payload, **kwargs ) + * POST /api/jobs/{job_id}/files + Populate an output file (formal dataset, task split part, working + directory file (such as those related to metadata)). This should be + a multipart post with a 'file' parameter containing the contents of + the actual file to create. + + :type job_id: str + :param job_id: encoded id string of the job + :type payload: dict + :param payload: dictionary structure containing:: + 'job_key' = Key authenticating + 'path' = Path to file to create. + + ..note: + This API method is intended only for consumption by job runners, + not end users. + + :rtype: dict + :returns: an okay message + """ + job = self.__authorize_job_access( trans, job_id, **payload ) + path = payload.get( "path" ) + self.__check_job_can_write_to_path( trans, job, path ) + + # Is this writing an unneeded file? Should this just copy in Python? + input_file = payload.get( "file", payload.get( "__file", None ) ).file + try: + shutil.copyfile( input_file.name, path ) + finally: + input_file.close() + return {"message": "ok"} + + def __authorize_job_access(self, trans, encoded_job_id, **kwargs): + for key in [ "path", "job_key" ]: + if key not in kwargs: + error_message = "Job files action requires a valid '%s'." % key + raise exceptions.ObjectAttributeMissingException( error_message ) + + job_id = trans.security.decode_id( encoded_job_id ) + job_key = trans.security.encode_id( job_id, kind="jobs_files" ) + if not util.safe_str_cmp( kwargs[ "job_key" ], job_key ): + raise exceptions.ItemAccessibilityException("Invalid job_key supplied.") + + # Verify job is active. Don't update the contents of complete jobs. + job = trans.sa_session.query( model.Job ).get( job_id ) + if job.finished: + error_message = "Attempting to read or modify the files of a job that has already completed." + raise exceptions.MessageException( error_message ) + return job + + def __check_job_can_write_to_path( self, trans, job, path ): + """ Verify an idealized job runner should actually be able to write to + the specified path - it must be a dataset output, a dataset "extra + file", or a some place in the working directory of this job. + + Would like similar checks for reading the unstructured nature of loc + files make this very difficult. (See abandoned work here + https://gist.github.com/jmchilton/9103619.) + """ + in_work_dir = self.__in_working_directory( job, path, trans.app ) + if not in_work_dir and not self.__is_output_dataset_path( job, path ): + raise exceptions.ItemAccessibilityException("Job is not authorized to write to supplied path.") + + def __is_output_dataset_path( self, job, path ): + """ Check if is an output path for this job or a file in the an + output's extra files path. + """ + da_lists = [ job.output_datasets, job.output_library_datasets ] + for da_list in da_lists: + for job_dataset_association in da_list: + dataset = job_dataset_association.dataset + if not dataset: + continue + if os.path.abspath( dataset.file_name ) == os.path.abspath( path ): + return True + elif util.in_directory( path, dataset.extra_files_path ): + return True + return False + + def __in_working_directory( self, job, path, app ): + working_directory = app.object_store.get_filename(job, base_dir='job_work', dir_only=True, extra_dir=str(job.id)) + return util.in_directory( path, working_directory ) diff -r d847b69853302d9a127c31c612dd19cc7ffa9f61 -r 58acdb670e4b8ea9a35d3b19aaab1e66c5b75009 lib/galaxy/webapps/galaxy/buildapp.py --- a/lib/galaxy/webapps/galaxy/buildapp.py +++ b/lib/galaxy/webapps/galaxy/buildapp.py @@ -233,12 +233,20 @@ 'permissions', path_prefix='/api/libraries/:library_id', parent_resources=dict( member_name='library', collection_name='libraries' ) ) - - webapp.mapper.resource( 'job', - 'jobs', + + webapp.mapper.resource( 'job', + 'jobs', path_prefix='/api' ) webapp.mapper.connect( 'job_search', '/api/jobs/search', controller='jobs', action='search', conditions=dict( method=['POST'] ) ) + # Job files controllers. Only for consumption by remote job runners. + webapp.mapper.resource( 'file', + 'files', + controller="job_files", + name_prefix="job_", + path_prefix='/api/jobs/:job_id', + parent_resources=dict( member_name="job", collection_name="jobs") + ) _add_item_extended_metadata_controller( webapp, name_prefix="library_dataset_", https://bitbucket.org/galaxy/galaxy-central/commits/2727f1617161/ Changeset: 2727f1617161 User: jmchilton Date: 2014-03-17 01:54:02 Summary: Merged in jmchilton/galaxy-central-fork-1 (pull request #327) Implement job files API. Affected #: 7 files diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/jobs/runners/lwr.py --- a/lib/galaxy/jobs/runners/lwr.py +++ b/lib/galaxy/jobs/runners/lwr.py @@ -28,6 +28,10 @@ NO_REMOTE_GALAXY_FOR_METADATA_MESSAGE = "LWR misconfiguration - LWR client configured to set metadata remotely, but remote LWR isn't properly configured with a galaxy_home directory." NO_REMOTE_DATATYPES_CONFIG = "LWR client is configured to use remote datatypes configuration when setting metadata externally, but LWR is not configured with this information. Defaulting to datatypes_conf.xml." +# Is there a good way to infer some default for this? Can only use +# url_for from web threads. https://gist.github.com/jmchilton/9098762 +DEFAULT_GALAXY_URL = "http://localhost:8080" + class LwrJobRunner( AsynchronousJobRunner ): """ @@ -35,13 +39,14 @@ """ runner_name = "LWRRunner" - def __init__( self, app, nworkers, transport=None, cache=None, url=None ): + def __init__( self, app, nworkers, transport=None, cache=None, url=None, galaxy_url=DEFAULT_GALAXY_URL ): """Start the job runner """ super( LwrJobRunner, self ).__init__( app, nworkers ) self.async_status_updates = dict() self._init_monitor_thread() self._init_worker_threads() client_manager_kwargs = {'transport_type': transport, 'cache': string_as_bool_or_none(cache), "url": url} + self.galaxy_url = galaxy_url self.client_manager = build_client_manager(**client_manager_kwargs) def url_to_destination( self, url ): @@ -224,7 +229,21 @@ return self.get_client( job_destination_params, job_id ) def get_client( self, job_destination_params, job_id ): - return self.client_manager.get_client( job_destination_params, str( job_id ) ) + # Cannot use url_for outside of web thread. + #files_endpoint = url_for( controller="job_files", job_id=encoded_job_id ) + + encoded_job_id = self.app.security.encode_id(job_id) + job_key = self.app.security.encode_id( job_id, kind="jobs_files" ) + files_endpoint = "%s/api/jobs/%s/files?job_key=%s" % ( + self.galaxy_url, + encoded_job_id, + job_key + ) + get_client_kwds = dict( + job_id=str( job_id ), + files_endpoint=files_endpoint, + ) + return self.client_manager.get_client( job_destination_params, **get_client_kwds ) def finish_job( self, job_state ): stderr = stdout = '' diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py +++ b/lib/galaxy/model/__init__.py @@ -268,6 +268,16 @@ self.handler = None self.exit_code = None + @property + def finished( self ): + states = self.states + return self.state in [ + states.OK, + states.ERROR, + states.DELETED, + states.DELETED_NEW, + ] + # TODO: Add accessors for members defined in SQL Alchemy for the Job table and # for the mapper defined to the Job table. def get_external_output_metadata( self ): diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/web/__init__.py --- a/lib/galaxy/web/__init__.py +++ b/lib/galaxy/web/__init__.py @@ -20,3 +20,4 @@ from framework import _future_expose_api from framework import _future_expose_api_anonymous from framework import _future_expose_api_raw +from framework import _future_expose_api_raw_anonymous diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/web/framework/__init__.py --- a/lib/galaxy/web/framework/__init__.py +++ b/lib/galaxy/web/framework/__init__.py @@ -278,6 +278,10 @@ return _future_expose_api( func, to_json=False, user_required=True ) +def _future_expose_api_raw_anonymous( func ): + return _future_expose_api( func, to_json=False, user_required=False ) + + # TODO: rename as expose_api and make default. def _future_expose_api( func, to_json=True, user_required=True ): """ diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/web/security/__init__.py --- a/lib/galaxy/web/security/__init__.py +++ b/lib/galaxy/web/security/__init__.py @@ -1,3 +1,4 @@ +import collections import os, os.path, logging import pkg_resources @@ -37,27 +38,32 @@ self.id_secret = config['id_secret'] self.id_cipher = Blowfish.new( self.id_secret ) - def encode_id( self, obj_id ): + per_kind_id_secret_base = config.get( 'per_kind_id_secret_base', self.id_secret ) + self.id_ciphers_for_key = _cipher_cache( per_kind_id_secret_base ) + + def encode_id( self, obj_id, kind=None ): + id_cipher = self.__id_cipher( kind ) # Convert to string s = str( obj_id ) # Pad to a multiple of 8 with leading "!" s = ( "!" * ( 8 - len(s) % 8 ) ) + s # Encrypt - return self.id_cipher.encrypt( s ).encode( 'hex' ) + return id_cipher.encrypt( s ).encode( 'hex' ) - def encode_dict_ids( self, a_dict ): + def encode_dict_ids( self, a_dict, kind=None ): """ Encode all ids in dictionary. Ids are identified by (a) an 'id' key or (b) a key that ends with '_id' """ for key, val in a_dict.items(): if key == 'id' or key.endswith('_id'): - a_dict[ key ] = self.encode_id( val ) + a_dict[ key ] = self.encode_id( val, kind=kind ) return a_dict - def decode_id( self, obj_id ): - return int( self.id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( "!" ) ) + def decode_id( self, obj_id, kind=None ): + id_cipher = self.__id_cipher( kind ) + return int( id_cipher.decrypt( obj_id.decode( 'hex' ) ).lstrip( "!" ) ) def encode_guid( self, session_key ): # Session keys are strings @@ -73,3 +79,19 @@ def get_new_guid( self ): # Generate a unique, high entropy 128 bit random number return get_random_bytes( 16 ) + + def __id_cipher( self, kind ): + if not kind: + id_cipher = self.id_cipher + else: + id_cipher = self.id_ciphers_for_key[ kind ] + return id_cipher + + +class _cipher_cache( collections.defaultdict ): + + def __init__( self, secret_base ): + self.secret_base = secret_base + + def __missing__( self, key ): + return Blowfish.new( self.secret_base + "__" + key ) diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/webapps/galaxy/api/job_files.py --- /dev/null +++ b/lib/galaxy/webapps/galaxy/api/job_files.py @@ -0,0 +1,142 @@ +""" API for asynchronous job running mechanisms can use to fetch or put files +related to running and queued jobs. +""" +import os +import shutil + +from galaxy import exceptions +from galaxy import util +from galaxy import model +from galaxy.web.base.controller import BaseAPIController +from galaxy.web import _future_expose_api_anonymous as expose_api_anonymous +from galaxy.web import _future_expose_api_raw_anonymous as expose_api_raw_anonymous + + +import logging +log = logging.getLogger( __name__ ) + + +class JobFilesAPIController( BaseAPIController ): + """ This job files controller allows remote job running mechanisms to + read and modify the current state of files for queued and running jobs. + It is certainly not meant to represent part of Galaxy's stable, user + facing API. + + Furthermore, even if a user key corresponds to the user running the job, + it should not be accepted for authorization - this API allows access to + low-level unfiltered files and such authorization would break Galaxy's + security model for tool execution. + """ + + @expose_api_raw_anonymous + def index( self, trans, job_id, **kwargs ): + """ + index( self, trans, job_id, **kwargs ) + * GET /api/jobs/{job_id}/files + Get a file required to staging a job (proper datasets, extra inputs, + task-split inputs, working directory files). + + :type job_id: str + :param job_id: encoded id string of the job + :type path: str + :param path: Path to file. + :type job_key: str + :param job_key: A key used to authenticate this request as acting on + behalf or a job runner for the specified job. + ..note: + This API method is intended only for consumption by job runners, + not end users. + + :rtype: binary + :returns: contents of file + """ + self.__authorize_job_access( trans, job_id, **kwargs ) + path = kwargs.get("path", None) + return open(path, 'rb') + + @expose_api_anonymous + def create( self, trans, job_id, payload, **kwargs ): + """ + create( self, trans, job_id, payload, **kwargs ) + * POST /api/jobs/{job_id}/files + Populate an output file (formal dataset, task split part, working + directory file (such as those related to metadata)). This should be + a multipart post with a 'file' parameter containing the contents of + the actual file to create. + + :type job_id: str + :param job_id: encoded id string of the job + :type payload: dict + :param payload: dictionary structure containing:: + 'job_key' = Key authenticating + 'path' = Path to file to create. + + ..note: + This API method is intended only for consumption by job runners, + not end users. + + :rtype: dict + :returns: an okay message + """ + job = self.__authorize_job_access( trans, job_id, **payload ) + path = payload.get( "path" ) + self.__check_job_can_write_to_path( trans, job, path ) + + # Is this writing an unneeded file? Should this just copy in Python? + input_file = payload.get( "file", payload.get( "__file", None ) ).file + try: + shutil.copyfile( input_file.name, path ) + finally: + input_file.close() + return {"message": "ok"} + + def __authorize_job_access(self, trans, encoded_job_id, **kwargs): + for key in [ "path", "job_key" ]: + if key not in kwargs: + error_message = "Job files action requires a valid '%s'." % key + raise exceptions.ObjectAttributeMissingException( error_message ) + + job_id = trans.security.decode_id( encoded_job_id ) + job_key = trans.security.encode_id( job_id, kind="jobs_files" ) + if not util.safe_str_cmp( kwargs[ "job_key" ], job_key ): + raise exceptions.ItemAccessibilityException("Invalid job_key supplied.") + + # Verify job is active. Don't update the contents of complete jobs. + job = trans.sa_session.query( model.Job ).get( job_id ) + if job.finished: + error_message = "Attempting to read or modify the files of a job that has already completed." + raise exceptions.MessageException( error_message ) + return job + + def __check_job_can_write_to_path( self, trans, job, path ): + """ Verify an idealized job runner should actually be able to write to + the specified path - it must be a dataset output, a dataset "extra + file", or a some place in the working directory of this job. + + Would like similar checks for reading the unstructured nature of loc + files make this very difficult. (See abandoned work here + https://gist.github.com/jmchilton/9103619.) + """ + in_work_dir = self.__in_working_directory( job, path, trans.app ) + if not in_work_dir and not self.__is_output_dataset_path( job, path ): + raise exceptions.ItemAccessibilityException("Job is not authorized to write to supplied path.") + + def __is_output_dataset_path( self, job, path ): + """ Check if is an output path for this job or a file in the an + output's extra files path. + """ + da_lists = [ job.output_datasets, job.output_library_datasets ] + for da_list in da_lists: + for job_dataset_association in da_list: + dataset = job_dataset_association.dataset + if not dataset: + continue + if os.path.abspath( dataset.file_name ) == os.path.abspath( path ): + return True + elif util.in_directory( path, dataset.extra_files_path ): + return True + return False + + def __in_working_directory( self, job, path, app ): + working_directory = app.object_store.get_filename(job, base_dir='job_work', dir_only=True, extra_dir=str(job.id)) + return util.in_directory( path, working_directory ) diff -r 6495ceccc87d467f73d76e21f801acc78934fc8f -r 2727f16171610e0e27420d780e9a5ed56f1908f5 lib/galaxy/webapps/galaxy/buildapp.py --- a/lib/galaxy/webapps/galaxy/buildapp.py +++ b/lib/galaxy/webapps/galaxy/buildapp.py @@ -261,6 +261,14 @@ path_prefix='/api' ) webapp.mapper.connect( 'job_search', '/api/jobs/search', controller='jobs', action='search', conditions=dict( method=['POST'] ) ) + # Job files controllers. Only for consumption by remote job runners. + webapp.mapper.resource( 'file', + 'files', + controller="job_files", + name_prefix="job_", + path_prefix='/api/jobs/:job_id', + parent_resources=dict( member_name="job", collection_name="jobs") + ) _add_item_extended_metadata_controller( webapp, name_prefix="library_dataset_", Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jeremy goecks: Trackster: fix bug in edff37d that prevented naming and saving new track visualization.
by commits-noreply＠bitbucket.org 16 Mar '14

16 Mar '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/6495ceccc87d/ Changeset: 6495ceccc87d User: jeremy goecks Date: 2014-03-16 05:06:07 Summary: Trackster: fix bug in edff37d that prevented naming and saving new track visualization. Affected #: 1 file diff -r 6f0f77a940fc4966ce5534a06866ae57899dd867 -r 6495ceccc87d467f73d76e21f801acc78934fc8f static/scripts/viz/trackster/tracks.js --- a/static/scripts/viz/trackster/tracks.js +++ b/static/scripts/viz/trackster/tracks.js @@ -894,12 +894,13 @@ // Define track configuration this.config = config_mod.ConfigSettingCollection.from_models_and_saved_values( [ + { key: 'name', label: 'Name', type: 'text', default_value: '' }, { key: 'a_color', label: 'A Color', type: 'color', default_value: "#FF0000" }, { key: 'c_color', label: 'C Color', type: 'color', default_value: "#00FF00" }, { key: 'g_color', label: 'G Color', type: 'color', default_value: "#0000FF" }, { key: 't_color', label: 'T Color', type: 'color', default_value: "#FF00FF" }, { key: 'n_color', label: 'N Color', type: 'color', default_value: "#AAAAAA" } - ], obj_dict.prefs); + ], { name: obj_dict.name }); }, render: function() { Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0