- galaxy-commits - lists.galaxyproject.org

commit/galaxy-central: dannon: Branch prune.
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/6d5e465c5b8b/ Changeset: 6d5e465c5b8b Branch: BjoernGruening/add-install_dir-to-perl5lib-and-path-bef-1410949639333 User: dannon Date: 2014-09-22 15:15:58+00:00 Summary: Branch prune. Affected #: 0 files Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: dannon: Merged in BjoernGruening/galaxy-central-5/BjoernGruening/add-install_dir-to-perl5lib-and-path-bef-1410949639333 (pull request #499)
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/e3b16462112b/ Changeset: e3b16462112b User: dannon Date: 2014-09-22 15:09:07+00:00 Summary: Merged in BjoernGruening/galaxy-central-5/BjoernGruening/add-install_dir-to-perl5lib-and-path-bef-1410949639333 (pull request #499) Add $INSTALL_DIR to PERL5LIB and PATH before starting to install libraries. A few packages need this in order to recognise previous installed packages. Affected #: 1 file diff -r df92b55d194a4d67a1b863c85044eb91c138c74b -r e3b16462112b2ab8150564e183213048e7c027da lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py @@ -1105,6 +1105,8 @@ # If set to a true value then MakeMaker's prompt function will always # return the default without waiting for user input. cmd = '''PERL_MM_USE_DEFAULT=1; export PERL_MM_USE_DEFAULT; ''' + cmd += 'export PERL5LIB=$INSTALL_DIR/lib/perl5:$PERL5LIB;' + cmd += 'export PATH=$INSTALL_DIR/bin:$PATH;' if perl_package.find( '://' ) != -1: # We assume a URL to a gem file. url = perl_package Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/4969f6502d42/ Changeset: 4969f6502d42 Branch: BjoernGruening/add-install_dir-to-perl5lib-and-path-bef-1410949639333 User: BjoernGruening Date: 2014-09-17 10:27:56+00:00 Summary: Add $INSTALL_DIR to PERL5LIB and PATH before starting to install libraries. A few packages need this in order to recognise previous installed packages. Affected #: 1 file diff -r 02738142b8cf5e0372ac5c2b46eeff5f0ab8e87a -r 4969f6502d423feb580946d6b4812a477cff77b4 lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py @@ -1105,6 +1105,8 @@ # If set to a true value then MakeMaker's prompt function will always # return the default without waiting for user input. cmd = '''PERL_MM_USE_DEFAULT=1; export PERL_MM_USE_DEFAULT; ''' + cmd += 'export PERL5LIB=$INSTALL_DIR/lib/perl5:$PERL5LIB;' + cmd += 'export PATH=$INSTALL_DIR/bin:$PATH;' if perl_package.find( '://' ) != -1: # We assume a URL to a gem file. url = perl_package https://bitbucket.org/galaxy/galaxy-central/commits/e3b16462112b/ Changeset: e3b16462112b User: dannon Date: 2014-09-22 15:09:07+00:00 Summary: Merged in BjoernGruening/galaxy-central-5/BjoernGruening/add-install_dir-to-perl5lib-and-path-bef-1410949639333 (pull request #499) Add $INSTALL_DIR to PERL5LIB and PATH before starting to install libraries. A few packages need this in order to recognise previous installed packages. Affected #: 1 file diff -r df92b55d194a4d67a1b863c85044eb91c138c74b -r e3b16462112b2ab8150564e183213048e7c027da lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py +++ b/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py @@ -1105,6 +1105,8 @@ # If set to a true value then MakeMaker's prompt function will always # return the default without waiting for user input. cmd = '''PERL_MM_USE_DEFAULT=1; export PERL_MM_USE_DEFAULT; ''' + cmd += 'export PERL5LIB=$INSTALL_DIR/lib/perl5:$PERL5LIB;' + cmd += 'export PATH=$INSTALL_DIR/bin:$PATH;' if perl_package.find( '://' ) != -1: # We assume a URL to a gem file. url = perl_package Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: Merged in jmchilton/galaxy-central-fork-1 (pull request #496)
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/df92b55d194a/ Changeset: df92b55d194a User: jmchilton Date: 2014-09-22 14:24:39+00:00 Summary: Merged in jmchilton/galaxy-central-fork-1 (pull request #496) More consistent tool API for map/reduce operations. Affected #: 4 files diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py +++ b/lib/galaxy/tools/parameters/basic.py @@ -1887,6 +1887,11 @@ elif isinstance( value, dict ) and 'src' in value and 'id' in value: if value['src'] == 'hda': rval = trans.sa_session.query( trans.app.model.HistoryDatasetAssociation ).get( trans.app.security.decode_id(value['id']) ) + elif value['src'] == 'hdca': + decoded_id = trans.app.security.decode_id( value[ 'id' ] ) + rval = trans.sa_session.query( trans.app.model.HistoryDatasetCollectionAssociation ).get( decoded_id ) + else: + raise ValueError("Unknown input source %s passed to job submission API." % value['src']) elif str( value ).startswith( "__collection_reduce__|" ): encoded_id = str( value )[ len( "__collection_reduce__|" ): ] decoded_id = trans.app.security.decode_id( encoded_id ) @@ -1903,6 +1908,10 @@ raise ValueError( "The previously selected dataset has been previously deleted" ) if hasattr( v, "dataset" ) and v.dataset.state in [ galaxy.model.Dataset.states.ERROR, galaxy.model.Dataset.states.DISCARDED ]: raise ValueError( "The previously selected dataset has entered an unusable state" ) + if not self.multiple: + if len( values ) > 1: + raise ValueError( "More than one dataset supplied to single input dataset parameter.") + rval = values[ 0 ] return rval def to_string( self, value, app ): diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b lib/galaxy/tools/parameters/meta.py --- a/lib/galaxy/tools/parameters/meta.py +++ b/lib/galaxy/tools/parameters/meta.py @@ -14,7 +14,33 @@ execution). """ + def classify_unmodified_parameter( input_key ): + value = incoming[ input_key ] + if isinstance( value, dict ) and 'values' in value: + # Explicit meta wrapper for inputs... + is_batch = value.get( 'batch', False ) + is_linked = value.get( 'linked', True ) + if is_batch and is_linked: + classification = permutations.input_classification.MATCHED + elif is_batch: + classification = permutations.input_classification.MULTIPLIED + else: + classification = permutations.input_classification.SINGLE + if __collection_multirun_parameter( value ): + collection_value = value[ 'values' ][ 0 ] + values = __expand_collection_parameter( trans, input_key, collection_value, collections_to_match ) + else: + values = value[ 'values' ] + else: + classification = permutations.input_classification.SINGLE + values = value + return classification, values + + from galaxy.dataset_collections import matching + collections_to_match = matching.CollectionsToMatch() + def classifier( input_key ): + collection_multirun_key = "%s|__collection_multirun__" % input_key multirun_key = "%s|__multirun__" % input_key if multirun_key in incoming: multi_value = util.listify( incoming[ multirun_key ] ) @@ -24,41 +50,12 @@ if len( multi_value ) == 0: multi_value = None return permutations.input_classification.SINGLE, multi_value[ 0 ] + elif collection_multirun_key in incoming: + incoming_val = incoming[ collection_multirun_key ] + values = __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ) + return permutations.input_classification.MATCHED, values else: - return permutations.input_classification.SINGLE, incoming[ input_key ] - - from galaxy.dataset_collections import matching - collections_to_match = matching.CollectionsToMatch() - - def collection_classifier( input_key ): - multirun_key = "%s|__collection_multirun__" % input_key - if multirun_key in incoming: - incoming_val = incoming[ multirun_key ] - # If subcollectin multirun of data_collection param - value will - # be "hdca_id|subcollection_type" else it will just be hdca_id - if "|" in incoming_val: - encoded_hdc_id, subcollection_type = incoming_val.split( "|", 1 ) - else: - try: - src = incoming_val[ "src" ] - if src != "hdca": - raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) - encoded_hdc_id = incoming_val[ "id" ] - except TypeError: - encoded_hdc_id = incoming_val - subcollection_type = None - hdc_id = trans.app.security.decode_id( encoded_hdc_id ) - hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) - collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) - if subcollection_type is not None: - from galaxy.dataset_collections import subcollections - subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type ) - return permutations.input_classification.MATCHED, subcollection_elements - else: - hdas = hdc.collection.dataset_instances - return permutations.input_classification.MATCHED, hdas - else: - return permutations.input_classification.SINGLE, incoming[ input_key ] + return classify_unmodified_parameter( input_key ) # Stick an unexpanded version of multirun keys so they can be replaced, # by expand_mult_inputs. @@ -76,20 +73,59 @@ multirun_found = False collection_multirun_found = False for key, value in incoming.iteritems(): - multirun_found = try_replace_key( key, "|__multirun__" ) or multirun_found - collection_multirun_found = try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found + if isinstance( value, dict ) and 'values' in value: + batch = value.get( 'batch', False ) + if batch: + if __collection_multirun_parameter( value ): + collection_multirun_found = True + else: + multirun_found = True + else: + continue + else: + # Old-style batching (remove someday? - pretty hacky and didn't live in API long) + try_replace_key( key, "|__multirun__" ) or multirun_found + try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found - if sum( [ 1 if f else 0 for f in [ multirun_found, collection_multirun_found ] ] ) > 1: - # In theory doable, but to complicated for a first pass. - message = "Cannot specify parallel execution across both multiple datasets and dataset collections." - raise exceptions.ToolMetaParameterException( message ) + expanded_incomings = permutations.expand_multi_inputs( incoming_template, classifier ) + if collections_to_match.has_collections(): + collection_info = trans.app.dataset_collections_service.match_collections( collections_to_match ) + else: + collection_info = None + return expanded_incomings, collection_info - if multirun_found: - return permutations.expand_multi_inputs( incoming_template, classifier ), None + +def __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ): + # If subcollectin multirun of data_collection param - value will + # be "hdca_id|subcollection_type" else it will just be hdca_id + if "|" in incoming_val: + encoded_hdc_id, subcollection_type = incoming_val.split( "|", 1 ) else: - expanded_incomings = permutations.expand_multi_inputs( incoming_template, collection_classifier ) - if collections_to_match.has_collections(): - collection_info = trans.app.dataset_collections_service.match_collections( collections_to_match ) - else: - collection_info = None - return expanded_incomings, collection_info + try: + src = incoming_val[ "src" ] + if src != "hdca": + raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) + encoded_hdc_id = incoming_val[ "id" ] + subcollection_type = incoming_val.get( 'map_over_type', None ) + except TypeError: + encoded_hdc_id = incoming_val + subcollection_type = None + hdc_id = trans.app.security.decode_id( encoded_hdc_id ) + hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) + collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) + if subcollection_type is not None: + from galaxy.dataset_collections import subcollections + subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type ) + return subcollection_elements + else: + hdas = hdc.collection.dataset_instances + return hdas + + +def __collection_multirun_parameter( value ): + batch_values = util.listify( value[ 'values' ] ) + if len( batch_values ) == 1: + batch_over = batch_values[ 0 ] + if isinstance( batch_over, dict ) and ('src' in batch_over) and (batch_over[ 'src' ] == 'hdca'): + return True + return False diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -109,6 +109,35 @@ output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) self.assertEqual( output1_content.strip(), "Cat1Test" ) + @skip_without_tool( "cat1" ) + def test_run_cat1_listified_param( self ): + # Run simple non-upload tool with an input data parameter. + history_id = self.dataset_populator.new_history() + new_dataset = self.dataset_populator.new_dataset( history_id, content='Cat1Testlistified' ) + inputs = dict( + input1=[dataset_to_param( new_dataset )], + ) + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 1 ) + output1 = outputs[ 0 ] + output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) + self.assertEqual( output1_content.strip(), "Cat1Testlistified" ) + + @skip_without_tool( "cat1" ) + def test_run_cat1_single_meta_wrapper( self ): + # Wrap input in a no-op meta parameter wrapper like Sam is planning to + # use for all UI API submissions. + history_id = self.dataset_populator.new_history() + new_dataset = self.dataset_populator.new_dataset( history_id, content='123' ) + inputs = dict( + input1={ 'batch': False, 'values': [ dataset_to_param( new_dataset ) ] }, + ) + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 1 ) + output1 = outputs[ 0 ] + output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) + self.assertEqual( output1_content.strip(), "123" ) + @skip_without_tool( "validation_default" ) def test_validation( self ): history_id = self.dataset_populator.new_history() @@ -118,6 +147,20 @@ response = self._run( "validation_default", history_id, inputs ) self._assert_status_code_is( response, 400 ) + @skip_without_tool( "collection_paired_test" ) + def test_collection_parameter( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) + inputs = { + "f1": { "src": "hdca", "id": hdca_id }, + } + output = self._run( "collection_paired_test", history_id, inputs, assert_ok=True ) + assert len( output[ 'jobs' ] ) == 1 + assert len( output[ 'implicit_collections' ] ) == 0 + assert len( output[ 'outputs' ] ) == 1 + contents = self.dataset_populator.get_history_dataset_content( history_id, hid=4 ) + assert contents.strip() == "123\n456", contents + @skip_without_tool( "cat1" ) def test_run_cat1_with_two_inputs( self ): # Run tool with an multiple data parameter and grouping (repeat) @@ -135,16 +178,31 @@ self.assertEqual( output1_content.strip(), "Cat1Test\nCat2Test" ) @skip_without_tool( "cat1" ) + def test_multirun_cat1_legacy( self ): + history_id, datasets = self._prepare_cat1_multirun() + inputs = { + "input1|__multirun__": datasets, + } + self._check_cat1_multirun( history_id, inputs ) + + @skip_without_tool( "cat1" ) def test_multirun_cat1( self ): + history_id, datasets = self._prepare_cat1_multirun() + inputs = { + "input1": { + 'batch': True, + 'values': datasets, + }, + } + self._check_cat1_multirun( history_id, inputs ) + + def _prepare_cat1_multirun( self ): history_id = self.dataset_populator.new_history() new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) - inputs = { - "input1|__multirun__": [ - dataset_to_param( new_dataset1 ), - dataset_to_param( new_dataset2 ), - ], - } + return history_id, [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ] + + def _check_cat1_multirun( self, history_id, inputs ): outputs = self._cat1_outputs( history_id, inputs=inputs ) self.assertEquals( len( outputs ), 2 ) output1 = outputs[ 0 ] @@ -154,6 +212,20 @@ self.assertEquals( output1_content.strip(), "123" ) self.assertEquals( output2_content.strip(), "456" ) + @skip_without_tool( "random_lines1" ) + def test_multirun_non_data_parameter( self ): + history_id = self.dataset_populator.new_history() + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123\n456\n789' ) + inputs = { + 'input': dataset_to_param( new_dataset1 ), + 'num_lines': { 'batch': True, 'values': [ 1, 2, 3 ] } + } + outputs = self._run_and_get_outputs( 'random_lines1', history_id, inputs ) + # Assert we have three outputs with 1, 2, and 3 lines respectively. + assert len( outputs ) == 3 + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + assert sorted( map( lambda c: len( c.split( "\n" ) ), outputs_contents ) ) == [ 1, 2, 3 ] + @skip_without_tool( "cat1" ) def test_multirun_in_repeat( self ): history_id = self.dataset_populator.new_history() @@ -177,35 +249,60 @@ self.assertEquals( output2_content.strip(), "Common\n456" ) @skip_without_tool( "cat1" ) - def test_multirun_on_multiple_inputs( self ): - history_id = self.dataset_populator.new_history() - new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) - new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) - new_dataset3 = self.dataset_populator.new_dataset( history_id, content='789' ) - new_dataset4 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + def test_multirun_on_multiple_inputs_legacy( self ): + history_id, first_two, second_two = self._setup_two_multiruns() inputs = { - "input1|__multirun__": [ - dataset_to_param( new_dataset1 ), - dataset_to_param( new_dataset2 ), - ], - 'queries_0|input2|__multirun__': [ - dataset_to_param( new_dataset3 ), - dataset_to_param( new_dataset4 ), - ], + "input1|__multirun__": first_two, + 'queries_0|input2|__multirun__': second_two, } outputs = self._cat1_outputs( history_id, inputs=inputs ) self.assertEquals( len( outputs ), 2 ) outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] assert "123\n789" in outputs_contents assert "456\n0ab" in outputs_contents - # TODO: Once cross production (instead of linking inputs) is an option - # again redo test with these checks... - # self.assertEquals( len( outputs ), 4 ) - # assert "123\n0ab" in outputs_contents - # assert "456\n789" in outputs_contents @skip_without_tool( "cat1" ) - def test_map_over_collection( self ): + def test_multirun_on_multiple_inputs( self ): + history_id, first_two, second_two = self._setup_two_multiruns() + inputs = { + "input1": { 'batch': True, 'values': first_two }, + 'queries_0|input2': { 'batch': True, 'values': second_two }, + } + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 2 ) + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + assert "123\n789" in outputs_contents + assert "456\n0ab" in outputs_contents + + @skip_without_tool( "cat1" ) + def test_multirun_on_multiple_inputs_unlinked( self ): + history_id, first_two, second_two = self._setup_two_multiruns() + inputs = { + "input1": { 'batch': True, 'linked': False, 'values': first_two }, + 'queries_0|input2': { 'batch': True, 'linked': False, 'values': second_two }, + } + outputs = self._cat1_outputs( history_id, inputs=inputs ) + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + self.assertEquals( len( outputs ), 4 ) + assert "123\n789" in outputs_contents + assert "456\n0ab" in outputs_contents + assert "123\n0ab" in outputs_contents + assert "456\n789" in outputs_contents + + def _setup_two_multiruns( self ): + history_id = self.dataset_populator.new_history() + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) + new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) + new_dataset3 = self.dataset_populator.new_dataset( history_id, content='789' ) + new_dataset4 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + return ( + history_id, + [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ], + [ dataset_to_param( new_dataset3 ), dataset_to_param( new_dataset4 ) ] + ) + + @skip_without_tool( "cat1" ) + def test_map_over_collection_legacy( self ): history_id = self.dataset_populator.new_history() hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) inputs = { @@ -214,6 +311,18 @@ # first, next test method tests other. "input1|__collection_multirun__": hdca_id, } + self._run_and_check_simple_collection_mapping( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_collection( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) + inputs = { + "input1": { 'batch': True, 'values': [ { 'src': 'hdca', 'id': hdca_id } ] }, + } + self._run_and_check_simple_collection_mapping( history_id, inputs ) + + def _run_and_check_simple_collection_mapping( self, history_id, inputs ): create = self._run_cat1( history_id, inputs=inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -229,12 +338,24 @@ self.assertEquals( output2_content.strip(), "456" ) @skip_without_tool( "cat1" ) - def test_map_over_nested_collections( self ): + def test_map_over_nested_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca_id = self.__build_nested_list( history_id ) inputs = { "input1|__collection_multirun__": dict( src="hdca", id=hdca_id ), } + self._check_simple_cat1_over_nested_collections( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_nested_collections( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_nested_list( history_id ) + inputs = { + "input1": { 'batch': True, 'values': [ dict( src="hdca", id=hdca_id ) ] }, + } + self._check_simple_cat1_over_nested_collections( history_id, inputs ) + + def _check_simple_cat1_over_nested_collections( self, history_id, inputs ): create = self._run_cat1( history_id, inputs=inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -257,7 +378,7 @@ self.assertEquals( outputs[ 0 ][ "id" ], first_object_forward_element[ "object" ][ "id" ] ) @skip_without_tool( "cat1" ) - def test_map_over_two_collections( self ): + def test_map_over_two_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) @@ -265,7 +386,24 @@ "input1|__collection_multirun__": hdca1_id, "queries_0|input2|__collection_multirun__": hdca2_id, } - outputs = self._cat1_outputs( history_id, inputs=inputs ) + self._check_map_cat1_over_two_collections( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_two_collections( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) + inputs = { + "input1": { 'batch': True, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'values': [ { 'src': 'hdca', 'id': hdca2_id } ] }, + } + self._check_map_cat1_over_two_collections( history_id, inputs ) + + def _check_map_cat1_over_two_collections( self, history_id, inputs ): + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] self.assertEquals( len( outputs ), 2 ) output1 = outputs[ 0 ] output2 = outputs[ 1 ] @@ -274,6 +412,49 @@ self.assertEquals( output1_content.strip(), "123\n789" ) self.assertEquals( output2_content.strip(), "456\n0ab" ) + self.assertEquals( len( response_object[ 'jobs' ] ), 2 ) + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 1 ) + + @skip_without_tool( "cat1" ) + def test_map_over_two_collections_unlinked( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) + inputs = { + "input1": { 'batch': True, 'linked': False, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'linked': False, 'values': [ { 'src': 'hdca', 'id': hdca2_id } ] }, + } + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] + self.assertEquals( len( outputs ), 4 ) + + self.assertEquals( len( response_object[ 'jobs' ] ), 4 ) + # Implicit collections not created with unlinked inputs yet - this may + # be problematic. + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 0 ) + + @skip_without_tool( "cat1" ) + def test_map_over_collected_and_individual_datasets( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='789' ) + new_dataset2 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + + inputs = { + "input1": { 'batch': True, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'values': [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ] }, + } + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] + self.assertEquals( len( outputs ), 2 ) + + self.assertEquals( len( response_object[ 'jobs' ] ), 2 ) + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 1 ) + @skip_without_tool( "cat1" ) def test_cannot_map_over_incompatible_collections( self ): history_id = self.dataset_populator.new_history() @@ -289,7 +470,7 @@ assert run_response.status_code >= 400 @skip_without_tool( "multi_data_param" ) - def test_reduce_collections( self ): + def test_reduce_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) hdca2_id = self.dataset_collection_populator.create_list_in_history( history_id ).json()[ "id" ] @@ -297,6 +478,20 @@ "f1": "__collection_reduce__|%s" % hdca1_id, "f2": "__collection_reduce__|%s" % hdca2_id, } + self._check_simple_reduce_job( history_id, inputs ) + + @skip_without_tool( "multi_data_param" ) + def test_reduce_collections( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.dataset_collection_populator.create_list_in_history( history_id ).json()[ "id" ] + inputs = { + "f1": { 'src': 'hdca', 'id': hdca1_id }, + "f2": { 'src': 'hdca', 'id': hdca2_id }, + } + self._check_simple_reduce_job( history_id, inputs ) + + def _check_simple_reduce_job( self, history_id, inputs ): create = self._run( "multi_data_param", history_id, inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -310,12 +505,27 @@ assert len( output2_content.strip().split("\n") ) == 3, output2_content @skip_without_tool( "collection_paired_test" ) - def test_subcollection_mapping( self ): + def test_subcollection_mapping_legacy( self ): history_id = self.dataset_populator.new_history() hdca_list_id = self.__build_nested_list( history_id ) inputs = { "f1|__collection_multirun__": "%s|paired" % hdca_list_id } + self._check_simple_subcollection_mapping( history_id, inputs ) + + @skip_without_tool( "collection_paired_test" ) + def test_subcollection_mapping( self ): + history_id = self.dataset_populator.new_history() + hdca_list_id = self.__build_nested_list( history_id ) + inputs = { + "f1": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'map_over_type': 'paired', 'id': hdca_list_id }], + } + } + self._check_simple_subcollection_mapping( history_id, inputs ) + + def _check_simple_subcollection_mapping( self, history_id, inputs ): # Following wait not really needed - just getting so many database # locked errors with sqlite. self.dataset_populator.wait_for_history( history_id, assert_ok=True ) @@ -329,7 +539,7 @@ assert output2_content.strip() == "789\n0ab", output2_content @skip_without_tool( "collection_mixed_param" ) - def test_combined_mapping_and_subcollection_mapping( self ): + def test_combined_mapping_and_subcollection_mapping_legacy( self ): history_id = self.dataset_populator.new_history() nested_list_id = self.__build_nested_list( history_id ) create_response = self.dataset_collection_populator.create_list_in_history( history_id, contents=["xxx", "yyy"] ) @@ -338,6 +548,27 @@ "f1|__collection_multirun__": "%s|paired" % nested_list_id, "f2|__collection_multirun__": list_id, } + self._check_combined_mapping_and_subcollection_mapping( history_id, inputs ) + + @skip_without_tool( "collection_mixed_param" ) + def test_combined_mapping_and_subcollection_mapping( self ): + history_id = self.dataset_populator.new_history() + nested_list_id = self.__build_nested_list( history_id ) + create_response = self.dataset_collection_populator.create_list_in_history( history_id, contents=["xxx", "yyy"] ) + list_id = create_response.json()[ "id" ] + inputs = { + "f1": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'map_over_type': 'paired', 'id': nested_list_id }], + }, + "f2": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'id': list_id }], + }, + } + self._check_combined_mapping_and_subcollection_mapping( history_id, inputs ) + + def _check_combined_mapping_and_subcollection_mapping( self, history_id, inputs ): self.dataset_populator.wait_for_history( history_id, assert_ok=True ) outputs = self._run_and_get_outputs( "collection_mixed_param", history_id, inputs ) assert len( outputs ), 2 diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b test/api/test_workflows.py --- a/test/api/test_workflows.py +++ b/test/api/test_workflows.py @@ -361,13 +361,13 @@ hdca = self.dataset_collection_populator.create_pair_in_history( history_id, contents=["1 2 3\n4 5 6", "7 8 9\n10 11 10"] ).json() hdca_id = hdca[ "id" ] inputs1 = { - "input|__collection_multirun__": hdca_id, + "input": { "batch": True, "values": [ { "src": "hdca", "id": hdca_id } ] }, "num_lines": 2 } implicit_hdca1, job_id1 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs1 ) inputs2 = { - "f1": "__collection_reduce__|%s" % ( implicit_hdca1[ "id" ] ), - "f2": "__collection_reduce__|%s" % ( implicit_hdca1[ "id" ] ) + "f1": { "src": "hdca", "id": implicit_hdca1[ "id" ] }, + "f2": { "src": "hdca", "id": implicit_hdca1[ "id" ] }, } reduction_run_output = self.dataset_populator.run_tool( tool_id="multi_data_param", @@ -413,12 +413,12 @@ hdca = self.dataset_collection_populator.create_pair_in_history( history_id, contents=["1 2 3\n4 5 6", "7 8 9\n10 11 10"] ).json() hdca_id = hdca[ "id" ] inputs1 = { - "input|__collection_multirun__": hdca_id, + "input": { "batch": True, "values": [ { "src": "hdca", "id": hdca_id } ] }, "num_lines": 2 } implicit_hdca1, job_id1 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs1 ) inputs2 = { - "input|__collection_multirun__": implicit_hdca1[ "id" ], + "input": { "batch": True, "values": [ { "src": "hdca", "id": implicit_hdca1[ "id" ] } ] }, "num_lines": 1 } _, job_id2 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs2 ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 10 new changesets
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

10 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/d5218767204d/ Changeset: d5218767204d User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Allow sending singleton lists of datasets to single dataset parameters via API tool submission. Request from Sam. Affected #: 2 files diff -r 80d68ea889c01357b520ecefc56f86b0a44549fc -r d5218767204d12af6275b320409e507ae3dfe95b lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py +++ b/lib/galaxy/tools/parameters/basic.py @@ -1903,6 +1903,10 @@ raise ValueError( "The previously selected dataset has been previously deleted" ) if hasattr( v, "dataset" ) and v.dataset.state in [ galaxy.model.Dataset.states.ERROR, galaxy.model.Dataset.states.DISCARDED ]: raise ValueError( "The previously selected dataset has entered an unusable state" ) + if not self.multiple: + if len( values ) > 1: + raise ValueError( "More than one dataset supplied to single input dataset parameter.") + rval = values[ 0 ] return rval def to_string( self, value, app ): diff -r 80d68ea889c01357b520ecefc56f86b0a44549fc -r d5218767204d12af6275b320409e507ae3dfe95b test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -109,6 +109,20 @@ output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) self.assertEqual( output1_content.strip(), "Cat1Test" ) + @skip_without_tool( "cat1" ) + def test_run_cat1_listified_param( self ): + # Run simple non-upload tool with an input data parameter. + history_id = self.dataset_populator.new_history() + new_dataset = self.dataset_populator.new_dataset( history_id, content='Cat1Testlistified' ) + inputs = dict( + input1=[dataset_to_param( new_dataset )], + ) + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 1 ) + output1 = outputs[ 0 ] + output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) + self.assertEqual( output1_content.strip(), "Cat1Testlistified" ) + @skip_without_tool( "validation_default" ) def test_validation( self ): history_id = self.dataset_populator.new_history() https://bitbucket.org/galaxy/galaxy-central/commits/d9c75786e6b4/ Changeset: d9c75786e6b4 User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Overhaul multi-run and collection multi-run tool API jobs. Adding consistency allowing each parameter to be wrapped in a object describing the meta-properties of the submitting value - this was requested by Sam to make the new tool form easier to manage, it makes multi-running properties work for non-data parameters, and allows linked/unlinked specification of parameters. Affected #: 2 files diff -r d5218767204d12af6275b320409e507ae3dfe95b -r d9c75786e6b43f18f9c1eaded635d5d6b236e2a7 lib/galaxy/tools/parameters/meta.py --- a/lib/galaxy/tools/parameters/meta.py +++ b/lib/galaxy/tools/parameters/meta.py @@ -14,6 +14,28 @@ execution). """ + def classifiy_unmodified_parameter( input_key ): + value = incoming[ input_key ] + if isinstance( value, dict ) and 'values' in value: + # Explicit meta wrapper for inputs... + is_batch = value.get( 'batch', False ) + is_linked = value.get( 'linked', True ) + if is_batch and is_linked: + classification = permutations.input_classification.MATCHED + elif is_batch: + classification = permutations.input_classification.MULTIPLIED + else: + classification = permutations.input_classification.SINGLE + if __collection_multirun_parameter( value ): + collection_value = value[ 'values' ][ 0 ] + values = __expand_collection_parameter( trans, input_key, collection_value, collections_to_match ) + else: + values = value[ 'values' ] + else: + classification = permutations.input_classification.SINGLE + values = value + return classification, values + def classifier( input_key ): multirun_key = "%s|__multirun__" % input_key if multirun_key in incoming: @@ -25,7 +47,7 @@ multi_value = None return permutations.input_classification.SINGLE, multi_value[ 0 ] else: - return permutations.input_classification.SINGLE, incoming[ input_key ] + return classifiy_unmodified_parameter( input_key ) from galaxy.dataset_collections import matching collections_to_match = matching.CollectionsToMatch() @@ -34,31 +56,10 @@ multirun_key = "%s|__collection_multirun__" % input_key if multirun_key in incoming: incoming_val = incoming[ multirun_key ] - # If subcollectin multirun of data_collection param - value will - # be "hdca_id|subcollection_type" else it will just be hdca_id - if "|" in incoming_val: - encoded_hdc_id, subcollection_type = incoming_val.split( "|", 1 ) - else: - try: - src = incoming_val[ "src" ] - if src != "hdca": - raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) - encoded_hdc_id = incoming_val[ "id" ] - except TypeError: - encoded_hdc_id = incoming_val - subcollection_type = None - hdc_id = trans.app.security.decode_id( encoded_hdc_id ) - hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) - collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) - if subcollection_type is not None: - from galaxy.dataset_collections import subcollections - subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type ) - return permutations.input_classification.MATCHED, subcollection_elements - else: - hdas = hdc.collection.dataset_instances - return permutations.input_classification.MATCHED, hdas + values = __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ) + return permutations.input_classification.MATCHED, values else: - return permutations.input_classification.SINGLE, incoming[ input_key ] + return classifiy_unmodified_parameter( input_key ) # Stick an unexpanded version of multirun keys so they can be replaced, # by expand_mult_inputs. @@ -76,8 +77,19 @@ multirun_found = False collection_multirun_found = False for key, value in incoming.iteritems(): - multirun_found = try_replace_key( key, "|__multirun__" ) or multirun_found - collection_multirun_found = try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found + if isinstance( value, dict ) and 'values' in value: + batch = value.get( 'batch', False ) + if batch: + if __collection_multirun_parameter( value ): + collection_multirun_found = True + else: + multirun_found = True + else: + continue + else: + # Old-style batching (remove someday - didn't live in API long?) + multirun_found = try_replace_key( key, "|__multirun__" ) or multirun_found + collection_multirun_found = try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found if sum( [ 1 if f else 0 for f in [ multirun_found, collection_multirun_found ] ] ) > 1: # In theory doable, but to complicated for a first pass. @@ -93,3 +105,38 @@ else: collection_info = None return expanded_incomings, collection_info + + +def __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ): + # If subcollectin multirun of data_collection param - value will + # be "hdca_id|subcollection_type" else it will just be hdca_id + if "|" in incoming_val: + encoded_hdc_id, subcollection_type = incoming_val.split( "|", 1 ) + else: + try: + src = incoming_val[ "src" ] + if src != "hdca": + raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) + encoded_hdc_id = incoming_val[ "id" ] + except TypeError: + encoded_hdc_id = incoming_val + subcollection_type = None + hdc_id = trans.app.security.decode_id( encoded_hdc_id ) + hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) + collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) + if subcollection_type is not None: + from galaxy.dataset_collections import subcollections + subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type ) + return subcollection_elements + else: + hdas = hdc.collection.dataset_instances + return hdas + + +def __collection_multirun_parameter( value ): + batch_values = util.listify( value[ 'values' ] ) + if len( batch_values ) == 1: + batch_over = batch_values[ 0 ] + if isinstance( batch_over, dict ) and ('src' in batch_over) and (batch_over[ 'src' ] == 'hdca'): + return True + return False diff -r d5218767204d12af6275b320409e507ae3dfe95b -r d9c75786e6b43f18f9c1eaded635d5d6b236e2a7 test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -123,6 +123,22 @@ output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) self.assertEqual( output1_content.strip(), "Cat1Testlistified" ) + @skip_without_tool( "cat1" ) + def test_run_cat1_single_meta_wrapper( self ): + # Wrap input in a no-op meta parameter wrapper like Sam is planning to + # use for all UI API submissions. + history_id = self.dataset_populator.new_history() + new_dataset = self.dataset_populator.new_dataset( history_id, content='123' ) + inputs = dict( + input1={ 'batch': False, 'values': [ dataset_to_param( new_dataset ) ] }, + ) + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 1 ) + output1 = outputs[ 0 ] + output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) + self.assertEqual( output1_content.strip(), "123" ) + + @skip_without_tool( "validation_default" ) def test_validation( self ): history_id = self.dataset_populator.new_history() @@ -149,16 +165,31 @@ self.assertEqual( output1_content.strip(), "Cat1Test\nCat2Test" ) @skip_without_tool( "cat1" ) + def test_multirun_cat1_legacy( self ): + history_id, datasets = self._prepare_cat1_multirun() + inputs = { + "input1|__multirun__": datasets, + } + self._check_cat1_multirun( history_id, inputs ) + + @skip_without_tool( "cat1" ) def test_multirun_cat1( self ): + history_id, datasets = self._prepare_cat1_multirun() + inputs = { + "input1": { + 'batch': True, + 'values': datasets, + }, + } + self._check_cat1_multirun( history_id, inputs ) + + def _prepare_cat1_multirun( self ): history_id = self.dataset_populator.new_history() new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) - inputs = { - "input1|__multirun__": [ - dataset_to_param( new_dataset1 ), - dataset_to_param( new_dataset2 ), - ], - } + return history_id, [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ] + + def _check_cat1_multirun( self, history_id, inputs ): outputs = self._cat1_outputs( history_id, inputs=inputs ) self.assertEquals( len( outputs ), 2 ) output1 = outputs[ 0 ] @@ -168,6 +199,20 @@ self.assertEquals( output1_content.strip(), "123" ) self.assertEquals( output2_content.strip(), "456" ) + @skip_without_tool( "random_lines1" ) + def test_multirun_non_data_parameter( self ): + history_id = self.dataset_populator.new_history() + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123\n456\n789' ) + inputs = { + 'input': dataset_to_param( new_dataset1 ), + 'num_lines': { 'batch': True, 'values': [ 1, 2, 3 ] } + } + outputs = self._run_and_get_outputs( 'random_lines1', history_id, inputs ) + # Assert we have three outputs with 1, 2, and 3 lines respectively. + assert len( outputs ) == 3 + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + assert sorted( map( lambda c: len( c.split( "\n" ) ), outputs_contents ) ) == [ 1, 2, 3 ] + @skip_without_tool( "cat1" ) def test_multirun_in_repeat( self ): history_id = self.dataset_populator.new_history() @@ -191,35 +236,60 @@ self.assertEquals( output2_content.strip(), "Common\n456" ) @skip_without_tool( "cat1" ) - def test_multirun_on_multiple_inputs( self ): - history_id = self.dataset_populator.new_history() - new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) - new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) - new_dataset3 = self.dataset_populator.new_dataset( history_id, content='789' ) - new_dataset4 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + def test_multirun_on_multiple_inputs_legacy( self ): + history_id, first_two, second_two = self._setup_two_multiruns() inputs = { - "input1|__multirun__": [ - dataset_to_param( new_dataset1 ), - dataset_to_param( new_dataset2 ), - ], - 'queries_0|input2|__multirun__': [ - dataset_to_param( new_dataset3 ), - dataset_to_param( new_dataset4 ), - ], + "input1|__multirun__": first_two, + 'queries_0|input2|__multirun__': second_two, } outputs = self._cat1_outputs( history_id, inputs=inputs ) self.assertEquals( len( outputs ), 2 ) outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] assert "123\n789" in outputs_contents assert "456\n0ab" in outputs_contents - # TODO: Once cross production (instead of linking inputs) is an option - # again redo test with these checks... - # self.assertEquals( len( outputs ), 4 ) - # assert "123\n0ab" in outputs_contents - # assert "456\n789" in outputs_contents @skip_without_tool( "cat1" ) - def test_map_over_collection( self ): + def test_multirun_on_multiple_inputs( self ): + history_id, first_two, second_two = self._setup_two_multiruns() + inputs = { + "input1": { 'batch': True, 'values': first_two }, + 'queries_0|input2': { 'batch': True, 'values': second_two }, + } + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 2 ) + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + assert "123\n789" in outputs_contents + assert "456\n0ab" in outputs_contents + + @skip_without_tool( "cat1" ) + def test_multirun_on_multiple_inputs_unlinked( self ): + history_id, first_two, second_two = self._setup_two_multiruns() + inputs = { + "input1": { 'batch': True, 'linked': False, 'values': first_two }, + 'queries_0|input2': { 'batch': True, 'linked': False, 'values': second_two }, + } + outputs = self._cat1_outputs( history_id, inputs=inputs ) + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + self.assertEquals( len( outputs ), 4 ) + assert "123\n789" in outputs_contents + assert "456\n0ab" in outputs_contents + assert "123\n0ab" in outputs_contents + assert "456\n789" in outputs_contents + + def _setup_two_multiruns( self ): + history_id = self.dataset_populator.new_history() + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) + new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) + new_dataset3 = self.dataset_populator.new_dataset( history_id, content='789' ) + new_dataset4 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + return ( + history_id, + [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ], + [ dataset_to_param( new_dataset3 ), dataset_to_param( new_dataset4 ) ] + ) + + @skip_without_tool( "cat1" ) + def test_map_over_collection_legacy( self ): history_id = self.dataset_populator.new_history() hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) inputs = { @@ -228,6 +298,18 @@ # first, next test method tests other. "input1|__collection_multirun__": hdca_id, } + self._run_and_check_simple_collection_mapping( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_collection( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) + inputs = { + "input1": { 'batch': True, 'values': [ { 'src': 'hdca', 'id': hdca_id } ] }, + } + self._run_and_check_simple_collection_mapping( history_id, inputs ) + + def _run_and_check_simple_collection_mapping( self, history_id, inputs ): create = self._run_cat1( history_id, inputs=inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -243,12 +325,24 @@ self.assertEquals( output2_content.strip(), "456" ) @skip_without_tool( "cat1" ) - def test_map_over_nested_collections( self ): + def test_map_over_nested_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca_id = self.__build_nested_list( history_id ) inputs = { "input1|__collection_multirun__": dict( src="hdca", id=hdca_id ), } + self._check_simple_cat1_over_nested_collections( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_nested_collections( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_nested_list( history_id ) + inputs = { + "input1": { 'batch': True, 'values': [ dict( src="hdca", id=hdca_id ) ] }, + } + self._check_simple_cat1_over_nested_collections( history_id, inputs ) + + def _check_simple_cat1_over_nested_collections( self, history_id, inputs ): create = self._run_cat1( history_id, inputs=inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -271,7 +365,7 @@ self.assertEquals( outputs[ 0 ][ "id" ], first_object_forward_element[ "object" ][ "id" ] ) @skip_without_tool( "cat1" ) - def test_map_over_two_collections( self ): + def test_map_over_two_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) @@ -279,7 +373,24 @@ "input1|__collection_multirun__": hdca1_id, "queries_0|input2|__collection_multirun__": hdca2_id, } - outputs = self._cat1_outputs( history_id, inputs=inputs ) + self._check_map_cat1_over_two_collections( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_two_collections( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) + inputs = { + "input1": { 'batch': True, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'values': [ { 'src': 'hdca', 'id': hdca2_id } ] }, + } + self._check_map_cat1_over_two_collections( history_id, inputs ) + + def _check_map_cat1_over_two_collections( self, history_id, inputs ): + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] self.assertEquals( len( outputs ), 2 ) output1 = outputs[ 0 ] output2 = outputs[ 1 ] @@ -288,6 +399,29 @@ self.assertEquals( output1_content.strip(), "123\n789" ) self.assertEquals( output2_content.strip(), "456\n0ab" ) + self.assertEquals( len( response_object[ 'jobs' ] ), 2 ) + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 1 ) + + @skip_without_tool( "cat1" ) + def test_map_over_two_collections_unlinked( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) + inputs = { + "input1": { 'batch': True, 'linked': False, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'linked': False, 'values': [ { 'src': 'hdca', 'id': hdca2_id } ] }, + } + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] + self.assertEquals( len( outputs ), 4 ) + + self.assertEquals( len( response_object[ 'jobs' ] ), 4 ) + # Implicit collections not created with unlinked inputs yet - this may + # be problematic. + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 0 ) + @skip_without_tool( "cat1" ) def test_cannot_map_over_incompatible_collections( self ): history_id = self.dataset_populator.new_history() https://bitbucket.org/galaxy/galaxy-central/commits/e5e67860cbde/ Changeset: e5e67860cbde User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Allow mixing batch multi-running of collections and individual datasets. I wouldn't really recommend doing this per se - but probably don't want to prevent it either. Affected #: 2 files diff -r d9c75786e6b43f18f9c1eaded635d5d6b236e2a7 -r e5e67860cbde0edb272e1dbd225f4195e8b4785f lib/galaxy/tools/parameters/meta.py --- a/lib/galaxy/tools/parameters/meta.py +++ b/lib/galaxy/tools/parameters/meta.py @@ -36,7 +36,11 @@ values = value return classification, values + from galaxy.dataset_collections import matching + collections_to_match = matching.CollectionsToMatch() + def classifier( input_key ): + collection_multirun_key = "%s|__collection_multirun__" % input_key multirun_key = "%s|__multirun__" % input_key if multirun_key in incoming: multi_value = util.listify( incoming[ multirun_key ] ) @@ -46,16 +50,8 @@ if len( multi_value ) == 0: multi_value = None return permutations.input_classification.SINGLE, multi_value[ 0 ] - else: - return classifiy_unmodified_parameter( input_key ) - - from galaxy.dataset_collections import matching - collections_to_match = matching.CollectionsToMatch() - - def collection_classifier( input_key ): - multirun_key = "%s|__collection_multirun__" % input_key - if multirun_key in incoming: - incoming_val = incoming[ multirun_key ] + elif collection_multirun_key in incoming: + incoming_val = incoming[ collection_multirun_key ] values = __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ) return permutations.input_classification.MATCHED, values else: @@ -87,24 +83,16 @@ else: continue else: - # Old-style batching (remove someday - didn't live in API long?) - multirun_found = try_replace_key( key, "|__multirun__" ) or multirun_found - collection_multirun_found = try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found + # Old-style batching (remove someday? - pretty hacky and didn't live in API long) + try_replace_key( key, "|__multirun__" ) or multirun_found + try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found - if sum( [ 1 if f else 0 for f in [ multirun_found, collection_multirun_found ] ] ) > 1: - # In theory doable, but to complicated for a first pass. - message = "Cannot specify parallel execution across both multiple datasets and dataset collections." - raise exceptions.ToolMetaParameterException( message ) - - if multirun_found: - return permutations.expand_multi_inputs( incoming_template, classifier ), None + expanded_incomings = permutations.expand_multi_inputs( incoming_template, classifier ) + if collections_to_match.has_collections(): + collection_info = trans.app.dataset_collections_service.match_collections( collections_to_match ) else: - expanded_incomings = permutations.expand_multi_inputs( incoming_template, collection_classifier ) - if collections_to_match.has_collections(): - collection_info = trans.app.dataset_collections_service.match_collections( collections_to_match ) - else: - collection_info = None - return expanded_incomings, collection_info + collection_info = None + return expanded_incomings, collection_info def __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ): diff -r d9c75786e6b43f18f9c1eaded635d5d6b236e2a7 -r e5e67860cbde0edb272e1dbd225f4195e8b4785f test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -423,6 +423,26 @@ self.assertEquals( len( response_object[ 'implicit_collections' ] ), 0 ) @skip_without_tool( "cat1" ) + def test_map_over_collected_and_individual_datasets( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='789' ) + new_dataset2 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + + inputs = { + "input1": { 'batch': True, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'values': [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ] }, + } + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] + self.assertEquals( len( outputs ), 2 ) + + self.assertEquals( len( response_object[ 'jobs' ] ), 2 ) + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 1 ) + + @skip_without_tool( "cat1" ) def test_cannot_map_over_incompatible_collections( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) https://bitbucket.org/galaxy/galaxy-central/commits/a7f6b4b1468c/ Changeset: a7f6b4b1468c User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Simplified dataset collection reductions via API. Old tool form needed to encode every value as a string so I had done "__collection_reduction__|<hdca_id>" to distinguish that value from an "<hda_id>" - since hdca and hdas can have the same encoded ids. The new tool form API is going to use the API which allows for richer object representations - so {"src": "hda", "id": "<hda_id>"} versus {"src": "hdca", "id": "<hdca_id>"} should be enough to distinguish between passing an HDA and an HDCA to a multiple input data parameter. Affected #: 2 files diff -r e5e67860cbde0edb272e1dbd225f4195e8b4785f -r a7f6b4b1468cc52237981a7e7c1645f4720f8b1e lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py +++ b/lib/galaxy/tools/parameters/basic.py @@ -1887,6 +1887,11 @@ elif isinstance( value, dict ) and 'src' in value and 'id' in value: if value['src'] == 'hda': rval = trans.sa_session.query( trans.app.model.HistoryDatasetAssociation ).get( trans.app.security.decode_id(value['id']) ) + elif value['src'] == 'hdca': + decoded_id = trans.app.security.decode_id( value[ 'id' ] ) + rval = trans.sa_session.query( trans.app.model.HistoryDatasetCollectionAssociation ).get( decoded_id ) + else: + raise ValueError("Unknown input source %s passed to job submission API." % value['src']) elif str( value ).startswith( "__collection_reduce__|" ): encoded_id = str( value )[ len( "__collection_reduce__|" ): ] decoded_id = trans.app.security.decode_id( encoded_id ) diff -r e5e67860cbde0edb272e1dbd225f4195e8b4785f -r a7f6b4b1468cc52237981a7e7c1645f4720f8b1e test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -457,7 +457,7 @@ assert run_response.status_code >= 400 @skip_without_tool( "multi_data_param" ) - def test_reduce_collections( self ): + def test_reduce_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) hdca2_id = self.dataset_collection_populator.create_list_in_history( history_id ).json()[ "id" ] @@ -465,6 +465,20 @@ "f1": "__collection_reduce__|%s" % hdca1_id, "f2": "__collection_reduce__|%s" % hdca2_id, } + self._check_simple_reduce_job( history_id, inputs ) + + @skip_without_tool( "multi_data_param" ) + def test_reduce_collections( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.dataset_collection_populator.create_list_in_history( history_id ).json()[ "id" ] + inputs = { + "f1": { 'src': 'hdca', 'id': hdca1_id }, + "f2": { 'src': 'hdca', 'id': hdca2_id }, + } + self._check_simple_reduce_job( history_id, inputs ) + + def _check_simple_reduce_job( self, history_id, inputs ): create = self._run( "multi_data_param", history_id, inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] https://bitbucket.org/galaxy/galaxy-central/commits/6185cea44918/ Changeset: 6185cea44918 User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Redo API for subcollection mapping steps in tools. Like the reductions - was previously constrained by sequeezing these values into simple strings - now the tool form will target the API I think this expanded version is a little more straight-forward (though verbose). Adds consistency with rest of the tool form API changes. Affected #: 2 files diff -r a7f6b4b1468cc52237981a7e7c1645f4720f8b1e -r 6185cea4491868aea6207879e88118402f4344d3 lib/galaxy/tools/parameters/meta.py --- a/lib/galaxy/tools/parameters/meta.py +++ b/lib/galaxy/tools/parameters/meta.py @@ -106,9 +106,10 @@ if src != "hdca": raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) encoded_hdc_id = incoming_val[ "id" ] + subcollection_type = incoming_val.get( 'map_over_type', None ) except TypeError: encoded_hdc_id = incoming_val - subcollection_type = None + subcollection_type = None hdc_id = trans.app.security.decode_id( encoded_hdc_id ) hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) diff -r a7f6b4b1468cc52237981a7e7c1645f4720f8b1e -r 6185cea4491868aea6207879e88118402f4344d3 test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -492,12 +492,27 @@ assert len( output2_content.strip().split("\n") ) == 3, output2_content @skip_without_tool( "collection_paired_test" ) - def test_subcollection_mapping( self ): + def test_subcollection_mapping_legacy( self ): history_id = self.dataset_populator.new_history() hdca_list_id = self.__build_nested_list( history_id ) inputs = { "f1|__collection_multirun__": "%s|paired" % hdca_list_id } + self._check_simple_subcollection_mapping( history_id, inputs ) + + @skip_without_tool( "collection_paired_test" ) + def test_subcollection_mapping( self ): + history_id = self.dataset_populator.new_history() + hdca_list_id = self.__build_nested_list( history_id ) + inputs = { + "f1": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'map_over_type': 'paired', 'id': hdca_list_id }], + } + } + self._check_simple_subcollection_mapping( history_id, inputs ) + + def _check_simple_subcollection_mapping( self, history_id, inputs ): # Following wait not really needed - just getting so many database # locked errors with sqlite. self.dataset_populator.wait_for_history( history_id, assert_ok=True ) @@ -511,7 +526,7 @@ assert output2_content.strip() == "789\n0ab", output2_content @skip_without_tool( "collection_mixed_param" ) - def test_combined_mapping_and_subcollection_mapping( self ): + def test_combined_mapping_and_subcollection_mapping_legacy( self ): history_id = self.dataset_populator.new_history() nested_list_id = self.__build_nested_list( history_id ) create_response = self.dataset_collection_populator.create_list_in_history( history_id, contents=["xxx", "yyy"] ) @@ -520,6 +535,27 @@ "f1|__collection_multirun__": "%s|paired" % nested_list_id, "f2|__collection_multirun__": list_id, } + self._check_combined_mapping_and_subcollection_mapping( history_id, inputs ) + + @skip_without_tool( "collection_mixed_param" ) + def test_combined_mapping_and_subcollection_mapping( self ): + history_id = self.dataset_populator.new_history() + nested_list_id = self.__build_nested_list( history_id ) + create_response = self.dataset_collection_populator.create_list_in_history( history_id, contents=["xxx", "yyy"] ) + list_id = create_response.json()[ "id" ] + inputs = { + "f1": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'map_over_type': 'paired', 'id': nested_list_id }], + }, + "f2": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'id': list_id }], + }, + } + self._check_combined_mapping_and_subcollection_mapping( history_id, inputs ) + + def _check_combined_mapping_and_subcollection_mapping( self, history_id, inputs ): self.dataset_populator.wait_for_history( history_id, assert_ok=True ) outputs = self._run_and_get_outputs( "collection_mixed_param", history_id, inputs ) assert len( outputs ), 2 https://bitbucket.org/galaxy/galaxy-central/commits/b8bc9b8b9806/ Changeset: b8bc9b8b9806 User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Add tools test for data_collection input parameter specification. Affected #: 1 file diff -r 6185cea4491868aea6207879e88118402f4344d3 -r b8bc9b8b980689cbd32ad999675a31203574afb9 test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -148,6 +148,20 @@ response = self._run( "validation_default", history_id, inputs ) self._assert_status_code_is( response, 400 ) + @skip_without_tool( "collection_paired_test" ) + def test_collection_parameter( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) + inputs = { + "f1": { "src": "hdca", "id": hdca_id }, + } + output = self._run( "collection_paired_test", history_id, inputs, assert_ok=True ) + assert len( output[ 'jobs' ] ) == 1 + assert len( output[ 'implicit_collections' ] ) == 0 + assert len( output[ 'outputs' ] ) == 1 + contents = self.dataset_populator.get_history_dataset_content( history_id, hid=4 ) + assert contents.strip() == "123\n456", contents + @skip_without_tool( "cat1" ) def test_run_cat1_with_two_inputs( self ): # Run tool with an multiple data parameter and grouping (repeat) https://bitbucket.org/galaxy/galaxy-central/commits/f5ea6456c64e/ Changeset: f5ea6456c64e User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: PEP-8 fix. Affected #: 1 file diff -r b8bc9b8b980689cbd32ad999675a31203574afb9 -r f5ea6456c64ece76db8528eee5c88eb839cbaa34 test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -138,7 +138,6 @@ output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) self.assertEqual( output1_content.strip(), "123" ) - @skip_without_tool( "validation_default" ) def test_validation( self ): history_id = self.dataset_populator.new_history() https://bitbucket.org/galaxy/galaxy-central/commits/e66166960524/ Changeset: e66166960524 User: jmchilton Date: 2014-09-16 15:57:48+00:00 Summary: Update map/reduce workflow tests for newer API constructs. Old ones still work - but I wanted to verify the new changes didn't cause any unintended consequences with workflows. Affected #: 1 file diff -r f5ea6456c64ece76db8528eee5c88eb839cbaa34 -r e6616696052491bf563b690865f307f2de96d57c test/api/test_workflows.py --- a/test/api/test_workflows.py +++ b/test/api/test_workflows.py @@ -350,13 +350,13 @@ hdca = self.dataset_collection_populator.create_pair_in_history( history_id, contents=["1 2 3\n4 5 6", "7 8 9\n10 11 10"] ).json() hdca_id = hdca[ "id" ] inputs1 = { - "input|__collection_multirun__": hdca_id, + "input": { "batch": True, "values": [ { "src": "hdca", "id": hdca_id } ] }, "num_lines": 2 } implicit_hdca1, job_id1 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs1 ) inputs2 = { - "f1": "__collection_reduce__|%s" % ( implicit_hdca1[ "id" ] ), - "f2": "__collection_reduce__|%s" % ( implicit_hdca1[ "id" ] ) + "f1": { "src": "hdca", "id": implicit_hdca1[ "id" ] }, + "f2": { "src": "hdca", "id": implicit_hdca1[ "id" ] }, } reduction_run_output = self.dataset_populator.run_tool( tool_id="multi_data_param", @@ -402,12 +402,12 @@ hdca = self.dataset_collection_populator.create_pair_in_history( history_id, contents=["1 2 3\n4 5 6", "7 8 9\n10 11 10"] ).json() hdca_id = hdca[ "id" ] inputs1 = { - "input|__collection_multirun__": hdca_id, + "input": { "batch": True, "values": [ { "src": "hdca", "id": hdca_id } ] }, "num_lines": 2 } implicit_hdca1, job_id1 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs1 ) inputs2 = { - "input|__collection_multirun__": implicit_hdca1[ "id" ], + "input": { "batch": True, "values": [ { "src": "hdca", "id": implicit_hdca1[ "id" ] } ] }, "num_lines": 1 } _, job_id2 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs2 ) https://bitbucket.org/galaxy/galaxy-central/commits/708d988d3e7b/ Changeset: 708d988d3e7b User: jmchilton Date: 2014-09-16 18:51:31+00:00 Summary: Fix spelling error in method name (thanks Nicola!). Affected #: 1 file diff -r e6616696052491bf563b690865f307f2de96d57c -r 708d988d3e7bf0ea9c91e0c2b2cbdc53037d3544 lib/galaxy/tools/parameters/meta.py --- a/lib/galaxy/tools/parameters/meta.py +++ b/lib/galaxy/tools/parameters/meta.py @@ -14,7 +14,7 @@ execution). """ - def classifiy_unmodified_parameter( input_key ): + def classify_unmodified_parameter( input_key ): value = incoming[ input_key ] if isinstance( value, dict ) and 'values' in value: # Explicit meta wrapper for inputs... @@ -55,7 +55,7 @@ values = __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ) return permutations.input_classification.MATCHED, values else: - return classifiy_unmodified_parameter( input_key ) + return classify_unmodified_parameter( input_key ) # Stick an unexpanded version of multirun keys so they can be replaced, # by expand_mult_inputs. https://bitbucket.org/galaxy/galaxy-central/commits/df92b55d194a/ Changeset: df92b55d194a User: jmchilton Date: 2014-09-22 14:24:39+00:00 Summary: Merged in jmchilton/galaxy-central-fork-1 (pull request #496) More consistent tool API for map/reduce operations. Affected #: 4 files diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py +++ b/lib/galaxy/tools/parameters/basic.py @@ -1887,6 +1887,11 @@ elif isinstance( value, dict ) and 'src' in value and 'id' in value: if value['src'] == 'hda': rval = trans.sa_session.query( trans.app.model.HistoryDatasetAssociation ).get( trans.app.security.decode_id(value['id']) ) + elif value['src'] == 'hdca': + decoded_id = trans.app.security.decode_id( value[ 'id' ] ) + rval = trans.sa_session.query( trans.app.model.HistoryDatasetCollectionAssociation ).get( decoded_id ) + else: + raise ValueError("Unknown input source %s passed to job submission API." % value['src']) elif str( value ).startswith( "__collection_reduce__|" ): encoded_id = str( value )[ len( "__collection_reduce__|" ): ] decoded_id = trans.app.security.decode_id( encoded_id ) @@ -1903,6 +1908,10 @@ raise ValueError( "The previously selected dataset has been previously deleted" ) if hasattr( v, "dataset" ) and v.dataset.state in [ galaxy.model.Dataset.states.ERROR, galaxy.model.Dataset.states.DISCARDED ]: raise ValueError( "The previously selected dataset has entered an unusable state" ) + if not self.multiple: + if len( values ) > 1: + raise ValueError( "More than one dataset supplied to single input dataset parameter.") + rval = values[ 0 ] return rval def to_string( self, value, app ): diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b lib/galaxy/tools/parameters/meta.py --- a/lib/galaxy/tools/parameters/meta.py +++ b/lib/galaxy/tools/parameters/meta.py @@ -14,7 +14,33 @@ execution). """ + def classify_unmodified_parameter( input_key ): + value = incoming[ input_key ] + if isinstance( value, dict ) and 'values' in value: + # Explicit meta wrapper for inputs... + is_batch = value.get( 'batch', False ) + is_linked = value.get( 'linked', True ) + if is_batch and is_linked: + classification = permutations.input_classification.MATCHED + elif is_batch: + classification = permutations.input_classification.MULTIPLIED + else: + classification = permutations.input_classification.SINGLE + if __collection_multirun_parameter( value ): + collection_value = value[ 'values' ][ 0 ] + values = __expand_collection_parameter( trans, input_key, collection_value, collections_to_match ) + else: + values = value[ 'values' ] + else: + classification = permutations.input_classification.SINGLE + values = value + return classification, values + + from galaxy.dataset_collections import matching + collections_to_match = matching.CollectionsToMatch() + def classifier( input_key ): + collection_multirun_key = "%s|__collection_multirun__" % input_key multirun_key = "%s|__multirun__" % input_key if multirun_key in incoming: multi_value = util.listify( incoming[ multirun_key ] ) @@ -24,41 +50,12 @@ if len( multi_value ) == 0: multi_value = None return permutations.input_classification.SINGLE, multi_value[ 0 ] + elif collection_multirun_key in incoming: + incoming_val = incoming[ collection_multirun_key ] + values = __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ) + return permutations.input_classification.MATCHED, values else: - return permutations.input_classification.SINGLE, incoming[ input_key ] - - from galaxy.dataset_collections import matching - collections_to_match = matching.CollectionsToMatch() - - def collection_classifier( input_key ): - multirun_key = "%s|__collection_multirun__" % input_key - if multirun_key in incoming: - incoming_val = incoming[ multirun_key ] - # If subcollectin multirun of data_collection param - value will - # be "hdca_id|subcollection_type" else it will just be hdca_id - if "|" in incoming_val: - encoded_hdc_id, subcollection_type = incoming_val.split( "|", 1 ) - else: - try: - src = incoming_val[ "src" ] - if src != "hdca": - raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) - encoded_hdc_id = incoming_val[ "id" ] - except TypeError: - encoded_hdc_id = incoming_val - subcollection_type = None - hdc_id = trans.app.security.decode_id( encoded_hdc_id ) - hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) - collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) - if subcollection_type is not None: - from galaxy.dataset_collections import subcollections - subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type ) - return permutations.input_classification.MATCHED, subcollection_elements - else: - hdas = hdc.collection.dataset_instances - return permutations.input_classification.MATCHED, hdas - else: - return permutations.input_classification.SINGLE, incoming[ input_key ] + return classify_unmodified_parameter( input_key ) # Stick an unexpanded version of multirun keys so they can be replaced, # by expand_mult_inputs. @@ -76,20 +73,59 @@ multirun_found = False collection_multirun_found = False for key, value in incoming.iteritems(): - multirun_found = try_replace_key( key, "|__multirun__" ) or multirun_found - collection_multirun_found = try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found + if isinstance( value, dict ) and 'values' in value: + batch = value.get( 'batch', False ) + if batch: + if __collection_multirun_parameter( value ): + collection_multirun_found = True + else: + multirun_found = True + else: + continue + else: + # Old-style batching (remove someday? - pretty hacky and didn't live in API long) + try_replace_key( key, "|__multirun__" ) or multirun_found + try_replace_key( key, "|__collection_multirun__" ) or collection_multirun_found - if sum( [ 1 if f else 0 for f in [ multirun_found, collection_multirun_found ] ] ) > 1: - # In theory doable, but to complicated for a first pass. - message = "Cannot specify parallel execution across both multiple datasets and dataset collections." - raise exceptions.ToolMetaParameterException( message ) + expanded_incomings = permutations.expand_multi_inputs( incoming_template, classifier ) + if collections_to_match.has_collections(): + collection_info = trans.app.dataset_collections_service.match_collections( collections_to_match ) + else: + collection_info = None + return expanded_incomings, collection_info - if multirun_found: - return permutations.expand_multi_inputs( incoming_template, classifier ), None + +def __expand_collection_parameter( trans, input_key, incoming_val, collections_to_match ): + # If subcollectin multirun of data_collection param - value will + # be "hdca_id|subcollection_type" else it will just be hdca_id + if "|" in incoming_val: + encoded_hdc_id, subcollection_type = incoming_val.split( "|", 1 ) else: - expanded_incomings = permutations.expand_multi_inputs( incoming_template, collection_classifier ) - if collections_to_match.has_collections(): - collection_info = trans.app.dataset_collections_service.match_collections( collections_to_match ) - else: - collection_info = None - return expanded_incomings, collection_info + try: + src = incoming_val[ "src" ] + if src != "hdca": + raise exceptions.ToolMetaParameterException( "Invalid dataset collection source type %s" % src ) + encoded_hdc_id = incoming_val[ "id" ] + subcollection_type = incoming_val.get( 'map_over_type', None ) + except TypeError: + encoded_hdc_id = incoming_val + subcollection_type = None + hdc_id = trans.app.security.decode_id( encoded_hdc_id ) + hdc = trans.sa_session.query( model.HistoryDatasetCollectionAssociation ).get( hdc_id ) + collections_to_match.add( input_key, hdc, subcollection_type=subcollection_type ) + if subcollection_type is not None: + from galaxy.dataset_collections import subcollections + subcollection_elements = subcollections.split_dataset_collection_instance( hdc, subcollection_type ) + return subcollection_elements + else: + hdas = hdc.collection.dataset_instances + return hdas + + +def __collection_multirun_parameter( value ): + batch_values = util.listify( value[ 'values' ] ) + if len( batch_values ) == 1: + batch_over = batch_values[ 0 ] + if isinstance( batch_over, dict ) and ('src' in batch_over) and (batch_over[ 'src' ] == 'hdca'): + return True + return False diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b test/api/test_tools.py --- a/test/api/test_tools.py +++ b/test/api/test_tools.py @@ -109,6 +109,35 @@ output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) self.assertEqual( output1_content.strip(), "Cat1Test" ) + @skip_without_tool( "cat1" ) + def test_run_cat1_listified_param( self ): + # Run simple non-upload tool with an input data parameter. + history_id = self.dataset_populator.new_history() + new_dataset = self.dataset_populator.new_dataset( history_id, content='Cat1Testlistified' ) + inputs = dict( + input1=[dataset_to_param( new_dataset )], + ) + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 1 ) + output1 = outputs[ 0 ] + output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) + self.assertEqual( output1_content.strip(), "Cat1Testlistified" ) + + @skip_without_tool( "cat1" ) + def test_run_cat1_single_meta_wrapper( self ): + # Wrap input in a no-op meta parameter wrapper like Sam is planning to + # use for all UI API submissions. + history_id = self.dataset_populator.new_history() + new_dataset = self.dataset_populator.new_dataset( history_id, content='123' ) + inputs = dict( + input1={ 'batch': False, 'values': [ dataset_to_param( new_dataset ) ] }, + ) + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 1 ) + output1 = outputs[ 0 ] + output1_content = self.dataset_populator.get_history_dataset_content( history_id, dataset=output1 ) + self.assertEqual( output1_content.strip(), "123" ) + @skip_without_tool( "validation_default" ) def test_validation( self ): history_id = self.dataset_populator.new_history() @@ -118,6 +147,20 @@ response = self._run( "validation_default", history_id, inputs ) self._assert_status_code_is( response, 400 ) + @skip_without_tool( "collection_paired_test" ) + def test_collection_parameter( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) + inputs = { + "f1": { "src": "hdca", "id": hdca_id }, + } + output = self._run( "collection_paired_test", history_id, inputs, assert_ok=True ) + assert len( output[ 'jobs' ] ) == 1 + assert len( output[ 'implicit_collections' ] ) == 0 + assert len( output[ 'outputs' ] ) == 1 + contents = self.dataset_populator.get_history_dataset_content( history_id, hid=4 ) + assert contents.strip() == "123\n456", contents + @skip_without_tool( "cat1" ) def test_run_cat1_with_two_inputs( self ): # Run tool with an multiple data parameter and grouping (repeat) @@ -135,16 +178,31 @@ self.assertEqual( output1_content.strip(), "Cat1Test\nCat2Test" ) @skip_without_tool( "cat1" ) + def test_multirun_cat1_legacy( self ): + history_id, datasets = self._prepare_cat1_multirun() + inputs = { + "input1|__multirun__": datasets, + } + self._check_cat1_multirun( history_id, inputs ) + + @skip_without_tool( "cat1" ) def test_multirun_cat1( self ): + history_id, datasets = self._prepare_cat1_multirun() + inputs = { + "input1": { + 'batch': True, + 'values': datasets, + }, + } + self._check_cat1_multirun( history_id, inputs ) + + def _prepare_cat1_multirun( self ): history_id = self.dataset_populator.new_history() new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) - inputs = { - "input1|__multirun__": [ - dataset_to_param( new_dataset1 ), - dataset_to_param( new_dataset2 ), - ], - } + return history_id, [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ] + + def _check_cat1_multirun( self, history_id, inputs ): outputs = self._cat1_outputs( history_id, inputs=inputs ) self.assertEquals( len( outputs ), 2 ) output1 = outputs[ 0 ] @@ -154,6 +212,20 @@ self.assertEquals( output1_content.strip(), "123" ) self.assertEquals( output2_content.strip(), "456" ) + @skip_without_tool( "random_lines1" ) + def test_multirun_non_data_parameter( self ): + history_id = self.dataset_populator.new_history() + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123\n456\n789' ) + inputs = { + 'input': dataset_to_param( new_dataset1 ), + 'num_lines': { 'batch': True, 'values': [ 1, 2, 3 ] } + } + outputs = self._run_and_get_outputs( 'random_lines1', history_id, inputs ) + # Assert we have three outputs with 1, 2, and 3 lines respectively. + assert len( outputs ) == 3 + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + assert sorted( map( lambda c: len( c.split( "\n" ) ), outputs_contents ) ) == [ 1, 2, 3 ] + @skip_without_tool( "cat1" ) def test_multirun_in_repeat( self ): history_id = self.dataset_populator.new_history() @@ -177,35 +249,60 @@ self.assertEquals( output2_content.strip(), "Common\n456" ) @skip_without_tool( "cat1" ) - def test_multirun_on_multiple_inputs( self ): - history_id = self.dataset_populator.new_history() - new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) - new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) - new_dataset3 = self.dataset_populator.new_dataset( history_id, content='789' ) - new_dataset4 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + def test_multirun_on_multiple_inputs_legacy( self ): + history_id, first_two, second_two = self._setup_two_multiruns() inputs = { - "input1|__multirun__": [ - dataset_to_param( new_dataset1 ), - dataset_to_param( new_dataset2 ), - ], - 'queries_0|input2|__multirun__': [ - dataset_to_param( new_dataset3 ), - dataset_to_param( new_dataset4 ), - ], + "input1|__multirun__": first_two, + 'queries_0|input2|__multirun__': second_two, } outputs = self._cat1_outputs( history_id, inputs=inputs ) self.assertEquals( len( outputs ), 2 ) outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] assert "123\n789" in outputs_contents assert "456\n0ab" in outputs_contents - # TODO: Once cross production (instead of linking inputs) is an option - # again redo test with these checks... - # self.assertEquals( len( outputs ), 4 ) - # assert "123\n0ab" in outputs_contents - # assert "456\n789" in outputs_contents @skip_without_tool( "cat1" ) - def test_map_over_collection( self ): + def test_multirun_on_multiple_inputs( self ): + history_id, first_two, second_two = self._setup_two_multiruns() + inputs = { + "input1": { 'batch': True, 'values': first_two }, + 'queries_0|input2': { 'batch': True, 'values': second_two }, + } + outputs = self._cat1_outputs( history_id, inputs=inputs ) + self.assertEquals( len( outputs ), 2 ) + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + assert "123\n789" in outputs_contents + assert "456\n0ab" in outputs_contents + + @skip_without_tool( "cat1" ) + def test_multirun_on_multiple_inputs_unlinked( self ): + history_id, first_two, second_two = self._setup_two_multiruns() + inputs = { + "input1": { 'batch': True, 'linked': False, 'values': first_two }, + 'queries_0|input2': { 'batch': True, 'linked': False, 'values': second_two }, + } + outputs = self._cat1_outputs( history_id, inputs=inputs ) + outputs_contents = [ self.dataset_populator.get_history_dataset_content( history_id, dataset=o ).strip() for o in outputs ] + self.assertEquals( len( outputs ), 4 ) + assert "123\n789" in outputs_contents + assert "456\n0ab" in outputs_contents + assert "123\n0ab" in outputs_contents + assert "456\n789" in outputs_contents + + def _setup_two_multiruns( self ): + history_id = self.dataset_populator.new_history() + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='123' ) + new_dataset2 = self.dataset_populator.new_dataset( history_id, content='456' ) + new_dataset3 = self.dataset_populator.new_dataset( history_id, content='789' ) + new_dataset4 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + return ( + history_id, + [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ], + [ dataset_to_param( new_dataset3 ), dataset_to_param( new_dataset4 ) ] + ) + + @skip_without_tool( "cat1" ) + def test_map_over_collection_legacy( self ): history_id = self.dataset_populator.new_history() hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) inputs = { @@ -214,6 +311,18 @@ # first, next test method tests other. "input1|__collection_multirun__": hdca_id, } + self._run_and_check_simple_collection_mapping( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_collection( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_pair( history_id, [ "123", "456" ] ) + inputs = { + "input1": { 'batch': True, 'values': [ { 'src': 'hdca', 'id': hdca_id } ] }, + } + self._run_and_check_simple_collection_mapping( history_id, inputs ) + + def _run_and_check_simple_collection_mapping( self, history_id, inputs ): create = self._run_cat1( history_id, inputs=inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -229,12 +338,24 @@ self.assertEquals( output2_content.strip(), "456" ) @skip_without_tool( "cat1" ) - def test_map_over_nested_collections( self ): + def test_map_over_nested_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca_id = self.__build_nested_list( history_id ) inputs = { "input1|__collection_multirun__": dict( src="hdca", id=hdca_id ), } + self._check_simple_cat1_over_nested_collections( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_nested_collections( self ): + history_id = self.dataset_populator.new_history() + hdca_id = self.__build_nested_list( history_id ) + inputs = { + "input1": { 'batch': True, 'values': [ dict( src="hdca", id=hdca_id ) ] }, + } + self._check_simple_cat1_over_nested_collections( history_id, inputs ) + + def _check_simple_cat1_over_nested_collections( self, history_id, inputs ): create = self._run_cat1( history_id, inputs=inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -257,7 +378,7 @@ self.assertEquals( outputs[ 0 ][ "id" ], first_object_forward_element[ "object" ][ "id" ] ) @skip_without_tool( "cat1" ) - def test_map_over_two_collections( self ): + def test_map_over_two_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) @@ -265,7 +386,24 @@ "input1|__collection_multirun__": hdca1_id, "queries_0|input2|__collection_multirun__": hdca2_id, } - outputs = self._cat1_outputs( history_id, inputs=inputs ) + self._check_map_cat1_over_two_collections( history_id, inputs ) + + @skip_without_tool( "cat1" ) + def test_map_over_two_collections( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) + inputs = { + "input1": { 'batch': True, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'values': [ { 'src': 'hdca', 'id': hdca2_id } ] }, + } + self._check_map_cat1_over_two_collections( history_id, inputs ) + + def _check_map_cat1_over_two_collections( self, history_id, inputs ): + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] self.assertEquals( len( outputs ), 2 ) output1 = outputs[ 0 ] output2 = outputs[ 1 ] @@ -274,6 +412,49 @@ self.assertEquals( output1_content.strip(), "123\n789" ) self.assertEquals( output2_content.strip(), "456\n0ab" ) + self.assertEquals( len( response_object[ 'jobs' ] ), 2 ) + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 1 ) + + @skip_without_tool( "cat1" ) + def test_map_over_two_collections_unlinked( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.__build_pair( history_id, [ "789", "0ab" ] ) + inputs = { + "input1": { 'batch': True, 'linked': False, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'linked': False, 'values': [ { 'src': 'hdca', 'id': hdca2_id } ] }, + } + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] + self.assertEquals( len( outputs ), 4 ) + + self.assertEquals( len( response_object[ 'jobs' ] ), 4 ) + # Implicit collections not created with unlinked inputs yet - this may + # be problematic. + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 0 ) + + @skip_without_tool( "cat1" ) + def test_map_over_collected_and_individual_datasets( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + new_dataset1 = self.dataset_populator.new_dataset( history_id, content='789' ) + new_dataset2 = self.dataset_populator.new_dataset( history_id, content='0ab' ) + + inputs = { + "input1": { 'batch': True, 'values': [ {'src': 'hdca', 'id': hdca1_id } ] }, + "queries_0|input2": { 'batch': True, 'values': [ dataset_to_param( new_dataset1 ), dataset_to_param( new_dataset2 ) ] }, + } + response = self._run_cat1( history_id, inputs ) + self._assert_status_code_is( response, 200 ) + response_object = response.json() + outputs = response_object[ 'outputs' ] + self.assertEquals( len( outputs ), 2 ) + + self.assertEquals( len( response_object[ 'jobs' ] ), 2 ) + self.assertEquals( len( response_object[ 'implicit_collections' ] ), 1 ) + @skip_without_tool( "cat1" ) def test_cannot_map_over_incompatible_collections( self ): history_id = self.dataset_populator.new_history() @@ -289,7 +470,7 @@ assert run_response.status_code >= 400 @skip_without_tool( "multi_data_param" ) - def test_reduce_collections( self ): + def test_reduce_collections_legacy( self ): history_id = self.dataset_populator.new_history() hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) hdca2_id = self.dataset_collection_populator.create_list_in_history( history_id ).json()[ "id" ] @@ -297,6 +478,20 @@ "f1": "__collection_reduce__|%s" % hdca1_id, "f2": "__collection_reduce__|%s" % hdca2_id, } + self._check_simple_reduce_job( history_id, inputs ) + + @skip_without_tool( "multi_data_param" ) + def test_reduce_collections( self ): + history_id = self.dataset_populator.new_history() + hdca1_id = self.__build_pair( history_id, [ "123", "456" ] ) + hdca2_id = self.dataset_collection_populator.create_list_in_history( history_id ).json()[ "id" ] + inputs = { + "f1": { 'src': 'hdca', 'id': hdca1_id }, + "f2": { 'src': 'hdca', 'id': hdca2_id }, + } + self._check_simple_reduce_job( history_id, inputs ) + + def _check_simple_reduce_job( self, history_id, inputs ): create = self._run( "multi_data_param", history_id, inputs, assert_ok=True ) outputs = create[ 'outputs' ] jobs = create[ 'jobs' ] @@ -310,12 +505,27 @@ assert len( output2_content.strip().split("\n") ) == 3, output2_content @skip_without_tool( "collection_paired_test" ) - def test_subcollection_mapping( self ): + def test_subcollection_mapping_legacy( self ): history_id = self.dataset_populator.new_history() hdca_list_id = self.__build_nested_list( history_id ) inputs = { "f1|__collection_multirun__": "%s|paired" % hdca_list_id } + self._check_simple_subcollection_mapping( history_id, inputs ) + + @skip_without_tool( "collection_paired_test" ) + def test_subcollection_mapping( self ): + history_id = self.dataset_populator.new_history() + hdca_list_id = self.__build_nested_list( history_id ) + inputs = { + "f1": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'map_over_type': 'paired', 'id': hdca_list_id }], + } + } + self._check_simple_subcollection_mapping( history_id, inputs ) + + def _check_simple_subcollection_mapping( self, history_id, inputs ): # Following wait not really needed - just getting so many database # locked errors with sqlite. self.dataset_populator.wait_for_history( history_id, assert_ok=True ) @@ -329,7 +539,7 @@ assert output2_content.strip() == "789\n0ab", output2_content @skip_without_tool( "collection_mixed_param" ) - def test_combined_mapping_and_subcollection_mapping( self ): + def test_combined_mapping_and_subcollection_mapping_legacy( self ): history_id = self.dataset_populator.new_history() nested_list_id = self.__build_nested_list( history_id ) create_response = self.dataset_collection_populator.create_list_in_history( history_id, contents=["xxx", "yyy"] ) @@ -338,6 +548,27 @@ "f1|__collection_multirun__": "%s|paired" % nested_list_id, "f2|__collection_multirun__": list_id, } + self._check_combined_mapping_and_subcollection_mapping( history_id, inputs ) + + @skip_without_tool( "collection_mixed_param" ) + def test_combined_mapping_and_subcollection_mapping( self ): + history_id = self.dataset_populator.new_history() + nested_list_id = self.__build_nested_list( history_id ) + create_response = self.dataset_collection_populator.create_list_in_history( history_id, contents=["xxx", "yyy"] ) + list_id = create_response.json()[ "id" ] + inputs = { + "f1": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'map_over_type': 'paired', 'id': nested_list_id }], + }, + "f2": { + 'batch': True, + 'values': [ { 'src': 'hdca', 'id': list_id }], + }, + } + self._check_combined_mapping_and_subcollection_mapping( history_id, inputs ) + + def _check_combined_mapping_and_subcollection_mapping( self, history_id, inputs ): self.dataset_populator.wait_for_history( history_id, assert_ok=True ) outputs = self._run_and_get_outputs( "collection_mixed_param", history_id, inputs ) assert len( outputs ), 2 diff -r 42f677bab2e4880b62758614fcb2c7dc27013788 -r df92b55d194a4d67a1b863c85044eb91c138c74b test/api/test_workflows.py --- a/test/api/test_workflows.py +++ b/test/api/test_workflows.py @@ -361,13 +361,13 @@ hdca = self.dataset_collection_populator.create_pair_in_history( history_id, contents=["1 2 3\n4 5 6", "7 8 9\n10 11 10"] ).json() hdca_id = hdca[ "id" ] inputs1 = { - "input|__collection_multirun__": hdca_id, + "input": { "batch": True, "values": [ { "src": "hdca", "id": hdca_id } ] }, "num_lines": 2 } implicit_hdca1, job_id1 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs1 ) inputs2 = { - "f1": "__collection_reduce__|%s" % ( implicit_hdca1[ "id" ] ), - "f2": "__collection_reduce__|%s" % ( implicit_hdca1[ "id" ] ) + "f1": { "src": "hdca", "id": implicit_hdca1[ "id" ] }, + "f2": { "src": "hdca", "id": implicit_hdca1[ "id" ] }, } reduction_run_output = self.dataset_populator.run_tool( tool_id="multi_data_param", @@ -413,12 +413,12 @@ hdca = self.dataset_collection_populator.create_pair_in_history( history_id, contents=["1 2 3\n4 5 6", "7 8 9\n10 11 10"] ).json() hdca_id = hdca[ "id" ] inputs1 = { - "input|__collection_multirun__": hdca_id, + "input": { "batch": True, "values": [ { "src": "hdca", "id": hdca_id } ] }, "num_lines": 2 } implicit_hdca1, job_id1 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs1 ) inputs2 = { - "input|__collection_multirun__": implicit_hdca1[ "id" ], + "input": { "batch": True, "values": [ { "src": "hdca", "id": implicit_hdca1[ "id" ] } ] }, "num_lines": 1 } _, job_id2 = self._run_tool_get_collection_and_job_id( history_id, "random_lines1", inputs2 ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: Allow configuration of Galaxy via environment variables.
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/42f677bab2e4/ Changeset: 42f677bab2e4 User: jmchilton Date: 2014-09-22 02:02:09+00:00 Summary: Allow configuration of Galaxy via environment variables. Allow overriding config defaults (those not specified in the configuration file) with GALAXY_CONFIG_<config-name> and allow overriding actual configured values with GALAXY_CONFIG_OVERRIDE_<config-name>. Here config-name is the actual value in galaxy.ini but in upper case. Affected #: 1 file diff -r 5fac4d4b7d7d6b056056bf2a6094c7777dc28a71 -r 42f677bab2e4880b62758614fcb2c7dc27013788 lib/galaxy/config.py --- a/lib/galaxy/config.py +++ b/lib/galaxy/config.py @@ -20,6 +20,9 @@ log = logging.getLogger( __name__ ) +CONFIG_DEFAULT_PREFIX = "GALAXY_CONFIG_" +CONFIG_OVERRIDE_PREFIX = "GALAXY_CONFIG_OVERRIDE_" + def resolve_path( path, root ): """If 'path' is relative make absolute by prepending 'root'""" @@ -36,6 +39,15 @@ deprecated_options = ( 'database_file', ) def __init__( self, **kwargs ): + for key in os.environ: + if key.startswith( CONFIG_OVERRIDE_PREFIX ): + config_key = key[ len( CONFIG_OVERRIDE_PREFIX ): ].lower() + kwargs[ config_key ] = os.environ[ key ] + elif key.startswith( CONFIG_DEFAULT_PREFIX ): + config_key = key[ len( CONFIG_DEFAULT_PREFIX ): ].lower() + if config_key not in kwargs: + kwargs[ config_key ] = os.environ[ key ] + self.config_dict = kwargs self.root = kwargs.get( 'root_dir', '.' ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by commits-noreply＠bitbucket.org 22 Sep '14

22 Sep '14

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/87f7abe35390/ Changeset: 87f7abe35390 User: jmchilton Date: 2014-09-22 01:48:58+00:00 Summary: Reduce code duplication between job metrics and dependency resolution plugins. Same code reused again downstream in workflow schedulers plugin framework. Affected #: 3 files diff -r 6fe929724b066bd8b61ee0149479aba1e4e42f56 -r 87f7abe35390d8822f04e3e03dcc0598b5bf076c lib/galaxy/jobs/metrics/__init__.py --- a/lib/galaxy/jobs/metrics/__init__.py +++ b/lib/galaxy/jobs/metrics/__init__.py @@ -3,7 +3,7 @@ from xml.etree import ElementTree -from galaxy.util.submodules import submodules +from galaxy.util import plugin_config from galaxy import util from ..metrics import formatting @@ -49,17 +49,8 @@ return self.job_instrumenters[ destination_id ].collect_properties( job_id, job_directory ) def __plugins_dict( self ): - plugin_dict = {} - for plugin_module in self.__plugin_modules(): - for clazz in plugin_module.__all__: - plugin_type = getattr( clazz, 'plugin_type', None ) - if plugin_type: - plugin_dict[ plugin_type ] = clazz - return plugin_dict - - def __plugin_modules( self ): import galaxy.jobs.metrics.instrumenters - return submodules( galaxy.jobs.metrics.instrumenters ) + return plugin_config.plugins_dict( galaxy.jobs.metrics.instrumenters, 'plugin_type' ) class NullJobInstrumenter( object ): @@ -117,14 +108,7 @@ return per_plugin_properites def __plugins_for_element( self, plugins_element ): - plugins = [] - for plugin_element in plugins_element.getchildren(): - plugin_type = plugin_element.tag - plugin_kwds = dict( plugin_element.items() ) - plugin_kwds.update( self.extra_kwargs ) - plugin = self.plugin_classes[ plugin_type ]( **plugin_kwds ) - plugins.append( plugin ) - return plugins + return plugin_config.load_plugins_from_element(self.plugin_classes, plugins_element, self.extra_kwargs) @staticmethod def from_file( plugin_classes, conf_file, **kwargs ): diff -r 6fe929724b066bd8b61ee0149479aba1e4e42f56 -r 87f7abe35390d8822f04e3e03dcc0598b5bf076c lib/galaxy/tools/deps/__init__.py --- a/lib/galaxy/tools/deps/__init__.py +++ b/lib/galaxy/tools/deps/__init__.py @@ -12,7 +12,7 @@ from .resolvers import INDETERMINATE_DEPENDENCY from .resolvers.galaxy_packages import GalaxyPackageDependencyResolver from .resolvers.tool_shed_packages import ToolShedPackageDependencyResolver -from galaxy.util.submodules import submodules +from galaxy.util import plugin_config def build_dependency_manager( config ): @@ -110,24 +110,9 @@ :param root: Object representing the root ``<dependency_resolvers>`` object in the file. :type root: ``xml.etree.ElementTree.Element`` """ - resolvers = [] - resolvers_element = root - for resolver_element in resolvers_element.getchildren(): - resolver_type = resolver_element.tag - resolver_kwds = dict(resolver_element.items()) - resolver = self.resolver_classes[resolver_type](self, **resolver_kwds) - resolvers.append(resolver) - return resolvers + extra_kwds = dict( dependency_manager=self ) + return plugin_config.load_plugins_from_element( self.resolver_classes, root, extra_kwds ) def __resolvers_dict( self ): - resolver_dict = {} - for resolver_module in self.__resolver_modules(): - for clazz in resolver_module.__all__: - resolver_type = getattr(clazz, 'resolver_type', None) - if resolver_type: - resolver_dict[resolver_type] = clazz - return resolver_dict - - def __resolver_modules( self ): import galaxy.tools.deps.resolvers - return submodules( galaxy.tools.deps.resolvers ) + return plugin_config.plugins_dict( galaxy.tools.deps.resolvers, 'resolver_type' ) diff -r 6fe929724b066bd8b61ee0149479aba1e4e42f56 -r 87f7abe35390d8822f04e3e03dcc0598b5bf076c lib/galaxy/util/plugin_config.py --- /dev/null +++ b/lib/galaxy/util/plugin_config.py @@ -0,0 +1,32 @@ +from galaxy.util.submodules import submodules + + +def plugins_dict(module, plugin_type_identifier): + """ Walk through all classes in submodules of module and find ones labelled + with specified plugin_type_identifier and throw in a dictionary to allow + constructions from plugins by these types later on. + """ + plugin_dict = {} + + for plugin_module in submodules( module ): + # FIXME: this is not how one is suppose to use __all__ why did you do + # this past John? + for clazz in plugin_module.__all__: + plugin_type = getattr( clazz, plugin_type_identifier, None ) + if plugin_type: + plugin_dict[ plugin_type ] = clazz + + return plugin_dict + + +def load_plugins_from_element(plugins_dict, plugins_element, extra_kwds={}): + plugins = [] + + for plugin_element in plugins_element.getchildren(): + plugin_type = plugin_element.tag + plugin_kwds = dict( plugin_element.items() ) + plugin_kwds.update( extra_kwds ) + plugin = plugins_dict[ plugin_type ]( **plugin_kwds ) + plugins.append( plugin ) + + return plugins https://bitbucket.org/galaxy/galaxy-central/commits/5fac4d4b7d7d/ Changeset: 5fac4d4b7d7d User: jmchilton Date: 2014-09-22 01:48:58+00:00 Summary: Allow loading dependency resolvers and job metrics from YAML instead of XML. Mostly for downstream Pulsar configuration improvements - but I don't see a reason not to allow this in Galaxy as well. Besides simply being more pleasant to write - having everything configurable by Python dictionary should enable easier automation and programatic configuration of these job running systems. Affected #: 3 files diff -r 87f7abe35390d8822f04e3e03dcc0598b5bf076c -r 5fac4d4b7d7d6b056056bf2a6094c7777dc28a71 lib/galaxy/jobs/metrics/__init__.py --- a/lib/galaxy/jobs/metrics/__init__.py +++ b/lib/galaxy/jobs/metrics/__init__.py @@ -1,8 +1,6 @@ import collections import os -from xml.etree import ElementTree - from galaxy.util import plugin_config from galaxy import util @@ -37,7 +35,7 @@ self.set_destination_instrumenter( destination_id, instrumenter ) def set_destination_conf_element( self, destination_id, element ): - instrumenter = JobInstrumenter( self.plugin_classes, element ) + instrumenter = JobInstrumenter( self.plugin_classes, ('xml', element) ) self.set_destination_instrumenter( destination_id, instrumenter ) def set_destination_instrumenter( self, destination_id, job_instrumenter=None ): @@ -69,10 +67,10 @@ class JobInstrumenter( object ): - def __init__( self, plugin_classes, metrics_element, **kwargs ): + def __init__( self, plugin_classes, plugins_source, **kwargs ): self.extra_kwargs = kwargs self.plugin_classes = plugin_classes - self.plugins = self.__plugins_for_element( metrics_element ) + self.plugins = self.__plugins_from_source( plugins_source ) def pre_execute_commands( self, job_directory ): commands = [] @@ -107,12 +105,12 @@ log.exception( "Failed to collect job properties for plugin %s" % plugin ) return per_plugin_properites - def __plugins_for_element( self, plugins_element ): - return plugin_config.load_plugins_from_element(self.plugin_classes, plugins_element, self.extra_kwargs) + def __plugins_from_source( self, plugins_source ): + return plugin_config.load_plugins(self.plugin_classes, plugins_source, self.extra_kwargs) @staticmethod def from_file( plugin_classes, conf_file, **kwargs ): if not conf_file or not os.path.exists( conf_file ): return NULL_JOB_INSTRUMENTER - plugins_element = ElementTree.parse( conf_file ).getroot() - return JobInstrumenter( plugin_classes, plugins_element, **kwargs ) + plugins_source = plugin_config.plugin_source_from_path( conf_file ) + return JobInstrumenter( plugin_classes, plugins_source, **kwargs ) diff -r 87f7abe35390d8822f04e3e03dcc0598b5bf076c -r 5fac4d4b7d7d6b056056bf2a6094c7777dc28a71 lib/galaxy/tools/deps/__init__.py --- a/lib/galaxy/tools/deps/__init__.py +++ b/lib/galaxy/tools/deps/__init__.py @@ -7,8 +7,6 @@ import logging log = logging.getLogger( __name__ ) -from xml.etree import ElementTree - from .resolvers import INDETERMINATE_DEPENDENCY from .resolvers.galaxy_packages import GalaxyPackageDependencyResolver from .resolvers.tool_shed_packages import ToolShedPackageDependencyResolver @@ -94,8 +92,8 @@ def __build_dependency_resolvers( self, conf_file ): if not conf_file or not os.path.exists( conf_file ): return self.__default_dependency_resolvers() - root = ElementTree.parse( conf_file ).getroot() - return self.__parse_resolver_conf_xml( root ) + plugin_source = plugin_config.plugin_source_from_path( conf_file ) + return self.__parse_resolver_conf_xml( plugin_source ) def __default_dependency_resolvers( self ): return [ @@ -104,14 +102,11 @@ GalaxyPackageDependencyResolver(self, versionless=True), ] - def __parse_resolver_conf_xml(self, root): + def __parse_resolver_conf_xml(self, plugin_source): """ - - :param root: Object representing the root ``<dependency_resolvers>`` object in the file. - :type root: ``xml.etree.ElementTree.Element`` """ extra_kwds = dict( dependency_manager=self ) - return plugin_config.load_plugins_from_element( self.resolver_classes, root, extra_kwds ) + return plugin_config.load_plugins( self.resolver_classes, plugin_source, extra_kwds ) def __resolvers_dict( self ): import galaxy.tools.deps.resolvers diff -r 87f7abe35390d8822f04e3e03dcc0598b5bf076c -r 5fac4d4b7d7d6b056056bf2a6094c7777dc28a71 lib/galaxy/util/plugin_config.py --- a/lib/galaxy/util/plugin_config.py +++ b/lib/galaxy/util/plugin_config.py @@ -1,3 +1,16 @@ +from xml.etree import ElementTree + +try: + from galaxy import eggs + eggs.require('PyYAML') +except Exception: + # If not in Galaxy, ignore this. + pass +try: + import yaml +except ImportError: + yaml = None + from galaxy.util.submodules import submodules @@ -19,7 +32,15 @@ return plugin_dict -def load_plugins_from_element(plugins_dict, plugins_element, extra_kwds={}): +def load_plugins(plugins_dict, plugin_source, extra_kwds={}): + source_type, source = plugin_source + if source_type == "xml": + return __load_plugins_from_element(plugins_dict, source, extra_kwds) + else: + return __load_plugins_from_dicts(plugins_dict, source, extra_kwds) + + +def __load_plugins_from_element(plugins_dict, plugins_element, extra_kwds): plugins = [] for plugin_element in plugins_element.getchildren(): @@ -30,3 +51,31 @@ plugins.append( plugin ) return plugins + + +def __load_plugins_from_dicts(plugins_dict, configs, extra_kwds): + plugins = [] + + for config in configs: + plugin_type = config[ "type" ] + plugin_kwds = config + plugin_kwds.update( extra_kwds ) + plugin = plugins_dict[ plugin_type ]( **plugin_kwds ) + plugins.append( plugin ) + + return plugins + + +def plugin_source_from_path(path): + if path.endswith(".yaml") or path.endswith(".yml"): + return ('dict', __read_yaml(path)) + else: + return ('xml', ElementTree.parse( path ).getroot()) + + +def __read_yaml(path): + if yaml is None: + raise ImportError("Attempting to read YAML configuration file - but PyYAML dependency unavailable.") + + with open(path, "rb") as f: + return yaml.load(f) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

[galaxyproject/usegalaxy-playbook] d2e65c: Remove downtime message from Test and Main
by GitHub 20 Sep '14

20 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: d2e65c460baf48e0cc9e690541c8a5d8cfbf99c2 https://github.com/galaxyproject/usegalaxy-playbook/commit/d2e65c460baf48e0… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-20 (Sat, 20 Sep 2014) Changed paths: M production/group_vars/galaxyservers.yml M stage/group_vars/galaxyservers.yml Log Message: ----------- Remove downtime message from Test and Main

1 0

[galaxyproject/usegalaxy-playbook] 8d30db: Update Test/Main Pulsar and set new retry options ...
by GitHub 20 Sep '14

20 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: 8d30dbcb984e82bb4f1fc3d4f60f2e5bf1ed0363 https://github.com/galaxyproject/usegalaxy-playbook/commit/8d30dbcb984e82bb… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-19 (Fri, 19 Sep 2014) Changed paths: M production/group_vars/pulsarservers.yml M production/host_vars/login5.stampede.tacc.utexas.edu.yml M stage/group_vars/all.yml M stage/host_vars/login5.stampede.tacc.utexas.edu.yml Log Message: ----------- Update Test/Main Pulsar and set new retry options for transfers. Thanks @jmchilton.

1 0

[galaxyproject/usegalaxy-playbook] 23f869: Update Test (did this earlier, forgot to commit/pu...
by GitHub 20 Sep '14

20 Sep '14

Branch: refs/heads/master Home: https://github.com/galaxyproject/usegalaxy-playbook Commit: 23f869e97e75e93a1a8805f903688790391a1e14 https://github.com/galaxyproject/usegalaxy-playbook/commit/23f869e97e75e93a… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-19 (Fri, 19 Sep 2014) Changed paths: M stage/group_vars/all.yml Log Message: ----------- Update Test (did this earlier, forgot to commit/push). Commit: 53a0f63a3576fc40f630d1dc390d6d72eb52b722 https://github.com/galaxyproject/usegalaxy-playbook/commit/53a0f63a3576fc40… Author: Nate Coraor <nate(a)bx.psu.edu> Date: 2014-09-19 (Fri, 19 Sep 2014) Changed paths: M production/group_vars/pulsarservers.yml M stage/group_vars/pulsarservers.yml Log Message: ----------- Update Test/Main Pulsar (ditto). Compare: https://github.com/galaxyproject/usegalaxy-playbook/compare/7b729c1a8fac...…

1 0