- galaxy-commits - lists.galaxyproject.org

commit/galaxy-central: clements: Updated license to Academic Free License v3.0.
by Bitbucket 10 Jan '13

10 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/674273be9a1f/ changeset: 674273be9a1f user: clements date: 2013-01-10 22:22:08 summary: Updated license to Academic Free License v3.0. affected #: 1 file diff -r 58851b49011218dbb11370392bbc03f9b1e69a9f -r 674273be9a1ffba71ce1546= 2a9f5efd5569503a7 LICENSE.txt --- a/LICENSE.txt +++ b/LICENSE.txt @@ -1,25 +1,57 @@ -Copyright (c) 2005 Pennsylvania State University +Copyright (c) 2005-2013 Pennsylvania State University =20 -Permission is hereby granted, free of charge, to any person obtaining=20 -a copy of this software and associated documentation files (the=20 -"Software"), to deal in the Software without restriction, including=20 -without limitation the rights to use, copy, modify, merge, publish,=20 -distribute, sublicense, and/or sell copies of the Software, and to=20 -permit persons to whom the Software is furnished to do so, subject to=20 -the following conditions: +Licensed under the Academic Free License version 3.0 =20 -The above copyright notice and this permission notice shall be=20 -included in all copies or substantial portions of the Software. +1) Grant of Copyright License. Licensor grants You a worldwide, royalty-fr= ee, non-exclusive, sublicensable license, for the duration of the copyright= , to do the following: =20 -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,=20 -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF=20 -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.=20 -IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY=20 -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,=20 -TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE=20 -SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.=20 +a) to reproduce the Original Work in copies, either alone or as part of a = collective work; + +b) to translate, adapt, alter, transform, modify, or arrange the Original = Work, thereby creating derivative works ("Derivative Works") based upon the= Original Work; + +c) to distribute or communicate copies of the Original Work and Derivative= Works to the public, under any license of your choice that does not contra= dict the terms and conditions, including Licensor's reserved rights and rem= edies, in this Academic Free License; + +d) to perform the Original Work publicly; and + +e) to display the Original Work publicly. + +2) Grant of Patent License. Licensor grants You a worldwide, royalty-free,= non-exclusive, sublicensable license, under patent claims owned or control= led by the Licensor that are embodied in the Original Work as furnished by = the Licensor, for the duration of the patents, to make, use, sell, offer fo= r sale, have made, and import the Original Work and Derivative Works. + +3) Grant of Source Code License. The term "Source Code" means the preferre= d form of the Original Work for making modifications to it and all availabl= e documentation describing how to modify the Original Work. Licensor agrees= to provide a machine-readable copy of the Source Code of the Original Work= along with each copy of the Original Work that Licensor distributes. Licen= sor reserves the right to satisfy this obligation by placing a machine-read= able copy of the Source Code in an information repository reasonably calcul= ated to permit inexpensive and convenient access by You for as long as Lice= nsor continues to distribute the Original Work. + +4) Exclusions From License Grant. Neither the names of Licensor, nor the n= ames of any contributors to the Original Work, nor any of their trademarks = or service marks, may be used to endorse or promote products derived from t= his Original Work without express prior permission of the Licensor. Except = as expressly stated herein, nothing in this License grants any license to L= icensor's trademarks, copyrights, patents, trade secrets or any other intel= lectual property. No patent license is granted to make, use, sell, offer fo= r sale, have made, or import embodiments of any patent claims other than th= e licensed claims defined in Section 2. No license is granted to the tradem= arks of Licensor even if such marks are included in the Original Work. Noth= ing in this License shall be interpreted to prohibit Licensor from licensin= g under terms different from this License any Original Work that Licensor o= therwise would have a right to license. + +5) External Deployment. The term "External Deployment" means the use, dist= ribution, or communication of the Original Work or Derivative Works in any = way such that the Original Work or Derivative Works may be used by anyone o= ther than You, whether those works are distributed or communicated to those= persons or made available as an application intended for use over a networ= k. As an express condition for the grants of license hereunder, You must tr= eat any External Deployment by You of the Original Work or a Derivative Wor= k as a distribution under section 1(c). + +6) Attribution Rights. You must retain, in the Source Code of any Derivati= ve Works that You create, all copyright, patent, or trademark notices from = the Source Code of the Original Work, as well as any notices of licensing a= nd any descriptive text identified therein as an "Attribution Notice." You = must cause the Source Code for any Derivative Works that You create to carr= y a prominent Attribution Notice reasonably calculated to inform recipients= that You have modified the Original Work. + +7) Warranty of Provenance and Disclaimer of Warranty. Licensor warrants th= at the copyright in and to the Original Work and the patent rights granted = herein by Licensor are owned by the Licensor or are sublicensed to You unde= r the terms of this License with the permission of the contributor(s) of th= ose copyrights and patent rights. Except as expressly stated in the immedia= tely preceding sentence, the Original Work is provided under this License o= n an "AS IS" BASIS and WITHOUT WARRANTY, either express or implied, includi= ng, without limitation, the warranties of non-infringement, merchantability= or fitness for a particular purpose. THE ENTIRE RISK AS TO THE QUALITY OF = THE ORIGINAL WORK IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an e= ssential part of this License. No license to the Original Work is granted b= y this License except under this disclaimer. + +8) Limitation of Liability. Under no circumstances and under no legal theo= ry, whether in tort (including negligence), contract, or otherwise, shall t= he Licensor be liable to anyone for any indirect, special, incidental, or c= onsequential damages of any character arising as a result of this License o= r the use of the Original Work including, without limitation, damages for l= oss of goodwill, work stoppage, computer failure or malfunction, or any and= all other commercial damages or losses. This limitation of liability shall= not apply to the extent applicable law prohibits such limitation. + +9) Acceptance and Termination. If, at any time, You expressly assented to = this License, that assent indicates your clear and irrevocable acceptance o= f this License and all of its terms and conditions. If You distribute or co= mmunicate copies of the Original Work or a Derivative Work, You must make a= reasonable effort under the circumstances to obtain the express assent of = recipients to the terms of this License. This License conditions your right= s to undertake the activities listed in Section 1, including your right to = create Derivative Works based upon the Original Work, and doing so without = honoring these terms and conditions is prohibited by copyright law and inte= rnational treaty. Nothing in this License is intended to affect copyright e= xceptions and limitations (including "fair use" or "fair dealing"). This Li= cense shall terminate immediately and You may no longer exercise any of the= rights granted to You by this License upon your failure to honor the condi= tions in Section 1(c). + +10) Termination for Patent Action. This License shall terminate automatica= lly and You may no longer exercise any of the rights granted to You by this= License as of the date You commence an action, including a cross-claim or = counterclaim, against Licensor or any licensee alleging that the Original W= ork infringes a patent. This termination provision shall not apply for an a= ction alleging patent infringement by combinations of the Original Work wit= h other software or hardware. + +11) Jurisdiction, Venue and Governing Law. Any action or suit relating to = this License may be brought only in the courts of a jurisdiction wherein th= e Licensor resides or in which Licensor conducts its primary business, and = under the laws of that jurisdiction excluding its conflict-of-law provision= s. The application of the United Nations Convention on Contracts for the In= ternational Sale of Goods is expressly excluded. Any use of the Original Wo= rk outside the scope of this License or after its termination shall be subj= ect to the requirements and penalties of copyright or patent law in the app= ropriate jurisdiction. This section shall survive the termination of this L= icense. + +12) Attorneys' Fees. In any action to enforce the terms of this License or= seeking damages relating thereto, the prevailing party shall be entitled t= o recover its costs and expenses, including, without limitation, reasonable= attorneys' fees and costs incurred in connection with such action, includi= ng any appeal of such action. This section shall survive the termination of= this License. + +13) Miscellaneous. If any provision of this License is held to be unenforc= eable, such provision shall be reformed only to the extent necessary to mak= e it enforceable. + +14) Definition of "You" in This License. "You" throughout this License, wh= ether in upper or lower case, means an individual or a legal entity exercis= ing rights under, and complying with all of the terms of, this License. For= legal entities, "You" includes any entity that controls, is controlled by,= or is under common control with you. For purposes of this definition, "con= trol" means (i) the power, direct or indirect, to cause the direction or ma= nagement of such entity, whether by contract or otherwise, or (ii) ownershi= p of fifty percent (50%) or more of the outstanding shares, or (iii) benefi= cial ownership of such entity. + +15) Right to Use. You may use the Original Work in all ways not otherwise = restricted or conditioned by this License or by law, and Licensor promises = not to interfere with or be responsible for such uses by You. + +16) Modification of This License. This License is Copyright =C2=A9 2005 La= wrence Rosen. Permission is granted to copy, distribute, or communicate thi= s License without modification. Nothing in this License permits You to modi= fy this License as applied to the Original Work or to Derivative Works. How= ever, You may modify the text of this License and copy, distribute or commu= nicate your modified version (the "Modified License") and apply it to other= original works of authorship subject to the following conditions: (i) You = may not indicate in any way that your Modified License is the "Academic Fre= e License" or "AFL" and you may not use those names in the name of your Mod= ified License; (ii) You must replace the notice specified in the first para= graph above with the notice "Licensed under <insert your license name here>= " or with a notice of your own that is not confusingly similar to the notic= e in this License; and (iii) You may not claim that your original works are= open source software unless your Modified License has been approved by Ope= n Source Initiative (OSI) and You comply with its license review and certif= ication process. + +A brief explanation of this license is available: http://www.rosenlaw.com/= OSL3.0-explained.pdf =20 Some icons found in Galaxy are from the Silk Icons set, available under the Creative Commons Attribution 2.5 License, from: =20 http://www.famfamfam.com/lab/icons/silk/ + + +Other images and documentation are licensed under the Creative Commons Att= ribution 3.0 (CC BY 3.0) License. See=20 + +http://creativecommons.org/licenses/by/3.0/ Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by Bitbucket 10 Jan '13

10 Jan '13

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/a4113cc1cb5e/ changeset: a4113cc1cb5e user: dan date: 2013-01-10 20:57:37 summary: Fixes for Tool.check_and_update_param_values_helper() to check that the type of value provided is valid for the input parameter currently declared. Fixes an issue where rurun would select the wrong input dataset. affected #: 6 files diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r a4113cc1cb5eaa68091c9a73375f00555b66dd11 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -2128,16 +2128,16 @@ return params_to_strings( self.inputs, params, app ) def params_from_strings( self, params, app, ignore_errors=False ): return params_from_strings( self.inputs, params, app, ignore_errors ) - def check_and_update_param_values( self, values, trans ): + def check_and_update_param_values( self, values, trans, update_values=True ): """ Check that all parameters have values, and fill in with default values where necessary. This could be called after loading values from a database in case new parameters have been added. """ messages = {} - self.check_and_update_param_values_helper( self.inputs, values, trans, messages ) + self.check_and_update_param_values_helper( self.inputs, values, trans, messages, update_values=update_values ) return messages - def check_and_update_param_values_helper( self, inputs, values, trans, messages, context=None, prefix="" ): + def check_and_update_param_values_helper( self, inputs, values, trans, messages, context=None, prefix="", update_values=True ): """ Recursive helper for `check_and_update_param_values_helper` """ @@ -2183,10 +2183,11 @@ # Regular tool parameter, no recursion needed try: #this will fail when a parameter's type has changed to a non-compatible one: e.g. conditional group changed to dataset input - input.value_from_basic( values[ input.name ], trans.app, ignore_errors=False ) + input.value_from_basic( input.value_to_basic( values[ input.name ], trans.app ), trans.app, ignore_errors=False ) except: messages[ input.name ] = "Value no longer valid for '%s%s', replaced with default" % ( prefix, input.label ) - values[ input.name ] = input.get_initial_value( trans, context ) + if update_values: + values[ input.name ] = input.get_initial_value( trans, context ) def handle_unvalidated_param_values( self, input_values, app ): """ Find any instances of `UnvalidatedValue` within input_values and diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r a4113cc1cb5eaa68091c9a73375f00555b66dd11 lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py +++ b/lib/galaxy/tools/parameters/basic.py @@ -747,9 +747,9 @@ return { "__class__": "RuntimeValue" } return value def value_from_basic( self, value, app, ignore_errors=False ): - if isinstance( value, dict ) and value["__class__"] == "UnvalidatedValue": + if isinstance( value, dict ) and value.get( "__class__", None ) == "UnvalidatedValue": return UnvalidatedValue( value["value"] ) - return super( SelectToolParameter, self ).value_from_basic( value, app ) + return super( SelectToolParameter, self ).value_from_basic( value, app, ignore_errors=ignore_errors ) def need_late_validation( self, trans, context ): """ Determine whether we need to wait to validate this parameters value @@ -943,7 +943,7 @@ if not isinstance( value, list ): value = value.split( '\n' ) for column in value: - for column2 in column.split( ',' ): + for column2 in str( column ).split( ',' ): column2 = column2.strip() if column2: column_list.append( column2 ) @@ -1586,8 +1586,11 @@ elif isinstance( value, list) and len(value) > 0 and isinstance( value[0], DummyDataset): return None elif isinstance( value, list ): - return ",".join( [ val if isinstance( val, basestring ) else str(val.id) for val in value] ) - return value.id + return ",".join( [ str( self.to_string( val, app ) ) for val in value ] ) + try: + return value.id + except: + return str( value ) def to_python( self, value, app ): # Both of these values indicate that no dataset is selected. However, 'None' diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r a4113cc1cb5eaa68091c9a73375f00555b66dd11 lib/galaxy/tools/parameters/grouping.py --- a/lib/galaxy/tools/parameters/grouping.py +++ b/lib/galaxy/tools/parameters/grouping.py @@ -68,21 +68,25 @@ return rval def value_from_basic( self, value, app, ignore_errors=False ): rval = [] - for i, d in enumerate( value ): - rval_dict = {} - # If the special __index__ key is not set, create it (for backward - # compatibility) - rval_dict['__index__'] = d.get( '__index__', i ) - # Restore child inputs - for input in self.inputs.itervalues(): - if ignore_errors and input.name not in d: - # If we do not have a value, and are ignoring errors, we simply - # do nothing. There will be no value for the parameter in the - # conditional's values dictionary. - pass - else: - rval_dict[ input.name ] = input.value_from_basic( d[input.name], app, ignore_errors ) - rval.append( rval_dict ) + try: + for i, d in enumerate( value ): + rval_dict = {} + # If the special __index__ key is not set, create it (for backward + # compatibility) + rval_dict['__index__'] = d.get( '__index__', i ) + # Restore child inputs + for input in self.inputs.itervalues(): + if ignore_errors and input.name not in d: + # If we do not have a value, and are ignoring errors, we simply + # do nothing. There will be no value for the parameter in the + # conditional's values dictionary. + pass + else: + rval_dict[ input.name ] = input.value_from_basic( d[input.name], app, ignore_errors ) + rval.append( rval_dict ) + except Exception, e: + if not ignore_errors: + raise e return rval def visit_inputs( self, prefix, value, callback ): for i, d in enumerate( value ): @@ -441,24 +445,28 @@ return rval def value_from_basic( self, value, app, ignore_errors=False ): rval = dict() - current_case = rval['__current_case__'] = value['__current_case__'] - # Test param - if ignore_errors and self.test_param.name not in value: - # If ignoring errors, do nothing. However this is potentially very - # problematic since if we are missing the value of test param, - # the entire conditional is wrong. - pass - else: - rval[ self.test_param.name ] = self.test_param.value_from_basic( value[ self.test_param.name ], app, ignore_errors ) - # Inputs associated with current case - for input in self.cases[current_case].inputs.itervalues(): - if ignore_errors and input.name not in value: - # If we do not have a value, and are ignoring errors, we simply - # do nothing. There will be no value for the parameter in the - # conditional's values dictionary. + try: + current_case = rval['__current_case__'] = value['__current_case__'] + # Test param + if ignore_errors and self.test_param.name not in value: + # If ignoring errors, do nothing. However this is potentially very + # problematic since if we are missing the value of test param, + # the entire conditional is wrong. pass else: - rval[ input.name ] = input.value_from_basic( value[ input.name ], app, ignore_errors ) + rval[ self.test_param.name ] = self.test_param.value_from_basic( value[ self.test_param.name ], app, ignore_errors ) + # Inputs associated with current case + for input in self.cases[current_case].inputs.itervalues(): + if ignore_errors and input.name not in value: + # If we do not have a value, and are ignoring errors, we simply + # do nothing. There will be no value for the parameter in the + # conditional's values dictionary. + pass + else: + rval[ input.name ] = input.value_from_basic( value[ input.name ], app, ignore_errors ) + except Exception, e: + if not ignore_errors: + raise e return rval def visit_inputs( self, prefix, value, callback ): current_case = value['__current_case__'] diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r a4113cc1cb5eaa68091c9a73375f00555b66dd11 lib/galaxy/tools/parameters/validation.py --- a/lib/galaxy/tools/parameters/validation.py +++ b/lib/galaxy/tools/parameters/validation.py @@ -182,10 +182,13 @@ def from_element( cls, param, elem ): return cls( message=elem.get( 'message', None ), check=elem.get( 'check', "" ), skip=elem.get( 'skip', "" ) ) def validate( self, value, history=None ): - if value and value.missing_meta( check = self.check, skip = self.skip ): - if self.message is None: - self.message = "Metadata missing, click the pencil icon in the history item to edit / save the metadata attributes" - raise ValueError( self.message ) + if value: + if not isinstance( value, model.DatasetInstance ): + raise ValueError( 'A non-dataset value was provided.' ) + if value.missing_meta( check = self.check, skip = self.skip ): + if self.message is None: + self.message = "Metadata missing, click the pencil icon in the history item to edit / save the metadata attributes" + raise ValueError( self.message ) class UnspecifiedBuildValidator( Validator ): """ diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r a4113cc1cb5eaa68091c9a73375f00555b66dd11 lib/galaxy/util/hash_util.py --- a/lib/galaxy/util/hash_util.py +++ b/lib/galaxy/util/hash_util.py @@ -32,3 +32,10 @@ def hmac_new( key, value ): return hmac.new( key, value, sha ).hexdigest() + +def is_hashable( value ): + try: + hash( value ) + except: + return False + return True diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r a4113cc1cb5eaa68091c9a73375f00555b66dd11 lib/galaxy/webapps/galaxy/controllers/tool_runner.py --- a/lib/galaxy/webapps/galaxy/controllers/tool_runner.py +++ b/lib/galaxy/webapps/galaxy/controllers/tool_runner.py @@ -4,6 +4,7 @@ from galaxy.web.base.controller import * from galaxy.util.bunch import Bunch +from galaxy.util.hash_util import is_hashable from galaxy.tools import DefaultToolState from galaxy.tools.parameters.basic import UnvalidatedValue from galaxy.tools.parameters import params_to_incoming @@ -174,7 +175,7 @@ params_objects = job.get_param_values( trans.app, ignore_errors = True ) except: raise Exception( "Failed to get parameters for dataset id %d " % data.id ) - upgrade_messages = tool.check_and_update_param_values( params_objects, trans ) + upgrade_messages = tool.check_and_update_param_values( params_objects, trans, update_values=False ) # Need to remap dataset parameters. Job parameters point to original # dataset used; parameter should be the analygous dataset in the # current history. @@ -199,12 +200,13 @@ if isinstance(value,list): values = [] for val in value: - if val in history.datasets: - values.append( val ) - elif val in hda_source_dict: - values.append( hda_source_dict[ val ]) + if is_hashable( val ): + if val in history.datasets: + values.append( val ) + elif val in hda_source_dict: + values.append( hda_source_dict[ val ]) return values - if value not in history.datasets and value in hda_source_dict: + if is_hashable( value ) and value not in history.datasets and value in hda_source_dict: return hda_source_dict[ value ] visit_input_values( tool.inputs, params_objects, rerun_callback ) # Create a fake tool_state for the tool, with the parameters values https://bitbucket.org/galaxy/galaxy-central/commits/58851b490112/ changeset: 58851b490112 user: dan date: 2013-01-10 22:15:30 summary: Merging heads affected #: 5 files diff -r ca12edce5f10d1411599dcb687d7d51706af99cb -r 58851b49011218dbb11370392bbc03f9b1e69a9f lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -2128,16 +2128,16 @@ return params_to_strings( self.inputs, params, app ) def params_from_strings( self, params, app, ignore_errors=False ): return params_from_strings( self.inputs, params, app, ignore_errors ) - def check_and_update_param_values( self, values, trans ): + def check_and_update_param_values( self, values, trans, update_values=True ): """ Check that all parameters have values, and fill in with default values where necessary. This could be called after loading values from a database in case new parameters have been added. """ messages = {} - self.check_and_update_param_values_helper( self.inputs, values, trans, messages ) + self.check_and_update_param_values_helper( self.inputs, values, trans, messages, update_values=update_values ) return messages - def check_and_update_param_values_helper( self, inputs, values, trans, messages, context=None, prefix="" ): + def check_and_update_param_values_helper( self, inputs, values, trans, messages, context=None, prefix="", update_values=True ): """ Recursive helper for `check_and_update_param_values_helper` """ @@ -2183,10 +2183,11 @@ # Regular tool parameter, no recursion needed try: #this will fail when a parameter's type has changed to a non-compatible one: e.g. conditional group changed to dataset input - input.value_from_basic( values[ input.name ], trans.app, ignore_errors=False ) + input.value_from_basic( input.value_to_basic( values[ input.name ], trans.app ), trans.app, ignore_errors=False ) except: messages[ input.name ] = "Value no longer valid for '%s%s', replaced with default" % ( prefix, input.label ) - values[ input.name ] = input.get_initial_value( trans, context ) + if update_values: + values[ input.name ] = input.get_initial_value( trans, context ) def handle_unvalidated_param_values( self, input_values, app ): """ Find any instances of `UnvalidatedValue` within input_values and diff -r ca12edce5f10d1411599dcb687d7d51706af99cb -r 58851b49011218dbb11370392bbc03f9b1e69a9f lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py +++ b/lib/galaxy/tools/parameters/basic.py @@ -747,9 +747,9 @@ return { "__class__": "RuntimeValue" } return value def value_from_basic( self, value, app, ignore_errors=False ): - if isinstance( value, dict ) and value["__class__"] == "UnvalidatedValue": + if isinstance( value, dict ) and value.get( "__class__", None ) == "UnvalidatedValue": return UnvalidatedValue( value["value"] ) - return super( SelectToolParameter, self ).value_from_basic( value, app ) + return super( SelectToolParameter, self ).value_from_basic( value, app, ignore_errors=ignore_errors ) def need_late_validation( self, trans, context ): """ Determine whether we need to wait to validate this parameters value @@ -943,7 +943,7 @@ if not isinstance( value, list ): value = value.split( '\n' ) for column in value: - for column2 in column.split( ',' ): + for column2 in str( column ).split( ',' ): column2 = column2.strip() if column2: column_list.append( column2 ) @@ -1586,8 +1586,11 @@ elif isinstance( value, list) and len(value) > 0 and isinstance( value[0], DummyDataset): return None elif isinstance( value, list ): - return ",".join( [ val if isinstance( val, basestring ) else str(val.id) for val in value] ) - return value.id + return ",".join( [ str( self.to_string( val, app ) ) for val in value ] ) + try: + return value.id + except: + return str( value ) def to_python( self, value, app ): # Both of these values indicate that no dataset is selected. However, 'None' diff -r ca12edce5f10d1411599dcb687d7d51706af99cb -r 58851b49011218dbb11370392bbc03f9b1e69a9f lib/galaxy/tools/parameters/validation.py --- a/lib/galaxy/tools/parameters/validation.py +++ b/lib/galaxy/tools/parameters/validation.py @@ -182,10 +182,13 @@ def from_element( cls, param, elem ): return cls( message=elem.get( 'message', None ), check=elem.get( 'check', "" ), skip=elem.get( 'skip', "" ) ) def validate( self, value, history=None ): - if value and value.missing_meta( check = self.check, skip = self.skip ): - if self.message is None: - self.message = "Metadata missing, click the pencil icon in the history item to edit / save the metadata attributes" - raise ValueError( self.message ) + if value: + if not isinstance( value, model.DatasetInstance ): + raise ValueError( 'A non-dataset value was provided.' ) + if value.missing_meta( check = self.check, skip = self.skip ): + if self.message is None: + self.message = "Metadata missing, click the pencil icon in the history item to edit / save the metadata attributes" + raise ValueError( self.message ) class UnspecifiedBuildValidator( Validator ): """ diff -r ca12edce5f10d1411599dcb687d7d51706af99cb -r 58851b49011218dbb11370392bbc03f9b1e69a9f lib/galaxy/util/hash_util.py --- a/lib/galaxy/util/hash_util.py +++ b/lib/galaxy/util/hash_util.py @@ -32,3 +32,10 @@ def hmac_new( key, value ): return hmac.new( key, value, sha ).hexdigest() + +def is_hashable( value ): + try: + hash( value ) + except: + return False + return True diff -r ca12edce5f10d1411599dcb687d7d51706af99cb -r 58851b49011218dbb11370392bbc03f9b1e69a9f lib/galaxy/webapps/galaxy/controllers/tool_runner.py --- a/lib/galaxy/webapps/galaxy/controllers/tool_runner.py +++ b/lib/galaxy/webapps/galaxy/controllers/tool_runner.py @@ -4,6 +4,7 @@ from galaxy.web.base.controller import * from galaxy.util.bunch import Bunch +from galaxy.util.hash_util import is_hashable from galaxy.tools import DefaultToolState from galaxy.tools.parameters.basic import UnvalidatedValue from galaxy.tools.parameters import params_to_incoming @@ -174,7 +175,7 @@ params_objects = job.get_param_values( trans.app, ignore_errors = True ) except: raise Exception( "Failed to get parameters for dataset id %d " % data.id ) - upgrade_messages = tool.check_and_update_param_values( params_objects, trans ) + upgrade_messages = tool.check_and_update_param_values( params_objects, trans, update_values=False ) # Need to remap dataset parameters. Job parameters point to original # dataset used; parameter should be the analygous dataset in the # current history. @@ -199,12 +200,13 @@ if isinstance(value,list): values = [] for val in value: - if val in history.datasets: - values.append( val ) - elif val in hda_source_dict: - values.append( hda_source_dict[ val ]) + if is_hashable( val ): + if val in history.datasets: + values.append( val ) + elif val in hda_source_dict: + values.append( hda_source_dict[ val ]) return values - if value not in history.datasets and value in hda_source_dict: + if is_hashable( value ) and value not in history.datasets and value in hda_source_dict: return hda_source_dict[ value ] visit_input_values( tool.inputs, params_objects, rerun_callback ) # Create a fake tool_state for the tool, with the parameters values Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Preserve dbkey when navigating histories and datasets in Trackster.
by Bitbucket 10 Jan '13

10 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/ca12edce5f10/ changeset: ca12edce5f10 user: jgoecks date: 2013-01-10 21:49:55 summary: Preserve dbkey when navigating histories and datasets in Trackster. affected #: 1 file diff -r 40a5470e3082c07fb30381ea0dce83543e85ad46 -r ca12edce5f10d1411599dcb687d7d51706af99cb templates/tracks/history_select_grid.mako --- a/templates/tracks/history_select_grid.mako +++ b/templates/tracks/history_select_grid.mako @@ -70,7 +70,14 @@ <% data_libraries_active = " activetab" %> %endif %endif - <a class="addtracktab${histories_active}" href="${h.url_for( action='list_histories' )}">Histories</a> + ## Add filter parameters manually because they include a hyphen and hence cannot be + ## added as key words. + <% + dbkey = '?' + if cur_filter_dict: + dbkey = cur_filter_dict.get( 'dbkey', '?' ) + %> + <a class="addtracktab${histories_active}" href="${h.url_for( action='list_histories')}?f-dbkey=${dbkey}">Histories</a><a class="addtracktab${data_libraries_active}" href="${h.url_for( action='list_libraries' )}">Data Libraries</a><div class="divider"></div></%def> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Default to first chromosome if viewport exists but is empty.
by Bitbucket 10 Jan '13

10 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/40a5470e3082/ changeset: 40a5470e3082 user: jgoecks date: 2013-01-10 21:26:21 summary: Default to first chromosome if viewport exists but is empty. affected #: 1 file diff -r b79236271786713189eae418da5626c159c4f119 -r 40a5470e3082c07fb30381ea0dce83543e85ad46 static/scripts/viz/trackster_ui.js --- a/static/scripts/viz/trackster_ui.js +++ b/static/scripts/viz/trackster_ui.js @@ -191,8 +191,7 @@ $.when( view.load_chroms_deferred ).then(function(chrom_info) { // Viewport config. if (viewport_config) { - var - chrom = viewport_config.chrom, + var chrom = viewport_config.chrom, start = viewport_config.start, end = viewport_config.end, overview_drawable_name = viewport_config.overview; @@ -200,6 +199,10 @@ if (chrom && (start !== undefined) && end) { view.change_chrom(chrom, start, end); } + else { + // No valid viewport, so use first chromosome. + view.change_chrom(chrom_info[0].chrom); + } } else { // No viewport, so use first chromosome. Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Visualization framework clean up: (a) remove 'circster' visualization type; and (b) add link to view trackster visualizations in Circster.
by Bitbucket 10 Jan '13

10 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/b79236271786/ changeset: b79236271786 user: jgoecks date: 2013-01-10 21:18:34 summary: Visualization framework clean up: (a) remove 'circster' visualization type; and (b) add link to view trackster visualizations in Circster. affected #: 3 files diff -r 77832c6e2c342dbedfe7d8fc2f2c28772b956b87 -r b79236271786713189eae418da5626c159c4f119 lib/galaxy/web/base/controller.py --- a/lib/galaxy/web/base/controller.py +++ b/lib/galaxy/web/base/controller.py @@ -315,7 +315,7 @@ UsesLibraryMixinItems ): """ Mixin for controllers that use Visualization objects. """ - viz_types = [ "trackster", "circster" ] + viz_types = [ "trackster" ] def create_visualization( self, trans, type, title="Untitled Genome Vis", slug=None, dbkey=None, annotation=None, config={}, save=True ): """ Create visualiation and first revision. """ diff -r 77832c6e2c342dbedfe7d8fc2f2c28772b956b87 -r b79236271786713189eae418da5626c159c4f119 lib/galaxy/webapps/galaxy/controllers/visualization.py --- a/lib/galaxy/webapps/galaxy/controllers/visualization.py +++ b/lib/galaxy/webapps/galaxy/controllers/visualization.py @@ -187,7 +187,8 @@ grids.GridAction( "Create new visualization", dict( action='create' ) ) ] operations = [ - grids.GridOperation( "View/Edit", allow_multiple=False, url_args=get_url_args ), + grids.GridOperation( "Open", allow_multiple=False, url_args=get_url_args ), + grids.GridOperation( "Open in Circster", allow_multiple=False, condition=( lambda item: item.type == 'trackster' ), url_args=dict( action='circster' ) ), grids.GridOperation( "Edit Attributes", allow_multiple=False, url_args=dict( action='edit') ), grids.GridOperation( "Copy", allow_multiple=False, condition=( lambda item: not item.deleted ), async_compatible=False, url_args=dict( action='clone') ), grids.GridOperation( "Share or Publish", allow_multiple=False, condition=( lambda item: not item.deleted ), async_compatible=False ), @@ -727,6 +728,7 @@ """ # Get dataset to add. + dataset = None if dataset_id: dataset = self.get_hda_or_ldda( trans, hda_ldda, dataset_id ) @@ -734,6 +736,7 @@ if id: # Display existing viz. vis = self.get_visualization( trans, id, check_ownership=False, check_accessible=True ) + dbkey = vis.dbkey else: # Create new viz. if not dbkey: diff -r 77832c6e2c342dbedfe7d8fc2f2c28772b956b87 -r b79236271786713189eae418da5626c159c4f119 templates/webapps/galaxy/base_panels.mako --- a/templates/webapps/galaxy/base_panels.mako +++ b/templates/webapps/galaxy/base_panels.mako @@ -98,7 +98,7 @@ ## Visualization menu. <% menu_options = [ - [_('New Visualization'), h.url_for( controller='/visualization', action='trackster' ) ], + [_('New Track Browser'), h.url_for( controller='/visualization', action='trackster' ) ], [_('Saved Visualizations'), h.url_for( controller='/visualization', action='list' ) ] ] tab( "visualization", _("Visualization"), h.url_for( controller='/visualization', action='list'), menu_options=menu_options ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by Bitbucket 10 Jan '13

10 Jan '13

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/cf7a999b8797/ changeset: cf7a999b8797 user: nsoranzo date: 2013-01-10 18:50:51 summary: Add missing samtools requirement to bowtie2 wrapper affected #: 1 file diff -r caaab03824478384c256fc6b5678bb39dbf8f9f2 -r cf7a999b8797b2bbd7281cd86464b8b324bdb64f tools/sr_mapping/bowtie2_wrapper.xml --- a/tools/sr_mapping/bowtie2_wrapper.xml +++ b/tools/sr_mapping/bowtie2_wrapper.xml @@ -4,6 +4,7 @@ <version_command>bowtie2 --version</version_command><requirements><requirement type="package">bowtie2</requirement> + <requirement type="package">samtools</requirement></requirements><command interpreter="python"> bowtie2_wrapper.py https://bitbucket.org/galaxy/galaxy-central/commits/77832c6e2c34/ changeset: 77832c6e2c34 user: jgoecks date: 2013-01-10 19:17:11 summary: Merged in nsoranzo/galaxy-central (pull request #104: Add missing samtools requirement to bowtie2 wrapper) affected #: 1 file diff -r c015b82b3944f967e2c859d5552c00e3e38a2da0 -r 77832c6e2c342dbedfe7d8fc2f2c28772b956b87 tools/sr_mapping/bowtie2_wrapper.xml --- a/tools/sr_mapping/bowtie2_wrapper.xml +++ b/tools/sr_mapping/bowtie2_wrapper.xml @@ -4,6 +4,7 @@ <version_command>bowtie2 --version</version_command><requirements><requirement type="package">bowtie2</requirement> + <requirement type="package">samtools</requirement></requirements><command interpreter="python"> bowtie2_wrapper.py Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: natefoo: DRMAA: Don't attempt to stop jobs that have no external ID set.
by Bitbucket 10 Jan '13

10 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/c015b82b3944/ changeset: c015b82b3944 user: natefoo date: 2013-01-10 18:25:27 summary: DRMAA: Don't attempt to stop jobs that have no external ID set. affected #: 1 file diff -r caaab03824478384c256fc6b5678bb39dbf8f9f2 -r c015b82b3944f967e2c859d5552c00e3e38a2da0 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py +++ b/lib/galaxy/jobs/runners/drmaa.py @@ -400,16 +400,18 @@ def stop_job( self, job ): """Attempts to delete a job from the DRM queue""" try: + ext_id = job.get_job_runner_external_id() + assert ext_id not in ( None, 'None' ), 'External job id is None' if self.external_killJob_script is None: - self.ds.control( job.get_job_runner_external_id(), drmaa.JobControlAction.TERMINATE ) + self.ds.control( ext_id, drmaa.JobControlAction.TERMINATE ) else: # FIXME: hardcoded path - subprocess.Popen( [ '/usr/bin/sudo', '-E', self.external_killJob_script, str( job.get_job_runner_external_id() ), str( self.userid ) ], shell=False ) - log.debug( "(%s/%s) Removed from DRM queue at user's request" % ( job.get_id(), job.get_job_runner_external_id() ) ) + subprocess.Popen( [ '/usr/bin/sudo', '-E', self.external_killJob_script, str( ext_id ), str( self.userid ) ], shell=False ) + log.debug( "(%s/%s) Removed from DRM queue at user's request" % ( job.get_id(), ext_id ) ) except drmaa.InvalidJobException: - log.debug( "(%s/%s) User killed running job, but it was already dead" % ( job.get_id(), job.get_job_runner_external_id() ) ) + log.debug( "(%s/%s) User killed running job, but it was already dead" % ( job.get_id(), ext_id ) ) except Exception, e: - log.debug( "(%s/%s) User killed running job, but error encountered removing from DRM queue: %s" % ( job.get_id(), job.get_job_runner_external_id(), e ) ) + log.debug( "(%s/%s) User killed running job, but error encountered removing from DRM queue: %s" % ( job.get_id(), ext_id, e ) ) def recover( self, job, job_wrapper ): """Recovers jobs stuck in the queued/running state when Galaxy started""" Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by Bitbucket 10 Jan '13

10 Jan '13

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/087a08c43cb5/ changeset: 087a08c43cb5 user: jmchilton date: 2012-12-07 18:48:07 summary: Fix upload of files to history through API (as outlined here http://dev.list.galaxyproject.org/Uploading-large-files-to-history-through-…) affected #: 2 files diff -r 330f05e523aee275c91f3e5de1177f8eb832d512 -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 lib/galaxy/web/framework/__init__.py --- a/lib/galaxy/web/framework/__init__.py +++ b/lib/galaxy/web/framework/__init__.py @@ -142,6 +142,14 @@ named_args, _, _, _ = inspect.getargspec(func) for arg in named_args: payload.pop(arg, None) + for k, v in payload.iteritems(): + if isinstance(v, (str, unicode)): + try: + payload[k] = simplejson.loads(v) + except: + # may not actually be json, just continue + pass + payload = util.recursively_stringify_dictionary_keys( payload ) else: # Assume application/json content type and parse request body manually, since wsgi won't do it. However, the order of this check # should ideally be in reverse, with the if clause being a check for application/json and the else clause assuming a standard encoding diff -r 330f05e523aee275c91f3e5de1177f8eb832d512 -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 lib/galaxy/webapps/galaxy/api/tools.py --- a/lib/galaxy/webapps/galaxy/api/tools.py +++ b/lib/galaxy/webapps/galaxy/api/tools.py @@ -72,6 +72,11 @@ # Set up inputs. inputs = payload[ 'inputs' ] + # Find files coming in as multipart file data and add to inputs. + for k, v in payload.iteritems(): + if k.startswith("files_"): + inputs[k] = v + # HACK: add run button so that tool.handle_input will run tool. inputs['runtool_btn'] = 'Execute' # TODO: encode data ids and decode ids. https://bitbucket.org/galaxy/galaxy-central/commits/caaab0382447/ changeset: caaab0382447 user: jmchilton date: 2013-01-10 17:51:57 summary: Merge. affected #: 253 files diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 .hgignore --- a/.hgignore +++ b/.hgignore @@ -74,6 +74,10 @@ */variables.less static/june_2007_style/blue/base_sprites.less +# Testing +selenium-server.jar +selenium_results.html + # Documentation build files. doc/build diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/Makefile --- a/doc/Makefile +++ b/doc/Makefile @@ -6,6 +6,9 @@ SPHINXBUILD = sphinx-build PAPER = BUILDDIR = build +UPDATEWORKDIR = /tmp/galaxySphinxUpdate +UPDATEWORKSOURCELIB = $(UPDATEWORKDIR)/source/lib +SPHINXAPIDOC = sphinx-apidoc # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 @@ -21,7 +24,7 @@ $(TOOLDATASHAREDDIR)/ncbi/builds.txt \ $(TOOLDATASHAREDDIR)/ucsc/publicbuilds.txt -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext updaterst # Sphinx wants the build files to be there; Copy the sample files into @@ -51,9 +54,13 @@ @echo " changes to make an overview of all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" + @echo " updaterst to update sphinx rst to reflect code structure changes" + +# might also want to do +# cd source/lib; hg revert; rm *.rst.orig; or not. clean: - -rm -rf $(BUILDDIR)/* + -rm -rf $(BUILDDIR)/* $(UPDATEWORKDIR) html: $(TOOLDATABUILDFILES) $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @@ -165,3 +172,20 @@ $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." + +# Targets for updating the structure of the Sphinx RST doc for lib/ + +$(UPDATEWORKSOURCELIB): + mkdir -p $(UPDATEWORKSOURCELIB) + +# Create a fresh version of the RST files for the lib, and then create a +# unified patch file (ignore all emacs leftovers). +# Feed that to our custom version of patch.py, which applies patches that +# are only adds, and reports everything else to the user to deal with manually +# +# Note: this is still a very rough process. the run of patch.py gets some +# errors that don't mean anything to us. And the manual process is not fun. +updaterst: $(UPDATEWORKSOURCELIB) + $(SPHINXAPIDOC) -o $(UPDATEWORKSOURCELIB) ../lib + -diff -x '*.rst~' -ru source/lib $(UPDATEWORKSOURCELIB) > $(UPDATEWORKDIR)/alldifs.patch + ./patch.py $(UPDATEWORKDIR)/alldifs.patch diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/patch.py --- /dev/null +++ b/doc/patch.py @@ -0,0 +1,1085 @@ +#!/usr/bin/env python +""" Patch utility to apply unified diffs + + Brute-force line-by-line non-recursive parsing + + Copyright (c) 2008-2012 anatoly techtonik + Available under the terms of MIT license + + Project home: http://code.google.com/p/python-patch/ + + + $Id: patch.py 181 2012-11-23 16:03:05Z techtonik $ + $HeadURL: https://python-patch.googlecode.com/svn/trunk/patch.py $ + + This program needs further tweaking for how we use it at Galaxy. +""" + +__author__ = "anatoly techtonik <techtonik(a)gmail.com>" +__version__ = "1.12.11" + +import copy +import logging +import re +# cStringIO doesn't support unicode in 2.5 +from StringIO import StringIO +import urllib2 + +from os.path import exists, isfile, abspath +import os +import shutil + +#------------------------------------------------ +# Logging is controlled by logger named after the +# module name (e.g. 'patch' for patch.py module) + +debugmode = False + +logger = logging.getLogger(__name__) + +debug = logger.debug +info = logger.info +warning = logger.warning + +class NullHandler(logging.Handler): + """ Copied from Python 2.7 to avoid getting + `No handlers could be found for logger "patch"` + http://bugs.python.org/issue16539 + """ + def handle(self, record): + pass + def emit(self, record): + pass + def createLock(self): + self.lock = None + +logger.addHandler(NullHandler()) + +#------------------------------------------------ +# Constants for Patch/PatchSet types + +DIFF = PLAIN = "plain" +GIT = "git" +HG = MERCURIAL = "mercurial" +SVN = SUBVERSION = "svn" +# mixed type is only actual when PatchSet contains +# Patches of different type +MIXED = MIXED = "mixed" + + +#------------------------------------------------ +# Helpers (these could come with Python stdlib) + +# x...() function are used to work with paths in +# cross-platform manner - all paths use forward +# slashes even on Windows. + +def xisabs(filename): + """ Cross-platform version of `os.path.isabs()` + Returns True if `filename` is absolute on + Linux, OS X or Windows. + """ + if filename.startswith('/'): # Linux/Unix + return True + elif filename.startswith('\\'): # Windows + return True + elif re.match(r'\w:[\\/]', filename): # Windows + return True + return False + +def xnormpath(path): + """ Cross-platform version of os.path.normpath """ + return os.path.normpath(path).replace(os.sep, '/') + +def xstrip(filename): + """ Make relative path out of absolute by stripping + prefixes used on Linux, OS X and Windows. + + This function is critical for security. + """ + while xisabs(filename): + # strip windows drive with all slashes + if re.match(r'\w:[\\/]', filename): + filename = re.sub(r'^\w+:[\\/]+', '', filename) + # strip all slashes + elif re.match(r'[\\/]', filename): + filename = re.sub(r'^[\\/]+', '', filename) + return filename + +#----------------------------------------------- +# Main API functions + +def fromfile(filename): + """ Parse patch file. If successful, returns + PatchSet() object. Otherwise returns False. + """ + patchset = PatchSet() + debug("reading %s" % filename) + fp = open(filename, "rb") + res = patchset.parse(fp) + fp.close() + if res == True: + return patchset + return False + + +def fromstring(s): + """ Parse text string and return PatchSet() + object (or False if parsing fails) + """ + ps = PatchSet( StringIO(s) ) + if ps.errors == 0: + return ps + return False + + +def fromurl(url): + """ Parse patch from an URL, return False + if an error occured. Note that this also + can throw urlopen() exceptions. + """ + ps = PatchSet( urllib2.urlopen(url) ) + if ps.errors == 0: + return ps + return False + + +# --- Utility functions --- +# [ ] reuse more universal pathsplit() +def pathstrip(path, n): + """ Strip n leading components from the given path """ + pathlist = [path] + while os.path.dirname(pathlist[0]) != '': + pathlist[0:1] = os.path.split(pathlist[0]) + return '/'.join(pathlist[n:]) +# --- /Utility function --- + + +class Hunk(object): + """ Parsed hunk data container (hunk starts with @@ -R +R @@) """ + + def __init__(self): + self.startsrc=None #: line count starts with 1 + self.linessrc=None + self.starttgt=None + self.linestgt=None + self.invalid=False + self.hasplus=False # True if any "+" lines in hunk + self.hasminus=False # True if any "-" lines in hunk + self.text=[] + + def originalText(self): + + return("@@ -" + str(self.startsrc) + + "," + str(self.linessrc) + + " +" + str(self.starttgt) + + "," + str(self.linestgt) + + "\n" + + self.printableText()) + + def printableText(self): + """Reformat text into printable text""" + + # yeah, there must be a better way to do this. + printable = "" + for line in self.text: + printable += line + + return printable + + + +# def apply(self, estream): +# """ write hunk data into enumerable stream +# return strings one by one until hunk is +# over +# +# enumerable stream are tuples (lineno, line) +# where lineno starts with 0 +# """ +# pass + + +class Patch(object): + """ Patch for a single file """ + def __init__(self): + self.source = None + self.target = None + self.hunks = [] + self.hunkends = [] + self.header = [] + + self.type = None + + +class PatchSet(object): + + def __init__(self, stream=None): + # --- API accessible fields --- + + # name of the PatchSet (filename or ...) + self.name = None + # patch set type - one of constants + self.type = None + + # list of Patch objects + self.items = [] + + self.errors = 0 # fatal parsing errors + self.warnings = 0 # non-critical warnings + # --- /API --- + + if stream: + self.parse(stream) + + def __len__(self): + return len(self.items) + + def parse(self, stream): + """ parse unified diff + return True on success + """ + lineends = dict(lf=0, crlf=0, cr=0) + nexthunkno = 0 #: even if index starts with 0 user messages number hunks from 1 + + p = None + hunk = None + # hunkactual variable is used to calculate hunk lines for comparison + hunkactual = dict(linessrc=None, linestgt=None) + + + class wrapumerate(enumerate): + """Enumerate wrapper that uses boolean end of stream status instead of + StopIteration exception, and properties to access line information. + """ + + def __init__(self, *args, **kwargs): + # we don't call parent, it is magically created by __new__ method + + self._exhausted = False + self._lineno = False # after end of stream equal to the num of lines + self._line = False # will be reset to False after end of stream + + def next(self): + """Try to read the next line and return True if it is available, + False if end of stream is reached.""" + if self._exhausted: + return False + + try: + self._lineno, self._line = super(wrapumerate, self).next() + except StopIteration: + self._exhausted = True + self._line = False + return False + return True + + @property + def is_empty(self): + return self._exhausted + + @property + def line(self): + return self._line + + @property + def lineno(self): + return self._lineno + + # define states (possible file regions) that direct parse flow + headscan = True # start with scanning header + filenames = False # lines starting with --- and +++ + + hunkhead = False # @@ -R +R @@ sequence + hunkbody = False # + hunkskip = False # skipping invalid hunk mode + + hunkparsed = False # state after successfully parsed hunk + + # regexp to match start of hunk, used groups - 1,3,4,6 + re_hunk_start = re.compile("^@@ -(\d+)(,(\d+))? \+(\d+)(,(\d+))?") + + self.errors = 0 + # temp buffers for header and filenames info + header = [] + srcname = None + tgtname = None + + # start of main cycle + # each parsing block already has line available in fe.line + fe = wrapumerate(stream) + while fe.next(): + + # -- deciders: these only switch state to decide who should process + # -- line fetched at the start of this cycle + if hunkparsed: + hunkparsed = False + if re_hunk_start.match(fe.line): + hunkhead = True + elif fe.line.startswith("--- "): + filenames = True + else: + headscan = True + # -- ------------------------------------ + + # read out header + if headscan: + while not fe.is_empty and not fe.line.startswith("--- "): + header.append(fe.line) + fe.next() + if fe.is_empty: + if p == None: + debug("no patch data found") # error is shown later + self.errors += 1 + else: + info("%d unparsed bytes left at the end of stream" % len(''.join(header))) + self.warnings += 1 + # TODO check for \No new line at the end.. + # TODO test for unparsed bytes + # otherwise error += 1 + # this is actually a loop exit + continue + + headscan = False + # switch to filenames state + filenames = True + + line = fe.line + lineno = fe.lineno + + + # hunkskip and hunkbody code skipped until definition of hunkhead is parsed + if hunkbody: + # [x] treat empty lines inside hunks as containing single space + # (this happens when diff is saved by copy/pasting to editor + # that strips trailing whitespace) + if line.strip("\r\n") == "": + debug("expanding empty line in a middle of hunk body") + self.warnings += 1 + line = ' ' + line + + # process line first + if re.match(r"^[- \+\\]", line): + # gather stats about line endings + if line.endswith("\r\n"): + p.hunkends["crlf"] += 1 + elif line.endswith("\n"): + p.hunkends["lf"] += 1 + elif line.endswith("\r"): + p.hunkends["cr"] += 1 + + if line.startswith("-"): + hunkactual["linessrc"] += 1 + hunk.hasminus = True + elif line.startswith("+"): + hunkactual["linestgt"] += 1 + hunk.hasplus = True + elif not line.startswith("\\"): + hunkactual["linessrc"] += 1 + hunkactual["linestgt"] += 1 + hunk.text.append(line) + # todo: handle \ No newline cases + else: + warning("invalid hunk no.%d at %d for target file %s" % (nexthunkno, lineno+1, p.target)) + # add hunk status node + hunk.invalid = True + p.hunks.append(hunk) + self.errors += 1 + # switch to hunkskip state + hunkbody = False + hunkskip = True + + # check exit conditions + if hunkactual["linessrc"] > hunk.linessrc or hunkactual["linestgt"] > hunk.linestgt: + warning("extra lines for hunk no.%d at %d for target %s" % (nexthunkno, lineno+1, p.target)) + # add hunk status node + hunk.invalid = True + p.hunks.append(hunk) + self.errors += 1 + # switch to hunkskip state + hunkbody = False + hunkskip = True + elif hunk.linessrc == hunkactual["linessrc"] and hunk.linestgt == hunkactual["linestgt"]: + # hunk parsed successfully + p.hunks.append(hunk) + # switch to hunkparsed state + hunkbody = False + hunkparsed = True + + # detect mixed window/unix line ends + ends = p.hunkends + if ((ends["cr"]!=0) + (ends["crlf"]!=0) + (ends["lf"]!=0)) > 1: + warning("inconsistent line ends in patch hunks for %s" % p.source) + self.warnings += 1 + if debugmode: + debuglines = dict(ends) + debuglines.update(file=p.target, hunk=nexthunkno) + debug("crlf: %(crlf)d lf: %(lf)d cr: %(cr)d\t - file: %(file)s hunk: %(hunk)d" % debuglines) + # fetch next line + continue + + if hunkskip: + if re_hunk_start.match(line): + # switch to hunkhead state + hunkskip = False + hunkhead = True + elif line.startswith("--- "): + # switch to filenames state + hunkskip = False + filenames = True + if debugmode and len(self.items) > 0: + debug("- %2d hunks for %s" % (len(p.hunks), p.source)) + + if filenames: + if line.startswith("--- "): + if srcname != None: + # XXX testcase + warning("skipping false patch for %s" % srcname) + srcname = None + # XXX header += srcname + # double source filename line is encountered + # attempt to restart from this second line + re_filename = "^--- ([^\t]+)" + match = re.match(re_filename, line) + # todo: support spaces in filenames + if match: + srcname = match.group(1).strip() + else: + warning("skipping invalid filename at line %d" % lineno) + self.errors += 1 + # XXX p.header += line + # switch back to headscan state + filenames = False + headscan = True + elif not line.startswith("+++ "): + if srcname != None: + warning("skipping invalid patch with no target for %s" % srcname) + self.errors += 1 + srcname = None + # XXX header += srcname + # XXX header += line + else: + # this should be unreachable + warning("skipping invalid target patch") + filenames = False + headscan = True + else: + if tgtname != None: + # XXX seems to be a dead branch + warning("skipping invalid patch - double target at line %d" % lineno) + self.errors += 1 + srcname = None + tgtname = None + # XXX header += srcname + # XXX header += tgtname + # XXX header += line + # double target filename line is encountered + # switch back to headscan state + filenames = False + headscan = True + else: + re_filename = "^\+\+\+ ([^\t]+)" + match = re.match(re_filename, line) + if not match: + warning("skipping invalid patch - no target filename at line %d" % lineno) + self.errors += 1 + srcname = None + # switch back to headscan state + filenames = False + headscan = True + else: + if p: # for the first run p is None + self.items.append(p) + p = Patch() + p.source = srcname + srcname = None + p.target = match.group(1).strip() + p.header = header + header = [] + # switch to hunkhead state + filenames = False + hunkhead = True + nexthunkno = 0 + p.hunkends = lineends.copy() + continue + + if hunkhead: + match = re.match("^@@ -(\d+)(,(\d+))? \+(\d+)(,(\d+))?", line) + if not match: + if not p.hunks: + warning("skipping invalid patch with no hunks for file %s" % p.source) + self.errors += 1 + # XXX review switch + # switch to headscan state + hunkhead = False + headscan = True + continue + else: + # TODO review condition case + # switch to headscan state + hunkhead = False + headscan = True + else: + hunk = Hunk() + hunk.startsrc = int(match.group(1)) + hunk.linessrc = 1 + if match.group(3): hunk.linessrc = int(match.group(3)) + hunk.starttgt = int(match.group(4)) + hunk.linestgt = 1 + if match.group(6): hunk.linestgt = int(match.group(6)) + hunk.invalid = False + hunk.text = [] + + hunkactual["linessrc"] = hunkactual["linestgt"] = 0 + + # switch to hunkbody state + hunkhead = False + hunkbody = True + nexthunkno += 1 + continue + + # /while fe.next() + + if p: + self.items.append(p) + + if not hunkparsed: + if hunkskip: + warning("warning: finished with errors, some hunks may be invalid") + elif headscan: + if len(self.items) == 0: + warning("error: no patch data found!") + return False + else: # extra data at the end of file + pass + else: + warning("error: patch stream is incomplete!") + self.errors += 1 + if len(self.items) == 0: + return False + + if debugmode and len(self.items) > 0: + debug("- %2d hunks for %s" % (len(p.hunks), p.source)) + + # XXX fix total hunks calculation + debug("total files: %d total hunks: %d" % (len(self.items), + sum(len(p.hunks) for p in self.items))) + + # ---- detect patch and patchset types ---- + for idx, p in enumerate(self.items): + self.items[idx].type = self._detect_type(p) + + types = set([p.type for p in self.items]) + if len(types) > 1: + self.type = MIXED + else: + self.type = types.pop() + # -------- + + self._normalize_filenames() + + return (self.errors == 0) + + def _detect_type(self, p): + """ detect and return type for the specified Patch object + analyzes header and filenames info + + NOTE: must be run before filenames are normalized + """ + + # check for SVN + # - header starts with Index: + # - next line is ===... delimiter + # - filename is followed by revision number + # TODO add SVN revision + if (len(p.header) > 1 and p.header[-2].startswith("Index: ") + and p.header[-1].startswith("="*67)): + return SVN + + # common checks for both HG and GIT + DVCS = ((p.source.startswith('a/') or p.source == '/dev/null') + and (p.target.startswith('b/') or p.target == '/dev/null')) + + # GIT type check + # - header[-2] is like "diff --git a/oldname b/newname" + # - header[-1] is like "index <hash>..<hash><mode>" + # TODO add git rename diffs and add/remove diffs + # add git diff with spaced filename + # TODO http://www.kernel.org/pub/software/scm/git/docs/git-diff.html + + # detect the start of diff header - there might be some comments before + if len(p.header) > 1: + for idx in reversed(range(len(p.header))): + if p.header[idx].startswith("diff --git"): + break + if re.match(r'diff --git a/[\w/.]+ b/[\w/.]+', p.header[idx]): + if (idx+1 < len(p.header) + and re.match(r'index \w{7}..\w{7} \d{6}', p.header[idx+1])): + if DVCS: + return GIT + + # HG check + # + # - for plain HG format header is like "diff -r b2d9961ff1f5 filename" + # - for Git-style HG patches it is "diff --git a/oldname b/newname" + # - filename starts with a/, b/ or is equal to /dev/null + # - exported changesets also contain the header + # # HG changeset patch + # # User name(a)example.com + # ... + # TODO add MQ + # TODO add revision info + if len(p.header) > 0: + if DVCS and re.match(r'diff -r \w{12} .*', p.header[-1]): + return HG + if DVCS and p.header[-1].startswith('diff --git a/'): + if len(p.header) == 1: # native Git patch header len is 2 + return HG + elif p.header[0].startswith('# HG changeset patch'): + return HG + + return PLAIN + + + def _normalize_filenames(self): + """ sanitize filenames, normalizing paths, i.e.: + 1. strip a/ and b/ prefixes from GIT and HG style patches + 2. remove all references to parent directories (with warning) + 3. translate any absolute paths to relative (with warning) + + [x] always use forward slashes to be crossplatform + (diff/patch were born as a unix utility after all) + + return None + """ + for i,p in enumerate(self.items): + if p.type in (HG, GIT): + # TODO: figure out how to deal with /dev/null entries + debug("stripping a/ and b/ prefixes") + if p.source != '/dev/null': + if not p.source.startswith("a/"): + warning("invalid source filename") + else: + p.source = p.source[2:] + if p.target != '/dev/null': + if not p.target.startswith("b/"): + warning("invalid target filename") + else: + p.target = p.target[2:] + + p.source = xnormpath(p.source) + p.target = xnormpath(p.target) + + sep = '/' # sep value can be hardcoded, but it looks nice this way + + # references to parent are not allowed + if p.source.startswith(".." + sep): + warning("error: stripping parent path for source file patch no.%d" % (i+1)) + self.warnings += 1 + while p.source.startswith(".." + sep): + p.source = p.source.partition(sep)[2] + if p.target.startswith(".." + sep): + warning("error: stripping parent path for target file patch no.%d" % (i+1)) + self.warnings += 1 + while p.target.startswith(".." + sep): + p.target = p.target.partition(sep)[2] + # absolute paths are not allowed + if xisabs(p.source) or xisabs(p.target): + warning("error: absolute paths are not allowed - file no.%d" % (i+1)) + self.warnings += 1 + if xisabs(p.source): + warning("stripping absolute path from source name '%s'" % p.source) + p.source = xstrip(p.source) + if xisabs(p.target): + warning("stripping absolute path from target name '%s'" % p.target) + p.target = xstrip(p.target) + + self.items[i].source = p.source + self.items[i].target = p.target + + + def diffstat(self): + """ calculate diffstat and return as a string + Notes: + - original diffstat ouputs target filename + - single + or - shouldn't escape histogram + """ + names = [] + insert = [] + delete = [] + namelen = 0 + maxdiff = 0 # max number of changes for single file + # (for histogram width calculation) + for patch in self.items: + i,d = 0,0 + for hunk in patch.hunks: + for line in hunk.text: + if line.startswith('+'): + i += 1 + elif line.startswith('-'): + d += 1 + names.append(patch.target) + insert.append(i) + delete.append(d) + namelen = max(namelen, len(patch.target)) + maxdiff = max(maxdiff, i+d) + output = '' + statlen = len(str(maxdiff)) # stats column width + for i,n in enumerate(names): + # %-19s | %-4d %s + format = " %-" + str(namelen) + "s | %" + str(statlen) + "s %s\n" + + hist = '' + # -- calculating histogram -- + width = len(format % ('', '', '')) + histwidth = max(2, 80 - width) + if maxdiff < histwidth: + hist = "+"*insert[i] + "-"*delete[i] + else: + iratio = (float(insert[i]) / maxdiff) * histwidth + dratio = (float(delete[i]) / maxdiff) * histwidth + + # make sure every entry gets at least one + or - + iwidth = 1 if 0 < iratio < 1 else int(iratio) + dwidth = 1 if 0 < dratio < 1 else int(dratio) + #print iratio, dratio, iwidth, dwidth, histwidth + hist = "+"*int(iwidth) + "-"*int(dwidth) + # -- /calculating +- histogram -- + output += (format % (names[i], insert[i] + delete[i], hist)) + + output += (" %d files changed, %d insertions(+), %d deletions(-)" + % (len(names), sum(insert), sum(delete))) + return output + + + def apply(self, strip=0): + """ apply parsed patch + return True on success + """ + + total = len(self.items) + errors = 0 + if strip: + # [ ] test strip level exceeds nesting level + # [ ] test the same only for selected files + # [ ] test if files end up being on the same level + try: + strip = int(strip) + except ValueError: + errors += 1 + warning("error: strip parameter '%s' must be an integer" % strip) + strip = 0 + + #for fileno, filename in enumerate(self.source): + for i,p in enumerate(self.items): + f2patch = p.source + if strip: + debug("stripping %s leading component from '%s'" % (strip, f2patch)) + f2patch = pathstrip(f2patch, strip) + if not exists(f2patch): + f2patch = p.target + if strip: + debug("stripping %s leading component from '%s'" % (strip, f2patch)) + f2patch = pathstrip(f2patch, strip) + if not exists(f2patch): + warning("source/target file does not exist\n--- %s\n+++ %s" % (p.source, f2patch)) + errors += 1 + continue + if not isfile(f2patch): + warning("not a file - %s" % f2patch) + errors += 1 + continue + filename = f2patch + + debug("processing %d/%d:\t %s" % (i+1, total, filename)) + + # validate before patching + f2fp = open(filename) + hunkno = 0 + hunk = p.hunks[hunkno] + hunkfind = [] + hunkreplace = [] + validhunks = 0 + canpatch = False + for lineno, line in enumerate(f2fp): + if lineno+1 < hunk.startsrc: + continue + elif lineno+1 == hunk.startsrc: + hunkfind = [x[1:].rstrip("\r\n") for x in hunk.text if x[0] in " -"] + hunkreplace = [x[1:].rstrip("\r\n") for x in hunk.text if x[0] in " +"] + #pprint(hunkreplace) + hunklineno = 0 + + # todo \ No newline at end of file + + # check hunks in source file + if lineno+1 < hunk.startsrc+len(hunkfind)-1: + if line.rstrip("\r\n") == hunkfind[hunklineno]: + hunklineno+=1 + else: + info("file %d/%d:\t %s" % (i+1, total, filename)) + info(" hunk no.%d doesn't match source file at line %d" % (hunkno+1, lineno)) + info(" expected: %s" % hunkfind[hunklineno]) + info(" actual : %s" % line.rstrip("\r\n")) + # not counting this as error, because file may already be patched. + # check if file is already patched is done after the number of + # invalid hunks if found + # TODO: check hunks against source/target file in one pass + # API - check(stream, srchunks, tgthunks) + # return tuple (srcerrs, tgterrs) + + # continue to check other hunks for completeness + hunkno += 1 + if hunkno < len(p.hunks): + hunk = p.hunks[hunkno] + continue + else: + break + + # check if processed line is the last line + if lineno+1 == hunk.startsrc+len(hunkfind)-1: + debug(" hunk no.%d for file %s -- is ready to be patched" % (hunkno+1, filename)) + hunkno+=1 + validhunks+=1 + if hunkno < len(p.hunks): + hunk = p.hunks[hunkno] + else: + if validhunks == len(p.hunks): + # patch file + canpatch = True + break + else: + if hunkno < len(p.hunks): + warning("premature end of source file %s at hunk %d" % (filename, hunkno+1)) + errors += 1 + + f2fp.close() + + if validhunks < len(p.hunks): + if self._match_file_hunks(filename, p.hunks): + warning("already patched %s" % filename) + else: + warning("source file is different - %s" % filename) + errors += 1 + if canpatch: + backupname = filename+".orig" + if exists(backupname): + warning("can't backup original file to %s - aborting" % backupname) + else: + import shutil + shutil.move(filename, backupname) + if self.write_hunks(backupname, filename, p.hunks): + info("successfully patched %d/%d:\t %s" % (i+1, total, filename)) + os.unlink(backupname) + else: + errors += 1 + warning("error patching file %s" % filename) + shutil.copy(filename, filename+".invalid") + warning("invalid version is saved to %s" % filename+".invalid") + # todo: proper rejects + shutil.move(backupname, filename) + + # todo: check for premature eof + return (errors == 0) + + + def can_patch(self, filename): + """ Check if specified filename can be patched. Returns None if file can + not be found among source filenames. False if patch can not be applied + clearly. True otherwise. + + :returns: True, False or None + """ + filename = abspath(filename) + for p in self.items: + if filename == abspath(p.source): + return self._match_file_hunks(filename, p.hunks) + return None + + + def _match_file_hunks(self, filepath, hunks): + matched = True + fp = open(abspath(filepath)) + + class NoMatch(Exception): + pass + + lineno = 1 + line = fp.readline() + hno = None + try: + for hno, h in enumerate(hunks): + # skip to first line of the hunk + while lineno < h.starttgt: + if not len(line): # eof + debug("check failed - premature eof before hunk: %d" % (hno+1)) + raise NoMatch + line = fp.readline() + lineno += 1 + for hline in h.text: + if hline.startswith("-"): + continue + if not len(line): + debug("check failed - premature eof on hunk: %d" % (hno+1)) + # todo: \ No newline at the end of file + raise NoMatch + if line.rstrip("\r\n") != hline[1:].rstrip("\r\n"): + debug("file is not patched - failed hunk: %d" % (hno+1)) + raise NoMatch + line = fp.readline() + lineno += 1 + + except NoMatch: + matched = False + # todo: display failed hunk, i.e. expected/found + + fp.close() + return matched + + + def patch_stream(self, instream, hunks): + """ Generator that yields stream patched with hunks iterable + + Converts lineends in hunk lines to the best suitable format + autodetected from input + """ + + # todo: At the moment substituted lineends may not be the same + # at the start and at the end of patching. Also issue a + # warning/throw about mixed lineends (is it really needed?) + + hunks = iter(hunks) + + srclineno = 1 + + lineends = {'\n':0, '\r\n':0, '\r':0} + def get_line(): + """ + local utility function - return line from source stream + collecting line end statistics on the way + """ + line = instream.readline() + # 'U' mode works only with text files + if line.endswith("\r\n"): + lineends["\r\n"] += 1 + elif line.endswith("\n"): + lineends["\n"] += 1 + elif line.endswith("\r"): + lineends["\r"] += 1 + return line + + for hno, h in enumerate(hunks): + debug("hunk %d" % (hno+1)) + if h.hasminus: + warning("Change removes/replaces some text; INVESTIGATE AND APPLY (OR NOT) MANUALLY") + warning("Change:") + changeText = h.originalText() + if len(changeText) > 1000: + changeText = changeText[0:999] + "...\n" + warning(changeText) + else: + # skip to line just before hunk starts + while srclineno < h.startsrc: + yield get_line() + srclineno += 1 + + for hline in h.text: + # todo: check \ No newline at the end of file + if hline.startswith("-") or hline.startswith("\\"): + get_line() + srclineno += 1 + continue + else: + if not hline.startswith("+"): + get_line() + srclineno += 1 + line2write = hline[1:] + # detect if line ends are consistent in source file + if sum([bool(lineends[x]) for x in lineends]) == 1: + newline = [x for x in lineends if lineends[x] != 0][0] + yield line2write.rstrip("\r\n")+newline + else: # newlines are mixed + yield line2write + + for line in instream: + yield line + + + def write_hunks(self, srcname, tgtname, hunks): + src = open(srcname, "rb") + tgt = open(tgtname, "wb") + + debug("processing target file %s" % tgtname) + + tgt.writelines(self.patch_stream(src, hunks)) + + tgt.close() + src.close() + # [ ] TODO: add test for permission copy + shutil.copymode(srcname, tgtname) + return True + + + +if __name__ == "__main__": + from optparse import OptionParser + from os.path import exists + import sys + + opt = OptionParser(usage="1. %prog [options] unified.diff\n" + " 2. %prog [options] http://host/patch\n" + " 3. %prog [options] -- < unified.diff", + version="python-patch %s" % __version__) + opt.add_option("-q", "--quiet", action="store_const", dest="verbosity", + const=0, help="print only warnings and errors", default=1) + opt.add_option("-v", "--verbose", action="store_const", dest="verbosity", + const=2, help="be verbose") + opt.add_option("--debug", action="store_true", dest="debugmode", help="debug mode") + opt.add_option("--diffstat", action="store_true", dest="diffstat", + help="print diffstat and exit") + opt.add_option("-p", "--strip", type="int", metavar='N', default=0, + help="strip N path components from filenames") + (options, args) = opt.parse_args() + + if not args and sys.argv[-1:] != ['--']: + opt.print_version() + opt.print_help() + sys.exit() + readstdin = (sys.argv[-1:] == ['--'] and not args) + + debugmode = options.debugmode + + verbosity_levels = {0:logging.WARNING, 1:logging.INFO, 2:logging.DEBUG} + loglevel = verbosity_levels[options.verbosity] + logformat = "%(message)s" + if debugmode: + loglevel = logging.DEBUG + logformat = "%(levelname)8s %(message)s" + logger.setLevel(loglevel) + loghandler = logging.StreamHandler() + loghandler.setFormatter(logging.Formatter(logformat)) + logger.addHandler(loghandler) + + + if readstdin: + patch = PatchSet(sys.stdin) + else: + patchfile = args[0] + urltest = patchfile.split(':')[0] + if (':' in patchfile and urltest.isalpha() + and len(urltest) > 1): # one char before : is a windows drive letter + patch = fromurl(patchfile) + else: + if not exists(patchfile) or not isfile(patchfile): + sys.exit("patch file does not exist - %s" % patchfile) + patch = fromfile(patchfile) + + if options.diffstat: + print patch.diffstat() + sys.exit(0) + + #pprint(patch) + patch.apply(options.strip) or sys.exit(-1) + + # todo: document and test line ends handling logic - patch.py detects proper line-endings + # for inserted hunks and issues a warning if patched file has incosistent line ends diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/conf.py --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -274,6 +274,6 @@ return Mock() # adding pbs_python, DRMAA_python, markupsafe, and drmaa here had no effect. -MOCK_MODULES = ['tables', 'decorator'] +MOCK_MODULES = ['tables', 'decorator', 'numpy'] for mod_name in MOCK_MODULES: sys.modules[mod_name] = Mock() diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/index.rst --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,12 +1,41 @@ Galaxy Code Documentation ************************* -Galaxy is an open, web-based platform for accessible, reproducible, and +Galaxy_ is an open, web-based platform for accessible, reproducible, and transparent computational biomedical research. -- Accessible: Users without programming experience can easily specify parameters and run tools and workflows. -- Reproducible: Galaxy captures information so that any user can repeat and understand a complete computational analysis. -- Transparent: Users share and publish analyses via the web and create Pages, interactive, web-based documents that describe a complete analysis. +- *Accessible:* Users without programming experience can easily specify parameters and run tools and workflows. +- *Reproducible:* Galaxy captures information so that any user can repeat and understand a complete computational analysis. +- *Transparent:* Users share and publish analyses via the web and create Pages, interactive, web-based documents that describe a complete analysis. + +Two copies of the Galaxy code doumentation are published by the Galaxy Project + +- Galaxy-Dist_: This describes the code in the `most recent official release`_ of Galaxy. +- Galaxy-Central_: Describes the `current code in the development branch`_ of Galaxy. This is the latest checkin, bleeding edge version of the code. The documentation should never be more than an hour behind the code. + +Both copies are hosted at ReadTheDocs_, a publicly supported web site for hosting project documentation. + +If you have your own copy of the Galaxy source code, you can also generate your own version of this documentation: + +:: + + $ cd doc + $ make html + +The generated documentation will be in ``doc/build/html/`` and can be viewed with a web browser. Note that you will need to install Sphinx and a fair number of module dependencies before this will produce output. + +.. _Galaxy: http://galaxyproject.org/ +.. _Galaxy-Dist: https://galaxy-dist.readthedocs.org/ +.. _most recent official release: https://bitbucket.org/galaxy/galaxy-dist +.. _Galaxy-Central: https://galaxy-central.readthedocs.org/ +.. _current code in the development branch: https://bitbucket.org/galaxy/galaxy-central +.. _ReadTheDocs: https://readthedocs.org/ + + +For more on the Galaxy Project, please visit the `project home page`_. + +.. _project home page: http://galaxyproject.org/ + Contents ======== diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.jobs.rst --- a/doc/source/lib/galaxy.jobs.rst +++ b/doc/source/lib/galaxy.jobs.rst @@ -48,6 +48,7 @@ galaxy.jobs.actions galaxy.jobs.deferred + galaxy.jobs.rules galaxy.jobs.runners galaxy.jobs.splitters diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.jobs.runners.rst --- a/doc/source/lib/galaxy.jobs.runners.rst +++ b/doc/source/lib/galaxy.jobs.runners.rst @@ -57,14 +57,6 @@ :undoc-members: :show-inheritance: -:mod:`sge` Module ------------------ - -.. automodule:: galaxy.jobs.runners.sge - :members: - :undoc-members: - :show-inheritance: - :mod:`tasks` Module ------------------- diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.tool_shed.rst --- a/doc/source/lib/galaxy.tool_shed.rst +++ b/doc/source/lib/galaxy.tool_shed.rst @@ -9,6 +9,14 @@ :undoc-members: :show-inheritance: +:mod:`common_util` Module +------------------------- + +.. automodule:: galaxy.tool_shed.common_util + :members: + :undoc-members: + :show-inheritance: + :mod:`encoding_util` Module --------------------------- diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.util.rst --- a/doc/source/lib/galaxy.util.rst +++ b/doc/source/lib/galaxy.util.rst @@ -25,6 +25,14 @@ :undoc-members: :show-inheritance: +:mod:`debugging` Module +----------------------- + +.. automodule:: galaxy.util.debugging + :members: + :undoc-members: + :show-inheritance: + :mod:`expressions` Module ------------------------- @@ -113,6 +121,14 @@ :undoc-members: :show-inheritance: +:mod:`shed_util_common` Module +------------------------------ + +.. automodule:: galaxy.util.shed_util_common + :members: + :undoc-members: + :show-inheritance: + :mod:`streamball` Module ------------------------ diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.webapps.community.rst --- a/doc/source/lib/galaxy.webapps.community.rst +++ b/doc/source/lib/galaxy.webapps.community.rst @@ -42,4 +42,5 @@ galaxy.webapps.community.framework galaxy.webapps.community.model galaxy.webapps.community.security + galaxy.webapps.community.util diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.webapps.community.util.rst --- /dev/null +++ b/doc/source/lib/galaxy.webapps.community.util.rst @@ -0,0 +1,27 @@ +util Package +============ + +:mod:`container_util` Module +---------------------------- + +.. automodule:: galaxy.webapps.community.util.container_util + :members: + :undoc-members: + :show-inheritance: + +:mod:`hgweb_config` Module +-------------------------- + +.. automodule:: galaxy.webapps.community.util.hgweb_config + :members: + :undoc-members: + :show-inheritance: + +:mod:`shed_statistics` Module +----------------------------- + +.. automodule:: galaxy.webapps.community.util.shed_statistics + :members: + :undoc-members: + :show-inheritance: + diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 doc/source/lib/galaxy.webapps.galaxy.api.rst --- a/doc/source/lib/galaxy.webapps.galaxy.api.rst +++ b/doc/source/lib/galaxy.webapps.galaxy.api.rst @@ -293,6 +293,14 @@ :undoc-members: :show-inheritance: +:mod:`item_tags` Module +----------------------- + +.. automodule:: galaxy.webapps.galaxy.api.item_tags + :members: + :undoc-members: + :show-inheritance: + :mod:`libraries` Module ----------------------- diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -29,6 +29,7 @@ simplejson = 2.1.1 threadframe = 0.2 guppy = 0.1.8 +; msgpack_python = 0.2.4 [eggs:noplatform] amqplib = 0.6.1 @@ -65,6 +66,7 @@ Babel = 0.9.4 wchartype = 0.1 Whoosh = 0.3.18 +; fluent_logger = 0.3.3 ; extra version information [tags] diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/app.py --- a/lib/galaxy/app.py +++ b/lib/galaxy/app.py @@ -23,11 +23,13 @@ """Encapsulates the state of a Universe application""" def __init__( self, **kwargs ): print >> sys.stderr, "python path is: " + ", ".join( sys.path ) + self.name = 'galaxy' self.new_installation = False # Read config file and check for errors self.config = config.Configuration( **kwargs ) self.config.check() config.configure_logging( self.config ) + self.configure_fluent_log() # Determine the database url if self.config.database_connection: db_url = self.config.database_connection @@ -53,7 +55,8 @@ db_url, self.config.database_engine_options, database_query_profiling_proxy = self.config.database_query_profiling_proxy, - object_store = self.object_store ) + object_store = self.object_store, + trace_logger=self.trace_logger ) # Manage installed tool shed repositories. self.installed_repository_manager = galaxy.tool_shed.InstalledRepositoryManager( self ) # Create an empty datatypes registry. @@ -148,6 +151,7 @@ self.job_stop_queue = self.job_manager.job_stop_queue # Initialize the external service types self.external_service_types = external_service_types.ExternalServiceTypesCollection( self.config.external_service_type_config_file, self.config.external_service_type_path, self ) + def shutdown( self ): self.job_manager.shutdown() self.object_store.shutdown() @@ -160,3 +164,10 @@ os.unlink( self.datatypes_registry.integrated_datatypes_configs ) except: pass + + def configure_fluent_log( self ): + if self.config.fluent_log: + from galaxy.util.log.fluent_log import FluentTraceLogger + self.trace_logger = FluentTraceLogger( 'galaxy', self.config.fluent_host, self.config.fluent_port ) + else: + self.trace_logger = None diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/config.py --- a/lib/galaxy/config.py +++ b/lib/galaxy/config.py @@ -261,6 +261,10 @@ self.api_folders = string_as_bool( kwargs.get( 'api_folders', False ) ) # This is for testing new library browsing capabilities. self.new_lib_browse = string_as_bool( kwargs.get( 'new_lib_browse', False ) ) + # Logging with fluentd + self.fluent_log = string_as_bool( kwargs.get( 'fluent_log', False ) ) + self.fluent_host = kwargs.get( 'fluent_host', 'localhost' ) + self.fluent_port = int( kwargs.get( 'fluent_port', 24224 ) ) def __read_tool_job_config( self, global_conf_parser, section, key ): try: diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/datatypes/registry.py --- a/lib/galaxy/datatypes/registry.py +++ b/lib/galaxy/datatypes/registry.py @@ -114,7 +114,8 @@ self.datatype_elems.remove( in_memory_elem ) else: # Keep an in-memory list of datatype elems to enable persistence. - self.datatype_elems.append( elem ) + if extension not in self.datatypes_by_extension: + self.datatype_elems.append( elem ) if extension and extension in self.datatypes_by_extension and deactivate: # We are deactivating an installed tool shed repository, so eliminate the datatype from the registry. # TODO: Handle deactivating datatype converters, etc before removing from self.datatypes_by_extension. diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/datatypes/tabular.py --- a/lib/galaxy/datatypes/tabular.py +++ b/lib/galaxy/datatypes/tabular.py @@ -267,20 +267,20 @@ def display_data(self, trans, dataset, preview=False, filename=None, to_ext=None, chunk=None): if chunk: return self.get_chunk(trans, dataset, chunk) + elif to_ext or not preview: + return self._serve_raw(trans, dataset, to_ext) elif dataset.metadata.columns > 50: #Fancy tabular display is only suitable for datasets without an incredibly large number of columns. #We should add a new datatype 'matrix', with it's own draw method, suitable for this kind of data. #For now, default to the old behavior, ugly as it is. Remove this after adding 'matrix'. max_peek_size = 1000000 # 1 MB - if not preview or os.stat( dataset.file_name ).st_size < max_peek_size: + if os.stat( dataset.file_name ).st_size < max_peek_size: return open( dataset.file_name ) else: trans.response.set_content_type( "text/html" ) return trans.stream_template_mako( "/dataset/large_file.mako", truncated_data = open( dataset.file_name ).read(max_peek_size), data = dataset) - elif to_ext or not preview: - return self._serve_raw(trans, dataset, to_ext) else: column_names = 'null' if dataset.metadata.column_names: diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py +++ b/lib/galaxy/jobs/__init__.py @@ -686,7 +686,7 @@ if self.app.config.set_metadata_externally: self.external_output_metadata.cleanup_external_metadata( self.sa_session ) galaxy.tools.imp_exp.JobExportHistoryArchiveWrapper( self.job_id ).cleanup_after_job( self.sa_session ) - galaxy.tools.imp_exp.JobImportHistoryArchiveWrapper( self.job_id ).cleanup_after_job( self.sa_session ) + galaxy.tools.imp_exp.JobImportHistoryArchiveWrapper( self.app, self.job_id ).cleanup_after_job() galaxy.tools.genome_index.GenomeIndexToolWrapper( self.job_id ).postprocessing( self.sa_session, self.app ) self.app.object_store.delete(self.get_job(), base_dir='job_work', entire_dir=True, dir_only=True, extra_dir=str(self.job_id)) except: diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/handler.py --- a/lib/galaxy/jobs/handler.py +++ b/lib/galaxy/jobs/handler.py @@ -61,7 +61,8 @@ # Helper for interruptable sleep self.sleeper = Sleeper() self.running = True - self.monitor_thread = threading.Thread( target=self.__monitor ) + self.monitor_thread = threading.Thread( name="JobHandlerQueue.monitor_thread", target=self.__monitor ) + self.monitor_thread.setDaemon( True ) def start( self ): """ @@ -353,7 +354,8 @@ # Helper for interruptable sleep self.sleeper = Sleeper() self.running = True - self.monitor_thread = threading.Thread( target=self.monitor ) + self.monitor_thread = threading.Thread( name="JobHandlerStopQueue.monitor_thread", target=self.monitor ) + self.monitor_thread.setDaemon( True ) self.monitor_thread.start() log.info( "job handler stop queue started" ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/manager.py --- a/lib/galaxy/jobs/manager.py +++ b/lib/galaxy/jobs/manager.py @@ -68,7 +68,8 @@ # Helper for interruptable sleep self.sleeper = Sleeper() self.running = True - self.monitor_thread = threading.Thread( target=self.__monitor ) + self.monitor_thread = threading.Thread( name="JobManagerQueue.monitor_thread", target=self.__monitor ) + self.monitor_thread.setDaemon( True ) # Recover jobs at startup self.__check_jobs_at_startup() # Start the queue @@ -219,7 +220,8 @@ # Helper for interruptable sleep self.sleeper = Sleeper() self.running = True - self.monitor_thread = threading.Thread( target=self.monitor ) + self.monitor_thread = threading.Thread( name="JobManagerStopQueue.monitor_thread", target=self.monitor ) + self.monitor_thread.setDaemon( True ) self.monitor_thread.start() log.info( "job manager stop queue started" ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py +++ b/lib/galaxy/jobs/runners/drmaa.py @@ -105,13 +105,14 @@ self.monitor_queue = Queue() self.ds = drmaa.Session() self.ds.initialize() - self.monitor_thread = threading.Thread( target=self.monitor ) + self.monitor_thread = threading.Thread( name="DRMAAJobRunner.monitor_thread", target=self.monitor ) + self.monitor_thread.setDaemon( True ) self.monitor_thread.start() self.work_queue = Queue() self.work_threads = [] nworkers = app.config.cluster_job_queue_workers for i in range( nworkers ): - worker = threading.Thread( target=self.run_next ) + worker = threading.Thread( name=( "DRMAAJobRunner.work_threads-%d" % i ), target=self.run_next ) worker.start() self.work_threads.append( worker ) log.debug( "%d workers ready" % nworkers ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/runners/local.py --- a/lib/galaxy/jobs/runners/local.py +++ b/lib/galaxy/jobs/runners/local.py @@ -37,7 +37,8 @@ nworkers = app.config.local_job_queue_workers log.info( "starting workers" ) for i in range( nworkers ): - worker = threading.Thread( target=self.run_next ) + worker = threading.Thread( name=( "LocalJobRunner.threads-%d" % i ), target=self.run_next ) + worker.setDaemon( True ) worker.start() self.threads.append( worker ) log.debug( "%d workers ready", nworkers ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/runners/lwr.py --- a/lib/galaxy/jobs/runners/lwr.py +++ b/lib/galaxy/jobs/runners/lwr.py @@ -229,7 +229,8 @@ nworkers = app.config.local_job_queue_workers log.info( "starting workers" ) for i in range( nworkers ): - worker = threading.Thread( target=self.run_next ) + worker = threading.Thread( ( name="LwrJobRunner.thread-%d" % i ), target=self.run_next ) + worker.setDaemon( True ) worker.start() self.threads.append( worker ) log.debug( "%d workers ready", nworkers ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/jobs/runners/tasks.py --- a/lib/galaxy/jobs/runners/tasks.py +++ b/lib/galaxy/jobs/runners/tasks.py @@ -29,7 +29,8 @@ nworkers = app.config.local_task_queue_workers log.info( "Starting tasked-job runners" ) for i in range( nworkers ): - worker = threading.Thread( target=self.run_next ) + worker = threading.Thread( name=( "TaskedJobRunner-%d" % i ), target=self.run_next ) + worker.setDaemon( True ) worker.start() self.threads.append( worker ) log.debug( "%d workers ready", nworkers ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py +++ b/lib/galaxy/model/__init__.py @@ -1335,7 +1335,9 @@ # Loop through sources until viable one is found. for source in source_list: msg = self.convert_dataset( trans, source ) - if msg == self.conversion_messages.PENDING: + # No message or PENDING means that source is viable. No + # message indicates conversion was done and is successful. + if not msg or msg == self.conversion_messages.PENDING: data_source = source break @@ -3014,6 +3016,7 @@ installation_status = Bunch( NEW='New', CLONING='Cloning', SETTING_TOOL_VERSIONS='Setting tool versions', + INSTALLING_REPOSITORY_DEPENDENCIES='Installing repository dependencies', INSTALLING_TOOL_DEPENDENCIES='Installing tool dependencies', LOADING_PROPRIETARY_DATATYPES='Loading proprietary datatypes', INSTALLED='Installed', @@ -3148,20 +3151,89 @@ def can_reinstall_or_activate( self ): return self.deleted @property + def has_repository_dependencies( self ): + if self.metadata: + return 'repository_dependencies' in self.metadata + return False + @property def includes_tools( self ): - return self.metadata and 'tools' in self.metadata + if self.metadata: + return 'tools' in self.metadata + return False @property def includes_tool_dependencies( self ): - return self.metadata and 'tool_dependencies' in self.metadata + if self.metadata: + return 'tool_dependencies' in self.metadata + return False @property def includes_workflows( self ): - return self.metadata and 'workflows' in self.metadata + if self.metadata: + return 'workflows' in self.metadata + return False @property def in_error_state( self ): return self.status == self.installation_status.ERROR @property def has_readme_files( self ): - return self.metadata and 'readme_files' in self.metadata + if self.metadata: + return 'readme_files' in self.metadata + return False + @property + def repository_dependencies( self ): + required_repositories = [] + for rrda in self.required_repositories: + repository_dependency = rrda.repository_dependency + required_repository = repository_dependency.repository + required_repositories.append( required_repository ) + return required_repositories + @property + def installed_repository_dependencies( self ): + """Return the repository's repository dependencies that are currently installed.""" + installed_required_repositories = [] + for required_repository in self.repository_dependencies: + if required_repository.status == self.installation_status.INSTALLED: + installed_required_repositories.append( required_repository ) + return installed_required_repositories + @property + def missing_repository_dependencies( self ): + """Return the repository's repository dependencies that are not currently installed, and may not ever have been installed.""" + missing_required_repositories = [] + for required_repository in self.repository_dependencies: + if required_repository.status not in [ self.installation_status.INSTALLED ]: + missing_required_repositories.append( required_repository ) + return missing_required_repositories + @property + def repository_dependencies_being_installed( self ): + required_repositories_being_installed = [] + for required_repository in self.repository_dependencies: + if tool_dependency.status == ToolDependency.installation_status.INSTALLING: + required_repositories_being_installed.append( required_repository ) + return required_repositories_being_installed + @property + def repository_dependencies_missing_or_being_installed( self ): + required_repositories_missing_or_being_installed = [] + for required_repository in self.repository_dependencies: + if required_repository.status in [ self.installation_status.ERROR, + self.installation_status.INSTALLING, + self.installation_status.NEVER_INSTALLED, + self.installation_status.UNINSTALLED ]: + required_repositories_missing_or_being_installed.append( required_repository ) + return required_repositories_missing_or_being_installed + @property + def repository_dependencies_with_installation_errors( self ): + required_repositories_with_installation_errors = [] + for required_repository in self.repository_dependencies: + if required_repository.status == self.installation_status.ERROR: + required_repositories_with_installation_errors.append( required_repository ) + return required_repositories_with_installation_errors + @property + def uninstalled_repository_dependencies( self ): + """Return the repository's repository dependencies that have been uninstalled.""" + uninstalled_required_repositories = [] + for required_repository in self.repository_dependencies: + if required_repository.status == self.installation_status.UNINSTALLED: + uninstalled_required_repositories.append( required_repository ) + return uninstalled_required_repositories @property def installed_tool_dependencies( self ): """Return the repository's tool dependencies that are currently installed.""" @@ -3211,6 +3283,15 @@ uninstalled_tool_dependencies.append( tool_dependency ) return uninstalled_tool_dependencies +class RepositoryRepositoryDependencyAssociation( object ): + def __init__( self, tool_shed_repository_id=None, repository_dependency_id=None ): + self.tool_shed_repository_id = tool_shed_repository_id + self.repository_dependency_id = repository_dependency_id + +class RepositoryDependency( object ): + def __init__( self, tool_shed_repository_id=None ): + self.tool_shed_repository_id = tool_shed_repository_id + class ToolDependency( object ): installation_status = Bunch( NEVER_INSTALLED='Never installed', INSTALLING='Installing', diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/model/mapping.py --- a/lib/galaxy/model/mapping.py +++ b/lib/galaxy/model/mapping.py @@ -409,6 +409,19 @@ Column( "status", TrimmedString( 255 ) ), Column( "error_message", TEXT ) ) +RepositoryRepositoryDependencyAssociation.table = Table( 'repository_repository_dependency_association', metadata, + Column( "id", Integer, primary_key=True ), + Column( "create_time", DateTime, default=now ), + Column( "update_time", DateTime, default=now, onupdate=now ), + Column( "tool_shed_repository_id", Integer, ForeignKey( "tool_shed_repository.id" ), index=True ), + Column( "repository_dependency_id", Integer, ForeignKey( "repository_dependency.id" ), index=True ) ) + +RepositoryDependency.table = Table( "repository_dependency", metadata, + Column( "id", Integer, primary_key=True ), + Column( "create_time", DateTime, default=now ), + Column( "update_time", DateTime, default=now, onupdate=now ), + Column( "tool_shed_repository_id", Integer, ForeignKey( "tool_shed_repository.id" ), index=True, nullable=False ) ) + ToolDependency.table = Table( "tool_dependency", metadata, Column( "id", Integer, primary_key=True ), Column( "create_time", DateTime, default=now ), @@ -1744,7 +1757,19 @@ tool_dependencies=relation( ToolDependency, primaryjoin=( ToolShedRepository.table.c.id == ToolDependency.table.c.tool_shed_repository_id ), order_by=ToolDependency.table.c.name, - backref='tool_shed_repository' ) ) ) + backref='tool_shed_repository' ), + required_repositories=relation( RepositoryRepositoryDependencyAssociation, + primaryjoin=( ToolShedRepository.table.c.id == RepositoryRepositoryDependencyAssociation.table.c.tool_shed_repository_id ) ) ) ) + +assign_mapper( context, RepositoryRepositoryDependencyAssociation, RepositoryRepositoryDependencyAssociation.table, + properties=dict( repository=relation( ToolShedRepository, + primaryjoin=( RepositoryRepositoryDependencyAssociation.table.c.tool_shed_repository_id == ToolShedRepository.table.c.id ) ), + repository_dependency=relation( RepositoryDependency, + primaryjoin=( RepositoryRepositoryDependencyAssociation.table.c.repository_dependency_id == RepositoryDependency.table.c.id ) ) ) ) + +assign_mapper( context, RepositoryDependency, RepositoryDependency.table, + properties=dict( repository=relation( ToolShedRepository, + primaryjoin=( RepositoryDependency.table.c.tool_shed_repository_id == ToolShedRepository.table.c.id ) ) ) ) assign_mapper( context, ToolDependency, ToolDependency.table ) @@ -1925,7 +1950,7 @@ # Let this go, it could possibly work with db's we don't support log.error( "database_connection contains an unknown SQLAlchemy database dialect: %s" % dialect ) -def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None ): +def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None, trace_logger=None ): """Connect mappings to the database""" # Connect dataset to the file path Dataset.file_path = file_path @@ -1937,6 +1962,10 @@ if database_query_profiling_proxy: import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy proxy = logging_connection_proxy.LoggingProxy() + # If metlog is enabled, do micrologging + elif trace_logger: + import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy + proxy = logging_connection_proxy.TraceLoggerProxy( trace_logger ) else: proxy = None # Create the database engine diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/model/migrate/versions/0109_add_repository_dependency_tables.py --- /dev/null +++ b/lib/galaxy/model/migrate/versions/0109_add_repository_dependency_tables.py @@ -0,0 +1,58 @@ +""" +Migration script to add the repository_dependency and repository_repository_dependency_association tables. +""" +from sqlalchemy import * +from sqlalchemy.orm import * +from migrate import * +from migrate.changeset import * +import sys, logging +from galaxy.model.custom_types import * +from sqlalchemy.exc import * +import datetime +now = datetime.datetime.utcnow + +log = logging.getLogger( __name__ ) +log.setLevel( logging.DEBUG ) +handler = logging.StreamHandler( sys.stdout ) +format = "%(name)s %(levelname)s %(asctime)s %(message)s" +formatter = logging.Formatter( format ) +handler.setFormatter( formatter ) +log.addHandler( handler ) + +metadata = MetaData( migrate_engine ) + +RepositoryDependency_table = Table( "repository_dependency", metadata, + Column( "id", Integer, primary_key=True ), + Column( "create_time", DateTime, default=now ), + Column( "update_time", DateTime, default=now, onupdate=now ), + Column( "tool_shed_repository_id", Integer, ForeignKey( "tool_shed_repository.id" ), index=True, nullable=False ) ) + +RepositoryRepositoryDependencyAssociation_table = Table( "repository_repository_dependency_association", metadata, + Column( "id", Integer, primary_key=True ), + Column( "create_time", DateTime, default=now ), + Column( "update_time", DateTime, default=now, onupdate=now ), + Column( "tool_shed_repository_id", Integer, ForeignKey( "tool_shed_repository.id" ), index=True ), + Column( "repository_dependency_id", Integer, ForeignKey( "repository_dependency.id" ), index=True ) ) + +def upgrade(): + print __doc__ + metadata.reflect() + try: + RepositoryDependency_table.create() + except Exception, e: + log.debug( "Creating repository_dependency table failed: %s" % str( e ) ) + try: + RepositoryRepositoryDependencyAssociation_table.create() + except Exception, e: + log.debug( "Creating repository_repository_dependency_association table failed: %s" % str( e ) ) + +def downgrade(): + metadata.reflect() + try: + RepositoryRepositoryDependencyAssociation_table.drop() + except Exception, e: + log.debug( "Dropping repository_repository_dependency_association table failed: %s" % str( e ) ) + try: + RepositoryDependency_table.drop() + except Exception, e: + log.debug( "Dropping repository_dependency table failed: %s" % str( e ) ) diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/model/orm/logging_connection_proxy.py --- a/lib/galaxy/model/orm/logging_connection_proxy.py +++ b/lib/galaxy/model/orm/logging_connection_proxy.py @@ -18,13 +18,31 @@ rval = [] for frame, fname, line, funcname, _, _ in inspect.stack()[2:]: rval.append( "%s:%s@%d" % ( stripwd( fname ), funcname, line ) ) - return " > ".join( rval ) + return rval class LoggingProxy(ConnectionProxy): + """ + Logs SQL statements using standard logging module + """ def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): start = time.clock() rval = execute(cursor, statement, parameters, context) duration = time.clock() - start log.debug( "statement: %r parameters: %r executemany: %r duration: %r stack: %r", - statement, parameters, executemany, duration, pretty_stack() ) + statement, parameters, executemany, duration, " > ".join( pretty_stack() ) ) return rval + +class TraceLoggerProxy(ConnectionProxy): + """ + Logs SQL statements using a metlog client + """ + def __init__( self, trace_logger ): + self.trace_logger = trace_logger + def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): + start = time.clock() + rval = execute(cursor, statement, parameters, context) + duration = time.clock() - start + self.trace_logger.log( "sqlalchemy_query", + message="Query executed", statement=statement, parameters=parameters, + executemany=executemany, duration=duration ) + return rval \ No newline at end of file diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/tool_shed/__init__.py --- a/lib/galaxy/tool_shed/__init__.py +++ b/lib/galaxy/tool_shed/__init__.py @@ -3,7 +3,8 @@ """ import os import galaxy.util.shed_util -from galaxy.model.orm import * +import galaxy.util.shed_util_common +from galaxy.model.orm import and_ from galaxy import eggs import pkg_resources @@ -27,7 +28,7 @@ ElementInclude.include( root ) tool_path = root.get( 'tool_path', None ) if tool_path: - tool_shed = galaxy.util.shed_util.clean_tool_shed_url( tool_shed_repository.tool_shed ) + tool_shed = galaxy.util.shed_util_common.clean_tool_shed_url( tool_shed_repository.tool_shed ) relative_path = os.path.join( tool_path, tool_shed, 'repos', diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/tool_shed/common_util.py --- a/lib/galaxy/tool_shed/common_util.py +++ b/lib/galaxy/tool_shed/common_util.py @@ -1,7 +1,7 @@ import os, urllib2 from galaxy import util from galaxy.util.odict import odict -from galaxy.tool_shed.encoding_util import * +from galaxy.tool_shed import encoding_util REPOSITORY_OWNER = 'devteam' @@ -36,7 +36,7 @@ print "The URL\n%s\nraised the exception:\n%s\n" % ( url, str( e ) ) if tool_shed_accessible: if text: - tool_dependencies_dict = tool_shed_decode( text ) + tool_dependencies_dict = encoding_util.tool_shed_decode( text ) for dependency_key, requirements_dict in tool_dependencies_dict.items(): tool_dependency_name = requirements_dict[ 'name' ] tool_dependency_version = requirements_dict[ 'version' ] diff -r 087a08c43cb5ac21b5c3f73240dbcde1355fa089 -r caaab03824478384c256fc6b5678bb39dbf8f9f2 lib/galaxy/tool_shed/encoding_util.py --- a/lib/galaxy/tool_shed/encoding_util.py +++ b/lib/galaxy/tool_shed/encoding_util.py @@ -1,5 +1,5 @@ -import binascii -from galaxy.util.hash_util import * +import binascii, logging +from galaxy.util.hash_util import hmac_new from galaxy.util.json import json_fix from galaxy import eggs @@ -8,7 +8,10 @@ pkg_resources.require( "simplejson" ) import simplejson +log = logging.getLogger( __name__ ) + encoding_sep = '__esep__' +encoding_sep2 = '__esepii__' def tool_shed_decode( value ): # Extract and verify hash @@ -21,12 +24,12 @@ try: values = simplejson.loads( value ) except Exception, e: - log.debug( "Decoding json value from tool shed threw exception: %s" % str( e ) ) + log.debug( "Decoding json value from tool shed for value '%s' threw exception: %s" % ( str( value ), str( e ) ) ) if values is not None: try: return json_fix( values ) except Exception, e: - log.debug( "Fixing decoded json value from tool shed threw exception: %s" % str( e ) ) + log.debug( "Fixing decoded json values '%s' from tool shed threw exception: %s" % ( str( values ), str( e ) ) ) fixed_values = values if values is None: values = value This diff is so big that we needed to truncate the remainder. Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Handle deactivated repository dependencies when installing or reinstalling a tool shed repository.
by Bitbucket 10 Jan '13

10 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/bcb446fd0248/ changeset: bcb446fd0248 user: greg date: 2013-01-10 00:08:51 summary: Handle deactivated repository dependencies when installing or reinstalling a tool shed repository. affected #: 2 files diff -r a91d8ea3c9e2e9d6cc8cfd85c7085bf1175b4f5c -r bcb446fd02481aaba2bbe72d8298d93ff6698c99 lib/galaxy/util/shed_util.py --- a/lib/galaxy/util/shed_util.py +++ b/lib/galaxy/util/shed_util.py @@ -23,6 +23,38 @@ log = logging.getLogger( __name__ ) +def activate_repository( trans, repository ): + repository_clone_url = suc.generate_clone_url_for_installed_repository( trans.app, repository ) + shed_tool_conf, tool_path, relative_install_dir = suc.get_tool_panel_config_tool_path_install_dir( trans.app, repository ) + repository.deleted = False + repository.status = trans.model.ToolShedRepository.installation_status.INSTALLED + if repository.includes_tools: + metadata = repository.metadata + repository_tools_tups = suc.get_repository_tools_tups( trans.app, metadata ) + # Reload tools into the appropriate tool panel section. + tool_panel_dict = repository.metadata[ 'tool_panel_section' ] + add_to_tool_panel( trans.app, + repository.name, + repository_clone_url, + repository.installed_changeset_revision, + repository_tools_tups, + repository.owner, + shed_tool_conf, + tool_panel_dict, + new_install=False ) + trans.sa_session.add( repository ) + trans.sa_session.flush() + if repository.includes_datatypes: + if tool_path: + repository_install_dir = os.path.abspath ( os.path.join( tool_path, relative_install_dir ) ) + else: + repository_install_dir = os.path.abspath ( relative_install_dir ) + # Activate proprietary datatypes. + installed_repository_dict = load_installed_datatypes( trans.app, repository, repository_install_dir, deactivate=False ) + if installed_repository_dict and 'converter_path' in installed_repository_dict: + load_installed_datatype_converters( trans.app, installed_repository_dict, deactivate=False ) + if installed_repository_dict and 'display_path' in installed_repository_dict: + load_installed_display_applications( trans.app, installed_repository_dict, deactivate=False ) def add_to_shed_tool_config( app, shed_tool_conf_dict, elem_list ): # A tool shed repository is being installed so change the shed_tool_conf file. Parse the config file to generate the entire list # of config_elems instead of using the in-memory list since it will be a subset of the entire list if one or more repositories have @@ -227,6 +259,10 @@ installed_changeset_revision = installed_tool_shed_repository.installed_changeset_revision metadata_dict = installed_tool_shed_repository.metadata dist_to_shed = installed_tool_shed_repository.dist_to_shed + elif installed_tool_shed_repository.status in [ trans.model.ToolShedRepository.installation_status.DEACTIVATED ]: + # The current tool shed repository is deactivated, so updating it's database record is not necessary - just activate it. + activate_repository( trans, installed_tool_shed_repository ) + can_update = False else: # The tool shed repository currently being processed is already installed or is in the process of being installed, so it's record # in the database cannot be updated. diff -r a91d8ea3c9e2e9d6cc8cfd85c7085bf1175b4f5c -r bcb446fd02481aaba2bbe72d8298d93ff6698c99 lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py --- a/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py +++ b/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py @@ -321,43 +321,19 @@ """Activate a repository that was deactivated but not uninstalled.""" repository_id = kwd[ 'id' ] repository = suc.get_installed_tool_shed_repository( trans, repository_id ) - shed_tool_conf, tool_path, relative_install_dir = suc.get_tool_panel_config_tool_path_install_dir( trans.app, repository ) - repository_clone_url = suc.generate_clone_url_for_installed_repository( trans.app, repository ) - repository.deleted = False - repository.status = trans.model.ToolShedRepository.installation_status.INSTALLED - if repository.includes_tools: - metadata = repository.metadata - try: - repository_tools_tups = suc.get_repository_tools_tups( trans.app, metadata ) - except Exception, e: - error = "Error activating repository %s: %s" % ( repository.name, str( e ) ) - log.debug( error ) - return trans.show_error_message( '%s.<br/>You may be able to resolve this by uninstalling and then reinstalling the repository. Click <a href="%s">here</a> to uninstall the repository.' - % ( error, web.url_for( controller='admin_toolshed', action='deactivate_or_uninstall_repository', id=trans.security.encode_id( repository.id ) ) ) ) - # Reload tools into the appropriate tool panel section. - tool_panel_dict = repository.metadata[ 'tool_panel_section' ] - shed_util.add_to_tool_panel( trans.app, - repository.name, - repository_clone_url, - repository.installed_changeset_revision, - repository_tools_tups, - repository.owner, - shed_tool_conf, - tool_panel_dict, - new_install=False ) - trans.sa_session.add( repository ) - trans.sa_session.flush() - if repository.includes_datatypes: - if tool_path: - repository_install_dir = os.path.abspath ( os.path.join( tool_path, relative_install_dir ) ) - else: - repository_install_dir = os.path.abspath ( relative_install_dir ) - # Activate proprietary datatypes. - installed_repository_dict = shed_util.load_installed_datatypes( trans.app, repository, repository_install_dir, deactivate=False ) - if installed_repository_dict and 'converter_path' in installed_repository_dict: - shed_util.load_installed_datatype_converters( trans.app, installed_repository_dict, deactivate=False ) - if installed_repository_dict and 'display_path' in installed_repository_dict: - shed_util.load_installed_display_applications( trans.app, installed_repository_dict, deactivate=False ) + try: + shed_util.activate_repository( trans, repository ) + except Exception, e: + error_message = "Error activating repository %s: %s" % ( repository.name, str( e ) ) + log.debug( error_message ) + message = '%s.<br/>You may be able to resolve this by uninstalling and then reinstalling the repository. Click <a href="%s">here</a> to uninstall the repository.' \ + % ( error_message, web.url_for( controller='admin_toolshed', action='deactivate_or_uninstall_repository', id=trans.security.encode_id( repository.id ) ) ) + status = 'error' + return trans.response.send_redirect( web.url_for( controller='admin_toolshed', + action='manage_repository', + id=repository_id, + message=message, + status=status ) ) message = 'The <b>%s</b> repository has been activated.' % repository.name status = 'done' return trans.response.send_redirect( web.url_for( controller='admin_toolshed', Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: inithello: Tool shed functional tests for installing repositories with circular dependencies into separate tool panel sections.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/a91d8ea3c9e2/ changeset: a91d8ea3c9e2 user: inithello date: 2013-01-09 22:20:07 summary: Tool shed functional tests for installing repositories with circular dependencies into separate tool panel sections. affected #: 2 files diff -r a7049f42a9d9b4bef6804dee53a2bc8bcc1a2916 -r a91d8ea3c9e2e9d6cc8cfd85c7085bf1175b4f5c test/tool_shed/base/twilltestcase.py --- a/test/tool_shed/base/twilltestcase.py +++ b/test/tool_shed/base/twilltestcase.py @@ -60,6 +60,10 @@ def check_for_valid_tools( self, repository, strings_displayed=[], strings_not_displayed=[] ): strings_displayed.append( 'Valid tools' ) self.display_manage_repository_page( repository, strings_displayed, strings_not_displayed ) + def check_galaxy_repository_db_status( self, repository_name, owner, expected_status ): + installed_repository = test_db_util.get_installed_repository_by_name_owner( repository_name, owner ) + assert installed_repository.status == expected_status, 'Status in database is %s, expected %s' % \ + ( installed_repository.status, expected_status ) def check_galaxy_repository_tool_panel_section( self, repository, expected_tool_panel_section ): metadata = repository.metadata assert 'tools' in metadata, 'Tools not found in metadata: %s' % metadata diff -r a7049f42a9d9b4bef6804dee53a2bc8bcc1a2916 -r a91d8ea3c9e2e9d6cc8cfd85c7085bf1175b4f5c test/tool_shed/functional/test_1040_install_repository_basic_circular_dependencies.py --- /dev/null +++ b/test/tool_shed/functional/test_1040_install_repository_basic_circular_dependencies.py @@ -0,0 +1,149 @@ +from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os +import tool_shed.base.test_db_util as test_db_util + +freebayes_repository_name = 'freebayes_0040' +freebayes_repository_description = "Galaxy's freebayes tool" +freebayes_repository_long_description = "Long description of Galaxy's freebayes tool" + +filtering_repository_name = 'filtering_0040' +filtering_repository_description = "Galaxy's filtering tool" +filtering_repository_long_description = "Long description of Galaxy's filtering tool" + +category_name = 'test_0040_repository_circular_dependencies' + +running_standalone = False + +class TestInstallingCircularDependencies( ShedTwillTestCase ): + '''Verify that the code correctly handles installing repositories with circular dependencies.''' + def test_0000_initiate_users( self ): + """Create necessary user accounts.""" + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + test_user_1 = test_db_util.get_user( common.test_user_1_email ) + assert test_user_1 is not None, 'Problem retrieving user with email %s from the database' % test_user_1_email + test_user_1_private_role = test_db_util.get_private_role( test_user_1 ) + self.logout() + self.login( email=common.admin_email, username=common.admin_username ) + admin_user = test_db_util.get_user( common.admin_email ) + assert admin_user is not None, 'Problem retrieving user with email %s from the database' % admin_email + admin_user_private_role = test_db_util.get_private_role( admin_user ) + def test_0005_create_freebayes_repository( self ): + '''Create and populate freebayes_0040.''' + global running_standalone + category = self.create_category( name=category_name, description='Testing handling of circular repository dependencies.' ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=freebayes_repository_name, + description=freebayes_repository_description, + long_description=freebayes_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + if self.repository_is_new( repository ): + running_standalone = True + self.upload_file( repository, + 'freebayes/freebayes.tar', + strings_displayed=[], + commit_message='Uploaded freebayes.tar.' ) + def test_0015_create_filtering_repository( self ): + '''Create and populate filtering_0040.''' + global running_standalone + category = self.create_category( name=category_name, description='Testing handling of circular repository dependencies.' ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=filtering_repository_name, + description=filtering_repository_description, + long_description=filtering_repository_long_description, + owner=common.test_user_1_name, + categories=[ 'test_0040_repository_circular_dependencies' ], + strings_displayed=[] ) + if self.repository_is_new( repository ): + running_standalone = True + self.upload_file( repository, + 'filtering/filtering_1.1.0.tar', + strings_displayed=[], + commit_message='Uploaded filtering.tar.' ) + def test_0020_create_repository_dependencies( self ): + '''Set up the filtering and freebayes repository dependencies.''' + # The dependency structure should look like: + # Filtering revision 0 -> freebayes revision 0. + # Freebayes revision 0 -> filtering revision 1. + # Filtering will have two revisions, one with just the filtering tool, and one with the filtering tool and a dependency on freebayes. + global running_standalone + if running_standalone: + repository = test_db_util.get_repository_by_name_and_owner( freebayes_repository_name, common.test_user_1_name ) + filtering_repository = test_db_util.get_repository_by_name_and_owner( filtering_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_0040', additional_paths=[ 'filtering' ] ) + self.generate_repository_dependency_xml( [ repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Filtering 1.1.0 depends on the freebayes repository.' ) + self.upload_file( filtering_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on freebayes' ) + repository = test_db_util.get_repository_by_name_and_owner( filtering_repository_name, common.test_user_1_name ) + freebayes_repository = test_db_util.get_repository_by_name_and_owner( freebayes_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_0040', additional_paths=[ 'freebayes' ] ) + self.generate_repository_dependency_xml( [ repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Freebayes depends on the filtering repository.' ) + self.upload_file( freebayes_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on filtering' ) + def test_0025_install_freebayes_repository( self ): + '''Install freebayes with blank tool panel section, without tool dependencies but with repository dependencies.''' + self.galaxy_logout() + self.galaxy_login( email=common.admin_email, username=common.admin_username ) + strings_displayed = [ 'Handle', 'tool dependencies', 'freebayes', '0.9.4_9696d0ce8a9', 'samtools', '0.1.18' ] + self.install_repository( freebayes_repository_name, + common.test_user_1_name, + category_name, + strings_displayed=strings_displayed, + install_tool_dependencies=False, + install_repository_dependencies=True ) + def test_0030_uninstall_freebayes_repository( self ): + '''Uninstall freebayes, verify tool panel section and missing repository dependency.''' + installed_freebayes_repository = test_db_util.get_installed_repository_by_name_owner( freebayes_repository_name, common.test_user_1_name ) + installed_filtering_repository = test_db_util.get_installed_repository_by_name_owner( filtering_repository_name, common.test_user_1_name ) + self.uninstall_repository( installed_freebayes_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_freebayes_repository ) + self.check_galaxy_repository_tool_panel_section( installed_freebayes_repository, '' ) + strings_displayed = [ 'Missing repository', 'freebayes' ] + self.display_installed_repository_manage_page( installed_filtering_repository, strings_displayed=strings_displayed ) + self.check_galaxy_repository_db_status( freebayes_repository_name, + common.test_user_1_name, + 'Uninstalled' ) + def test_0035_reinstall_freebayes_repository( self ): + '''Reinstall freebayes into 'freebayes' tool panel section.''' + installed_freebayes_repository = test_db_util.get_installed_repository_by_name_owner( freebayes_repository_name, + common.test_user_1_name ) + self.reinstall_repository( installed_freebayes_repository, + install_tool_dependencies=False, + install_repository_dependencies=True, + new_tool_panel_section='freebayes', + no_changes=False ) + def test_0040_uninstall_filtering_repository( self ): + '''Uninstall filtering, verify tool panel section.''' + installed_filtering_repository = test_db_util.get_installed_repository_by_name_owner( filtering_repository_name, common.test_user_1_name ) + installed_freebayes_repository = test_db_util.get_installed_repository_by_name_owner( freebayes_repository_name, common.test_user_1_name ) + self.uninstall_repository( installed_filtering_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_filtering_repository ) + self.check_galaxy_repository_tool_panel_section( installed_filtering_repository, '' ) + strings_displayed = [ 'Missing repository', 'filtering' ] + self.display_installed_repository_manage_page( installed_freebayes_repository, strings_displayed=strings_displayed ) + self.check_galaxy_repository_db_status( filtering_repository_name, + common.test_user_1_name, + 'Uninstalled' ) + def test_0045_uninstall_freebayes_repository( self ): + '''Uninstall freebayes, verify tool panel section and missing repository dependency.''' + installed_freebayes_repository = test_db_util.get_installed_repository_by_name_owner( freebayes_repository_name, common.test_user_1_name ) + installed_filtering_repository = test_db_util.get_installed_repository_by_name_owner( filtering_repository_name, common.test_user_1_name ) + self.uninstall_repository( installed_freebayes_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_freebayes_repository ) + self.check_galaxy_repository_tool_panel_section( installed_freebayes_repository, 'freebayes' ) + strings_displayed = [ 'Missing repository', 'freebayes' ] + self.display_installed_repository_manage_page( installed_filtering_repository, strings_displayed=strings_displayed ) + self.check_galaxy_repository_db_status( freebayes_repository_name, + common.test_user_1_name, + 'Uninstalled' ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Add parameter to Tophat2 so that output BAM is ordered using reference contigs ordering rather than sorted lexicographically.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/a7049f42a9d9/ changeset: a7049f42a9d9 user: jgoecks date: 2013-01-09 19:36:33 summary: Add parameter to Tophat2 so that output BAM is ordered using reference contigs ordering rather than sorted lexicographically. affected #: 1 file diff -r 184c5e409425da36556de7edda910a7ffe6bc9d0 -r a7049f42a9d9b4bef6804dee53a2bc8bcc1a2916 tools/ngs_rna/tophat2_wrapper.py --- a/tools/ngs_rna/tophat2_wrapper.py +++ b/tools/ngs_rna/tophat2_wrapper.py @@ -134,7 +134,7 @@ index_path = options.index_path # Build tophat command. - cmd = 'tophat2 %s %s %s' + cmd = 'tophat2 --keep-fasta-order %s %s %s' reads = options.input1 if options.input2: reads += ' ' + options.input2 Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: inithello: Tool shed functional test enhancements - verify tool dependency display for all relevant repositories.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/184c5e409425/ changeset: 184c5e409425 user: inithello date: 2013-01-09 19:27:44 summary: Tool shed functional test enhancements - verify tool dependency display for all relevant repositories. affected #: 10 files diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_0020_basic_repository_dependencies.py --- a/test/tool_shed/functional/test_0020_basic_repository_dependencies.py +++ b/test/tool_shed/functional/test_0020_basic_repository_dependencies.py @@ -63,12 +63,19 @@ 'repository_dependencies.xml', filepath=repository_dependencies_path, commit_message='Uploaded repository_dependencies.xml' ) - def test_0030_verify_emboss_5_repository_dependency_on_emboss_datatypes( self ): + def test_0030_verify_emboss_5_dependencies( self ): '''Verify that the emboss_5 repository now depends on the emboss_datatypes repository with correct name, owner, and changeset revision.''' repository = test_db_util.get_repository_by_name_and_owner( emboss_repository_name, common.test_user_1_name ) datatypes_repository = test_db_util.get_repository_by_name_and_owner( datatypes_repository_name, common.test_user_1_name ) changeset_revision = self.get_repository_tip( datatypes_repository ) - strings_displayed = [ datatypes_repository_name, common.test_user_1_name, changeset_revision, 'Repository dependencies' ] + strings_displayed = [ 'Tool dependencies', + 'emboss', + '5.0.0', + 'package', + datatypes_repository_name, + common.test_user_1_name, + changeset_revision, + 'Repository dependencies' ] self.display_manage_repository_page( repository, strings_displayed=strings_displayed ) def test_0040_verify_repository_metadata( self ): '''Verify that resetting the metadata does not change it.''' diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_0030_repository_dependency_revisions.py --- a/test/tool_shed/functional/test_0030_repository_dependency_revisions.py +++ b/test/tool_shed/functional/test_0030_repository_dependency_revisions.py @@ -108,17 +108,18 @@ datatypes_tip = self.get_repository_tip( datatypes_repository ) # Iterate through all metadata revisions and check for repository dependencies. for metadata, changeset_revision in repository_metadata: - repository_dependency_metadata = metadata[ 'repository_dependencies' ][ 'repository_dependencies' ][ 0 ] + strings_displayed = [ str( metadata_elem ) for metadata_elem in metadata[ 'repository_dependencies' ][ 'repository_dependencies' ][ 0 ] ] # Remove the tool shed URL, because that's not displayed on the page (yet?) - repository_dependency_metadata.pop( repository_dependency_metadata.index( self.url ) ) + strings_displayed.pop( strings_displayed.index( self.url ) ) # Add the dependency description and datatypes repository details to the strings to check. - repository_dependency_metadata.extend( [ metadata[ 'repository_dependencies' ][ 'description' ], - datatypes_repository_name, - datatypes_repository.user.username, - datatypes_tip ] ) + strings_displayed.extend( [ metadata[ 'repository_dependencies' ][ 'description' ], + datatypes_repository_name, + datatypes_repository.user.username, + datatypes_tip ] ) + strings_displayed.extend( [ 'Tool dependencies', 'emboss', '5.0.0', 'package' ] ) self.display_manage_repository_page( repository, changeset_revision=changeset_revision, - strings_displayed=[ str( metadata ) for metadata in repository_dependency_metadata ] ) + strings_displayed=strings_displayed ) def test_0040_verify_repository_metadata( self ): '''Verify that resetting the metadata does not change it.''' emboss_repository = test_db_util.get_repository_by_name_and_owner( emboss_repository_name, common.test_user_1_name ) diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_0040_repository_circular_dependencies.py --- a/test/tool_shed/functional/test_0040_repository_circular_dependencies.py +++ b/test/tool_shed/functional/test_0040_repository_circular_dependencies.py @@ -103,3 +103,9 @@ filtering_repository = test_db_util.get_repository_by_name_and_owner( filtering_repository_name, common.test_user_1_name ) for repository in [ freebayes_repository, filtering_repository ]: self.verify_unchanged_repository_metadata( repository ) + def test_0040_verify_tool_dependencies( self ): + '''Verify that freebayes displays tool dependencies.''' + repository = test_db_util.get_repository_by_name_and_owner( freebayes_repository_name, common.test_user_1_name ) + self.display_manage_repository_page( repository, + strings_displayed=[ 'freebayes', '0.9.4_9696d0ce8a9', 'samtools', '0.1.18', 'Valid tools' ], + strings_not_displayed=[ 'Invalid tools' ] ) diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_0050_circular_dependencies_4_levels.py --- a/test/tool_shed/functional/test_0050_circular_dependencies_4_levels.py +++ b/test/tool_shed/functional/test_0050_circular_dependencies_4_levels.py @@ -148,7 +148,14 @@ self.check_repository_dependency( freebayes_repository, emboss_repository, changeset_revision ) self.display_manage_repository_page( freebayes_repository, strings_displayed=[ 'Freebayes depends on the filtering repository.' ] ) - def test_0035_verify_repository_metadata( self ): + def test_0035_verify_tool_dependencies( self ): + '''Check that freebayes and emboss display tool dependencies.''' + freebayes_repository = test_db_util.get_repository_by_name_and_owner( freebayes_repository_name, common.test_user_1_name ) + emboss_repository = test_db_util.get_repository_by_name_and_owner( emboss_repository_name, common.test_user_1_name ) + self.display_manage_repository_page( freebayes_repository, + strings_displayed=[ 'freebayes', '0.9.4_9696d0ce8a9', 'samtools', '0.1.18', 'Tool dependencies' ] ) + self.display_manage_repository_page( emboss_repository, strings_displayed=[ 'Tool dependencies', 'emboss', '5.0.0', 'package' ] ) + def test_0040_verify_repository_metadata( self ): '''Verify that resetting the metadata does not change it.''' emboss_datatypes_repository = test_db_util.get_repository_by_name_and_owner( emboss_datatypes_repository_name, common.test_user_1_name ) emboss_repository = test_db_util.get_repository_by_name_and_owner( emboss_repository_name, common.test_user_1_name ) diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_1010_install_repository_with_tool_dependencies.py --- a/test/tool_shed/functional/test_1010_install_repository_with_tool_dependencies.py +++ b/test/tool_shed/functional/test_1010_install_repository_with_tool_dependencies.py @@ -1,9 +1,11 @@ from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os import tool_shed.base.test_db_util as test_db_util - +import logging repository_name = 'freebayes_0010' -repository_description="Galaxy's freebayes tool", -repository_long_description="Long description of Galaxy's freebayes tool", +repository_description="Galaxy's freebayes tool" +repository_long_description="Long description of Galaxy's freebayes tool" +category_name = 'Test 0010 Repository With Tool Dependencies' +log = logging.getLogger( __name__ ) class ToolWithToolDependencies( ShedTwillTestCase ): '''Test installing a repository with tool dependencies.''' @@ -26,7 +28,7 @@ admin_user_private_role = test_db_util.get_private_role( admin_user ) def test_0005_ensure_repositories_and_categories_exist( self ): '''Create the 0010 category and upload the freebayes repository to it, if necessary.''' - category = self.create_category( name='Test 0010 Repository With Tool Dependencies', description='Tests for a repository with tool dependencies.' ) + category = self.create_category( name=category_name, description='Tests for a repository with tool dependencies.' ) self.logout() self.login( email=common.test_user_1_email, username=common.test_user_1_name ) repository = self.get_or_create_repository( name=repository_name, @@ -63,16 +65,18 @@ """Browse the available tool sheds in this Galaxy instance and preview the freebayes tool.""" self.galaxy_logout() self.galaxy_login( email=common.admin_email, username=common.admin_username ) - self.browse_tool_shed( url=self.url, strings_displayed=[ 'Test 0010 Repository With Tool Dependencies' ] ) - category = test_db_util.get_category_by_name( 'Test 0010 Repository With Tool Dependencies' ) + self.browse_tool_shed( url=self.url, strings_displayed=[ category_name ] ) + category = test_db_util.get_category_by_name( category_name ) self.browse_category( category, strings_displayed=[ repository_name ] ) - self.preview_repository_in_tool_shed( repository_name, common.test_user_1_name, strings_displayed=[ repository_name, 'Valid tools', 'Tool dependencies' ] ) + strings_displayed = [ repository_name, 'Valid tools', 'Tool dependencies' ] + self.preview_repository_in_tool_shed( repository_name, common.test_user_1_name, strings_displayed=strings_displayed ) def test_0015_install_freebayes_repository( self ): '''Install the freebayes repository without installing tool dependencies.''' strings_displayed=[ 'set your tool_dependency_dir', 'can be automatically installed', 'Set the tool_dependency_dir' ] + strings_displayed.extend( [ 'Handle', 'tool dependencies', 'freebayes', '0.9.4_9696d0ce8a9', 'samtools', '0.1.18' ] ) self.install_repository( repository_name, common.test_user_1_name, - 'Test 0010 Repository With Tool Dependencies', + category_name, strings_displayed=strings_displayed, install_tool_dependencies=False, new_tool_panel_section='test_1010' ) diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_1020_install_repository_with_repository_dependencies.py --- a/test/tool_shed/functional/test_1020_install_repository_with_repository_dependencies.py +++ b/test/tool_shed/functional/test_1020_install_repository_with_repository_dependencies.py @@ -74,9 +74,11 @@ '''Install the emboss repository without installing tool dependencies.''' global repository_datatypes_count global base_datatypes_count + strings_displayed = [ 'Handle', 'Missing', 'tool dependencies', 'emboss', '5.0.0', 'package' ] self.install_repository( 'emboss_0020', common.test_user_1_name, - 'Test 0020 Basic Repository Dependencies', + 'Test 0020 Basic Repository Dependencies', + strings_displayed=strings_displayed, install_tool_dependencies=False, new_tool_panel_section='test_1020' ) installed_repository = test_db_util.get_installed_repository_by_name_owner( 'emboss_0020', common.test_user_1_name ) diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_1030_install_repository_with_dependency_revisions.py --- a/test/tool_shed/functional/test_1030_install_repository_with_dependency_revisions.py +++ b/test/tool_shed/functional/test_1030_install_repository_with_dependency_revisions.py @@ -113,9 +113,11 @@ global base_datatypes_count global running_standalone base_datatypes_count = int( self.get_datatypes_count() ) + strings_displayed = [ 'Handle', 'Missing', 'tool dependencies', 'emboss', '5.0.0', 'package' ] self.install_repository( 'emboss_0030', common.test_user_1_name, - 'Test 0030 Repository Dependency Revisions', + 'Test 0030 Repository Dependency Revisions', + strings_displayed=strings_displayed, install_tool_dependencies=False, new_tool_panel_section='test_1030' ) installed_repository = test_db_util.get_installed_repository_by_name_owner( 'emboss_0030', common.test_user_1_name ) diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_1210_uninstall_reinstall_repository_with_tool_dependencies.py --- a/test/tool_shed/functional/test_1210_uninstall_reinstall_repository_with_tool_dependencies.py +++ b/test/tool_shed/functional/test_1210_uninstall_reinstall_repository_with_tool_dependencies.py @@ -59,9 +59,11 @@ '''Install the freebayes repository into the Galaxy instance.''' self.galaxy_logout() self.galaxy_login( email=common.admin_email, username=common.admin_username ) + strings_displayed = [ 'Handle', 'tool dependencies', 'freebayes', '0.9.4_9696d0ce8a9', 'samtools', '0.1.18' ] self.install_repository( 'freebayes_0010', common.test_user_1_name, 'Test 0010 Repository With Tool Dependencies', + strings_displayed=strings_displayed, new_tool_panel_section='test_1210' ) installed_repository = test_db_util.get_installed_repository_by_name_owner( 'freebayes_0010', common.test_user_1_name ) strings_displayed = [ installed_repository.name, diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_1220_uninstall_reinstall_repository_with_repository_dependencies.py --- a/test/tool_shed/functional/test_1220_uninstall_reinstall_repository_with_repository_dependencies.py +++ b/test/tool_shed/functional/test_1220_uninstall_reinstall_repository_with_repository_dependencies.py @@ -70,9 +70,11 @@ self.galaxy_logout() self.galaxy_login( email=common.admin_email, username=common.admin_username ) base_datatypes_count = int( self.get_datatypes_count() ) + strings_displayed = [ 'Handle', 'Missing', 'tool dependencies', 'emboss', '5.0.0', 'package' ] self.install_repository( emboss_repository_name, common.test_user_1_name, - 'Test 0020 Basic Repository Dependencies', + 'Test 0020 Basic Repository Dependencies', + strings_displayed=strings_displayed, new_tool_panel_section='test_1210' ) installed_repository = test_db_util.get_installed_repository_by_name_owner( emboss_repository_name, common.test_user_1_name ) strings_displayed = [ installed_repository.name, diff -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 -r 184c5e409425da36556de7edda910a7ffe6bc9d0 test/tool_shed/functional/test_1230_uninstall_reinstall_repository_with_dependency_revisions.py --- a/test/tool_shed/functional/test_1230_uninstall_reinstall_repository_with_dependency_revisions.py +++ b/test/tool_shed/functional/test_1230_uninstall_reinstall_repository_with_dependency_revisions.py @@ -107,9 +107,11 @@ self.galaxy_logout() self.galaxy_login( email=common.admin_email, username=common.admin_username ) base_datatypes_count = int( self.get_datatypes_count() ) + strings_displayed = [ 'Handle', 'Missing', 'tool dependencies', 'emboss', '5.0.0', 'package' ] self.install_repository( emboss_repository_name, common.test_user_1_name, 'Test 0030 Repository Dependency Revisions', + strings_displayed=strings_displayed, new_tool_panel_section='test_1210' ) installed_repository = test_db_util.get_installed_repository_by_name_owner( emboss_repository_name, common.test_user_1_name ) strings_displayed = [ installed_repository.name, Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Create and use function for checking a slug's validity.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/b97bdb63115c/ changeset: b97bdb63115c user: jgoecks date: 2013-01-09 17:27:53 summary: Create and use function for checking a slug's validity. affected #: 3 files diff -r 757217fbae1981bc66d05076ab93dcbc92dca072 -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 lib/galaxy/web/base/controller.py --- a/lib/galaxy/web/base/controller.py +++ b/lib/galaxy/web/base/controller.py @@ -28,8 +28,12 @@ # States for passing messages SUCCESS, INFO, WARNING, ERROR = "done", "info", "warning", "error" -# RE that tests for valid slug. -VALID_SLUG_RE = re.compile( "^[a-z0-9\-]+$" ) +def _is_valid_slug( slug ): + """ Returns true if slug is valid. """ + + VALID_SLUG_RE = re.compile( "^[a-z0-9\-]+$" ) + return VALID_SLUG_RE.match( slug ) + class BaseController( object ): """ @@ -186,6 +190,7 @@ self.type_extension = type_extension self.mimetype = mimetype self.display_in_upload = display_in_upload + # # -- Mixins for working with Galaxy objects. -- # @@ -210,6 +215,7 @@ raise ItemAccessibilityException( "%s is not accessible to the current user" % item.__class__.__name__, type='error' ) return item + class UsesHistoryDatasetAssociationMixin: """ Mixin for controllers that use HistoryDatasetAssociation objects. """ @@ -552,7 +558,7 @@ title_err = slug_err = "" if not title: title_err = "visualization name is required" - elif slug and not VALID_SLUG_RE.match( slug ): + elif slug and not _is_valid_slug( slug ): slug_err = "visualization identifier must consist of only lowercase letters, numbers, and the '-' character" elif slug and trans.sa_session.query( trans.model.Visualization ).filter_by( user=user, slug=slug, deleted=False ).first(): slug_err = "visualization identifier must be unique" @@ -1465,6 +1471,10 @@ """ Mixin for a controller that manages an item that can be shared. """ # -- Implemented methods. -- + + def _is_valid_slug( self, slug ): + """ Returns true if slug is valid. """ + return _is_valid_slug( slug ) @web.expose @web.require_login( "share Galaxy items" ) diff -r 757217fbae1981bc66d05076ab93dcbc92dca072 -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 lib/galaxy/webapps/galaxy/controllers/page.py --- a/lib/galaxy/webapps/galaxy/controllers/page.py +++ b/lib/galaxy/webapps/galaxy/controllers/page.py @@ -2,7 +2,7 @@ from galaxy import model, web from galaxy.web import error, url_for from galaxy.model.item_attrs import UsesAnnotations, UsesItemRatings -from galaxy.web.base.controller import BaseUIController, SharableMixin, UsesHistoryMixin, UsesStoredWorkflowMixin, UsesVisualizationMixin, VALID_SLUG_RE +from galaxy.web.base.controller import BaseUIController, SharableMixin, UsesHistoryMixin, UsesStoredWorkflowMixin, UsesVisualizationMixin from galaxy.web.framework.helpers import time_ago, grids from galaxy import util from galaxy.util.sanitize_html import sanitize_html, _BaseHTMLProcessor @@ -342,7 +342,7 @@ page_title_err = "Page name is required" elif not page_slug: page_slug_err = "Page id is required" - elif not VALID_SLUG_RE.match( page_slug ): + elif not self._is_valid_slug( page_slug ): page_slug_err = "Page identifier must consist of only lowercase letters, numbers, and the '-' character" elif trans.sa_session.query( model.Page ).filter_by( user=user, slug=page_slug, deleted=False ).first(): page_slug_err = "Page id must be unique" @@ -398,7 +398,7 @@ page_title_err = "Page name is required" elif not page_slug: page_slug_err = "Page id is required" - elif not VALID_SLUG_RE.match( page_slug ): + elif not self._is_valid_slug( page_slug ): page_slug_err = "Page identifier must consist of only lowercase letters, numbers, and the '-' character" elif page_slug != page.slug and trans.sa_session.query( model.Page ).filter_by( user=user, slug=page_slug, deleted=False ).first(): page_slug_err = "Page id must be unique" diff -r 757217fbae1981bc66d05076ab93dcbc92dca072 -r b97bdb63115cf50562e423ac165cffcdd5bb64e1 lib/galaxy/webapps/galaxy/controllers/visualization.py --- a/lib/galaxy/webapps/galaxy/controllers/visualization.py +++ b/lib/galaxy/webapps/galaxy/controllers/visualization.py @@ -3,7 +3,7 @@ from sqlalchemy import desc, or_, and_ from galaxy import model, web from galaxy.model.item_attrs import UsesAnnotations, UsesItemRatings -from galaxy.web.base.controller import BaseUIController, SharableMixin, UsesVisualizationMixin, VALID_SLUG_RE +from galaxy.web.base.controller import BaseUIController, SharableMixin, UsesVisualizationMixin from galaxy.web.framework.helpers import time_ago, grids, iff from galaxy import util from galaxy.util.json import from_json_string @@ -636,7 +636,7 @@ visualization_title_err = "Visualization name is required" elif not visualization_slug: visualization_slug_err = "Visualization id is required" - elif not VALID_SLUG_RE.match( visualization_slug ): + elif not self._is_valid_slug( visualization_slug ): visualization_slug_err = "Visualization identifier must consist of only lowercase letters, numbers, and the '-' character" elif visualization_slug != visualization.slug and trans.sa_session.query( model.Visualization ).filter_by( user=visualization.user, slug=visualization_slug, deleted=False ).first(): visualization_slug_err = "Visualization id must be unique" Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Fix bug that prevented datasources from being returned correctly.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/757217fbae19/ changeset: 757217fbae19 user: jgoecks date: 2013-01-09 17:15:11 summary: Fix bug that prevented datasources from being returned correctly. affected #: 1 file diff -r 6ef9b038e329b6ee4e71ef1d1b1a22b9b2000ca1 -r 757217fbae1981bc66d05076ab93dcbc92dca072 lib/galaxy/model/__init__.py --- a/lib/galaxy/model/__init__.py +++ b/lib/galaxy/model/__init__.py @@ -1335,7 +1335,9 @@ # Loop through sources until viable one is found. for source in source_list: msg = self.convert_dataset( trans, source ) - if msg == self.conversion_messages.PENDING: + # No message or PENDING means that source is viable. No + # message indicates conversion was done and is successful. + if not msg or msg == self.conversion_messages.PENDING: data_source = source break Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by Bitbucket 09 Jan '13

09 Jan '13

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/0b2c782ca2d7/ changeset: 0b2c782ca2d7 user: inithello date: 2013-01-09 15:51:53 summary: Fix colspan on installed repository dependency folder. affected #: 1 file diff -r 857a60e69c7aee92f67f37311a40a825b40a6481 -r 0b2c782ca2d7a62f6a4e0781c8b729225806b442 templates/webapps/community/repository/common.mako --- a/templates/webapps/community/repository/common.mako +++ b/templates/webapps/community/repository/common.mako @@ -209,7 +209,7 @@ else: folder_label = "%s<i> - this repository's tools require handling of these missing dependencies</i>" % folder_label col_span_str = 'colspan="5"' - elif folder.label in [ 'Repository dependencies', 'Missing repository dependencies' ]: + elif folder.label in [ 'Installed repository dependencies', 'Repository dependencies', 'Missing repository dependencies' ]: if folder.description: folder_label = "%s<i> - %s</i>" % ( folder_label, folder.description ) else: @@ -245,7 +245,7 @@ </a></div></span> - <td> + </td></tr><% my_row = row_counter.count https://bitbucket.org/galaxy/galaxy-central/commits/6ef9b038e329/ changeset: 6ef9b038e329 user: inithello date: 2013-01-09 15:52:46 summary: Tool shed functional test enhancements - circular repository dependencies, tool panel sections. affected #: 5 files diff -r 0b2c782ca2d7a62f6a4e0781c8b729225806b442 -r 6ef9b038e329b6ee4e71ef1d1b1a22b9b2000ca1 test/tool_shed/base/twilltestcase.py --- a/test/tool_shed/base/twilltestcase.py +++ b/test/tool_shed/base/twilltestcase.py @@ -47,6 +47,9 @@ def browse_tool_shed( self, url, strings_displayed=[], strings_not_displayed=[] ): self.visit_galaxy_url( '/admin_toolshed/browse_tool_shed?tool_shed_url=%s' % url ) self.check_for_strings( strings_displayed, strings_not_displayed ) + def check_count_of_metadata_revisions_associated_with_repository( self, repository, metadata_count ): + self.check_repository_changelog( repository ) + self.check_string_count_in_page( 'Repository metadata is associated with this change set.', metadata_count ) def check_for_strings( self, strings_displayed=[], strings_not_displayed=[] ): if strings_displayed: for string in strings_displayed: @@ -57,9 +60,24 @@ def check_for_valid_tools( self, repository, strings_displayed=[], strings_not_displayed=[] ): strings_displayed.append( 'Valid tools' ) self.display_manage_repository_page( repository, strings_displayed, strings_not_displayed ) - def check_count_of_metadata_revisions_associated_with_repository( self, repository, metadata_count ): - self.check_repository_changelog( repository ) - self.check_string_count_in_page( 'Repository metadata is associated with this change set.', metadata_count ) + def check_galaxy_repository_tool_panel_section( self, repository, expected_tool_panel_section ): + metadata = repository.metadata + assert 'tools' in metadata, 'Tools not found in metadata: %s' % metadata + tool_metadata = metadata[ 'tools' ] + # If integrated_tool_panel.xml is to be tested, this test method will need to be enhanced to handle tools + # from the same repository in different tool panel sections. Getting the first tool guid is ok, because + # currently all tools contained in a single repository will be loaded into the same tool panel section. + tool_guid = tool_metadata[ 0 ][ 'guid' ] + assert 'tool_panel_section' in metadata, 'Tool panel section not found in metadata: %s' % metadata + tool_panel_section_metadata = metadata[ 'tool_panel_section' ] + # tool_section_dict = dict( tool_config=guids_and_configs[ guid ], + # id=section_id, + # name=section_name, + # version=section_version ) + # This dict is appended to tool_panel_section_metadata[ tool_guid ] + tool_panel_section = tool_panel_section_metadata[ tool_guid ][ 0 ][ 'name' ] + assert tool_panel_section == expected_tool_panel_section, 'Expected tool panel section %s, found %s\nMetadata: %s\n' % \ + ( expected_tool_panel_section, tool_panel_section, metadata ) def check_installed_repository_tool_dependencies( self, installed_repository, dependencies_installed=False ): # Tool dependencies are not being installed in these functional tests. If this is changed, the test method will also need to be updated. strings_not_displayed = [] @@ -150,6 +168,12 @@ self.visit_url( '/admin/manage_categories?operation=create' ) self.submit_form( form_no=1, button="create_category_button", **kwd ) return test_db_util.get_category_by_name( kwd[ 'name' ] ) + def create_checkbox_query_string( self, field_name, value ): + field_value = str( value ).lower() + if value: + return '%s=%s&%s=%s' % ( field_name, field_value, field_name, field_value ) + else: + return '%s=%s' % ( field_name, field_value ) def create_user_in_galaxy( self, cntrller='user', email='test(a)bx.psu.edu', password='testuser', username='admin-user', redirect='' ): self.visit_galaxy_url( "/user/create?cntrller=%s&use_panels=False" % cntrller ) tc.fv( '1', 'email', email ) @@ -306,7 +330,6 @@ self.visit_galaxy_url( "/user/logout" ) self.check_page_for_string( "You have been logged out" ) self.home() - def generate_repository_dependency_xml( self, repositories, xml_filename, dependency_description='' ): file_path = os.path.split( xml_filename )[0] if not os.path.exists( file_path ): @@ -369,7 +392,6 @@ request_param_path = base_path else: request_param_path = os.path.join( base_path, current_path ) - #request_param_path = request_param_path.replace( '/', '%2f' ) # Get the current folder's contents. url = '/repository/open_folder?folder_path=%s' % request_param_path self.visit_url( url ) @@ -431,7 +453,11 @@ if workflow_name not in strings_displayed: strings_displayed.append( workflow_name ) self.check_for_strings( strings_displayed, strings_not_displayed ) - def initiate_installation_process( self, install_tool_dependencies=False, install_repository_dependencies=True ): + def initiate_installation_process( self, + install_tool_dependencies=False, + install_repository_dependencies=True, + no_changes=True, + new_tool_panel_section=None ): html = self.last_page() # Since the installation process is by necessity asynchronous, we have to get the parameters to 'manually' initiate the # installation process. This regex will return the tool shed repository IDs in group(1), the encoded_kwd parameter in @@ -441,22 +467,21 @@ if install_parameters: iri_ids = install_parameters.group(1) # In some cases, the returned iri_ids are of the form: "[u'<encoded id>', u'<encoded id>']" - # This ensures that non-hex characters are stripped out of the list, so that util.listify/decode_id will handle them correctly. + # This regex ensures that non-hex characters are stripped out of the list, so that util.listify/decode_id + # will handle them correctly. It's safe to pass the cleaned list to manage_repositories, because it can parse + # comma-separated values. repository_ids = str( iri_ids ) repository_ids = re.sub( '[^a-fA-F0-9,]+', '', repository_ids ) - decoded_kwd = tool_shed_decode( install_parameters.group(2) ) - if 'install_tool_dependencies' in decoded_kwd: - decoded_kwd[ 'install_tool_dependencies' ] = install_tool_dependencies - if 'install_repository_dependencies' in decoded_kwd: - decoded_kwd[ 'install_repository_dependencies' ] = install_repository_dependencies + encoded_kwd = install_parameters.group(2) reinstalling = install_parameters.group(3) url = '/admin_toolshed/manage_repositories?operation=install&tool_shed_repository_ids=%s&encoded_kwd=%s&reinstalling=%s' % \ - ( ','.join( util.listify( repository_ids ) ), tool_shed_encode( decoded_kwd ), reinstalling ) + ( ','.join( util.listify( repository_ids ) ), encoded_kwd, reinstalling ) self.visit_galaxy_url( url ) return util.listify( repository_ids ) def install_repository( self, name, owner, category_name, install_tool_dependencies=False, - changeset_revision=None, strings_displayed=[], strings_not_displayed=[], - preview_strings_displayed=[], post_submit_strings_displayed=[], **kwd ): + install_repository_dependencies=True, changeset_revision=None, + strings_displayed=[], strings_not_displayed=[], preview_strings_displayed=[], + post_submit_strings_displayed=[], new_tool_panel_section=None, **kwd ): self.browse_tool_shed( url=self.url ) self.browse_category( test_db_util.get_category_by_name( category_name ) ) self.preview_repository_in_tool_shed( name, owner, strings_displayed=preview_strings_displayed ) @@ -468,6 +493,8 @@ ( changeset_revision, repository_id, self.galaxy_url ) self.visit_url( url ) self.check_for_strings( strings_displayed, strings_not_displayed ) + # This section is tricky, due to the way twill handles form submission. The tool dependency checkbox needs to + # be hacked in through tc.browser, putting the form field in kwd doesn't work. if 'install_tool_dependencies' in self.last_page(): form = tc.browser.get_form( 'select_tool_panel_section' ) checkbox = form.find_control( id="install_tool_dependencies" ) @@ -476,11 +503,15 @@ checkbox.selected = True else: checkbox.selected = False + if 'install_repository_dependencies' in self.last_page(): + kwd[ 'install_repository_dependencies' ] = str( install_repository_dependencies ).lower() if 'shed_tool_conf' not in kwd: kwd[ 'shed_tool_conf' ] = self.shed_tool_conf + if new_tool_panel_section: + kwd[ 'new_tool_panel_section' ] = new_tool_panel_section self.submit_form( 1, 'select_tool_panel_section_button', **kwd ) self.check_for_strings( post_submit_strings_displayed, strings_not_displayed ) - repository_ids = self.initiate_installation_process( install_tool_dependencies ) + repository_ids = self.initiate_installation_process( new_tool_panel_section=new_tool_panel_section ) self.wait_for_repository_installation( repository_ids ) def load_invalid_tool_page( self, repository, tool_xml, changeset_revision, strings_displayed=[], strings_not_displayed=[] ): url = '/repository/load_invalid_tool?repository_id=%s&tool_config=%s&changeset_revision=%s' % \ @@ -523,14 +554,40 @@ self.check_for_strings( strings_displayed, [] ) def reinstall_repository( self, installed_repository, - install_repository_dependencies='true', - install_tool_dependencies='false' ): + install_repository_dependencies=True, + install_tool_dependencies=False, + no_changes=True, + new_tool_panel_section='' ): url = '/admin_toolshed/reselect_tool_panel_section?id=%s' % self.security.encode_id( installed_repository.id ) self.visit_galaxy_url( url ) - url = '/admin_toolshed/reinstall_repository?id=%s&install_repository_dependencies=%s&install_repository_dependencies=%s' % \ - ( self.security.encode_id( installed_repository.id ), install_repository_dependencies, install_repository_dependencies ) + # From galaxy.web.form_builder.CheckboxField: + # The hidden field is necessary because if the check box is not checked on the form, it will + # not be included in the request params. The hidden field ensure that this will happen. When + # parsing the request, the value 'true' in the hidden field actually means it is NOT checked. + # See the is_checked() method below. The prefix is necessary in each case to ensure functional + # correctness when the param is inside a conditional. + # + # This may look strange upon initial inspection, but see the comments in the get_html() method + # above for clarification. Basically, if value is not True, then it will always be a list with + # 2 input fields ( a checkbox and a hidden field ) if the checkbox is checked. If it is not + # checked, then value will be only the hidden field. + # + # The create_checkbox_query_string method emulates the described behavior with URL query parameters. + # This is currently necessary because twill does not correctly parse the reselect tool panel section + # form, so the test method has to visit the intended form target "manually". + repo_dependencies = self.create_checkbox_query_string( field_name='install_repository_dependencies', value=install_repository_dependencies ) + tool_dependencies = self.create_checkbox_query_string( field_name='install_tool_dependencies', value=install_tool_dependencies ) + encoded_repository_id = self.security.encode_id( installed_repository.id ) + url = '/admin_toolshed/reinstall_repository?id=%s&%s&%s&no_changes=%s&new_tool_panel_section=%s' % \ + ( encoded_repository_id, repo_dependencies, tool_dependencies, str( no_changes ), new_tool_panel_section ) self.visit_galaxy_url( url ) - repository_ids = self.initiate_installation_process( install_tool_dependencies, install_repository_dependencies ) + # Then manually initiate the install process, as with installing a repository. See comments in the + # initiate_installation_process method for details. + repository_ids = self.initiate_installation_process( install_tool_dependencies, + install_repository_dependencies, + no_changes, + new_tool_panel_section ) + # Finally, wait until all repositories are in a final state (either Error or Installed) before returning. self.wait_for_repository_installation( repository_ids ) def repository_is_new( self, repository ): repo = hg.repository( ui.ui(), self.get_repo_path( repository ) ) @@ -619,6 +676,14 @@ def verify_installed_repository_data_table_entries( self, data_tables=[] ): data_table = util.parse_xml( self.shed_tool_data_table_conf ) found = False + # Tool data table xml structure: + # <tables> + #  + # <table name="all_fasta" comment_char="#"> + # <columns>value, dbkey, name, path</columns> + # <file path="tool-data/all_fasta.loc" /> + # </table> + # </tables> for table_elem in data_table.findall( 'table' ): for data_table in data_tables: if 'name' in table_elem.attrib and table_elem.attrib[ 'name' ] == data_table: @@ -643,6 +708,8 @@ self.reset_repository_metadata( repository ) for metadata in self.get_repository_metadata( repository ): new_metadata[ metadata.changeset_revision ] = metadata.metadata + # Python's dict comparison recursively compares sorted key => value pairs and returns true if any key or value differs, + # or if the number of keys differs. assert old_metadata == new_metadata, 'Metadata changed after reset on repository %s.' % repository.name def view_installed_workflow( self, repository, workflow_name, strings_displayed=[], strings_not_displayed=[] ): url = '/admin_toolshed/view_workflow?repository_id=%s&workflow_name=%s' % \ @@ -655,6 +722,8 @@ def wait_for_repository_installation( self, repository_ids ): final_states = [ galaxy_model.ToolShedRepository.installation_status.ERROR, galaxy_model.ToolShedRepository.installation_status.INSTALLED ] + # Wait until all repositories are in a final state before returning. This ensures that subsequent tests + # are running against an installed repository, and not one that is still in the process of installing. if repository_ids: for repository_id in repository_ids: galaxy_repository = test_db_util.get_installed_repository_by_id( self.security.decode_id( repository_id ) ) @@ -662,6 +731,7 @@ while galaxy_repository.status not in final_states: test_db_util.ga_refresh( galaxy_repository ) timeout_counter = timeout_counter + 1 + # This timeout currently defaults to 180 seconds, or 3 minutes. if timeout_counter > common.repository_installation_timeout: raise AssertionError( 'Repository installation timed out, %d seconds elapsed, repository state is %s.' % \ ( timeout_counter, repository.status ) ) diff -r 0b2c782ca2d7a62f6a4e0781c8b729225806b442 -r 6ef9b038e329b6ee4e71ef1d1b1a22b9b2000ca1 test/tool_shed/functional/test_1010_install_repository_with_tool_dependencies.py --- a/test/tool_shed/functional/test_1010_install_repository_with_tool_dependencies.py +++ b/test/tool_shed/functional/test_1010_install_repository_with_tool_dependencies.py @@ -1,6 +1,10 @@ from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os import tool_shed.base.test_db_util as test_db_util +repository_name = 'freebayes_0010' +repository_description="Galaxy's freebayes tool", +repository_long_description="Long description of Galaxy's freebayes tool", + class ToolWithToolDependencies( ShedTwillTestCase ): '''Test installing a repository with tool dependencies.''' def test_0000_initiate_users( self ): @@ -25,9 +29,9 @@ category = self.create_category( name='Test 0010 Repository With Tool Dependencies', description='Tests for a repository with tool dependencies.' ) self.logout() self.login( email=common.test_user_1_email, username=common.test_user_1_name ) - repository = self.get_or_create_repository( name='freebayes_0010', - description="Galaxy's freebayes tool", - long_description="Long description of Galaxy's freebayes tool", + repository = self.get_or_create_repository( name=repository_name, + description=repository_description, + long_description=repository_long_description, owner=common.test_user_1_name, category_id=self.security.encode_id( category.id ) ) if self.repository_is_new( repository ): @@ -61,18 +65,18 @@ self.galaxy_login( email=common.admin_email, username=common.admin_username ) self.browse_tool_shed( url=self.url, strings_displayed=[ 'Test 0010 Repository With Tool Dependencies' ] ) category = test_db_util.get_category_by_name( 'Test 0010 Repository With Tool Dependencies' ) - self.browse_category( category, strings_displayed=[ 'freebayes_0010' ] ) - self.preview_repository_in_tool_shed( 'freebayes_0010', common.test_user_1_name, strings_displayed=[ 'freebayes_0010', 'Valid tools', 'Tool dependencies' ] ) + self.browse_category( category, strings_displayed=[ repository_name ] ) + self.preview_repository_in_tool_shed( repository_name, common.test_user_1_name, strings_displayed=[ repository_name, 'Valid tools', 'Tool dependencies' ] ) def test_0015_install_freebayes_repository( self ): '''Install the freebayes repository without installing tool dependencies.''' strings_displayed=[ 'set your tool_dependency_dir', 'can be automatically installed', 'Set the tool_dependency_dir' ] - self.install_repository( 'freebayes_0010', + self.install_repository( repository_name, common.test_user_1_name, 'Test 0010 Repository With Tool Dependencies', strings_displayed=strings_displayed, install_tool_dependencies=False, new_tool_panel_section='test_1010' ) - installed_repository = test_db_util.get_installed_repository_by_name_owner( 'freebayes_0010', common.test_user_1_name ) + installed_repository = test_db_util.get_installed_repository_by_name_owner( repository_name, common.test_user_1_name ) strings_displayed = [ installed_repository.name, installed_repository.description, installed_repository.owner, @@ -85,7 +89,7 @@ self.verify_tool_metadata_for_installed_repository( installed_repository ) def test_0020_verify_installed_repository_metadata( self ): '''Verify that resetting the metadata on an installed repository does not change the metadata.''' - self.verify_installed_repository_metadata_unchanged( 'freebayes_0010', common.test_user_1_name ) + self.verify_installed_repository_metadata_unchanged( repository_name, common.test_user_1_name ) def test_0025_verify_sample_files( self ): '''Verify that the installed repository populated shed_tool_data_table.xml and the sample files.''' self.verify_installed_repository_data_table_entries( data_tables=[ 'sam_fa_indexes' ] ) diff -r 0b2c782ca2d7a62f6a4e0781c8b729225806b442 -r 6ef9b038e329b6ee4e71ef1d1b1a22b9b2000ca1 test/tool_shed/functional/test_1080_advanced_circular_dependency_installation.py --- a/test/tool_shed/functional/test_1080_advanced_circular_dependency_installation.py +++ b/test/tool_shed/functional/test_1080_advanced_circular_dependency_installation.py @@ -1,4 +1,4 @@ -from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os +from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os, logging import tool_shed.base.test_db_util as test_db_util column_repository_name = 'column_maker_0080' @@ -12,6 +12,10 @@ category_name = 'Test 0080 Advanced Circular Dependencies' category_description = 'Test circular dependency features' +log = logging.getLogger( __name__ ) + +running_standalone = False + class TestRepositoryDependencies( ShedTwillTestCase ): '''Testing freebayes with tool data table entries, .loc files, and tool dependencies.''' def test_0000_create_or_login_admin_user( self ): @@ -31,9 +35,10 @@ admin_user = test_db_util.get_user( common.admin_email ) assert admin_user is not None, 'Problem retrieving user with email %s from the database' % admin_email admin_user_private_role = test_db_util.get_private_role( admin_user ) - def test_0005_initiate_test_data( self ): - """Create a category for this test suite and add repositories to it.""" + def test_0005_create_and_populate_column_repository( self ): + """Create the category for this test suite, then create and populate column_maker.""" category = self.create_category( name=category_name, description=category_description ) + global running_standalone self.logout() self.login( email=common.test_user_1_email, username=common.test_user_1_name ) repository = self.get_or_create_repository( name=column_repository_name, @@ -47,16 +52,31 @@ 'column_maker/column_maker.tar', strings_displayed=[], commit_message='Uploaded column_maker.tar.' ) - repository = self.get_or_create_repository( name=convert_repository_name, - description=convert_repository_description, - long_description=convert_repository_long_description, - owner=common.test_user_1_name, - category_id=self.security.encode_id( category.id ), - strings_displayed=[] ) + running_standalone = True + def test_0010_create_and_populate_convert_repository( self ): + '''Create and populate the convert_chars repository.''' + global running_standalone + self.logout() + self.login( email=common.admin_email, username=common.admin_username ) + category = self.create_category( name=category_name, description=category_description ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=convert_repository_name, + description=convert_repository_description, + long_description=convert_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + if self.repository_is_new( repository ): self.upload_file( repository, 'convert_chars/convert_chars.tar', strings_displayed=[], commit_message='Uploaded convert_chars.tar.' ) + running_standalone = True + def test_0015_upload_dependency_xml_if_needed( self ): + '''If this test is being run by itself, it will not have repository dependencies configured yet.''' + global running_standalone + if running_standalone: convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) repository_dependencies_path = self.generate_temp_path( 'test_1080', additional_paths=[ 'convert' ] ) @@ -77,20 +97,20 @@ 'repository_dependencies.xml', filepath=repository_dependencies_path, commit_message='Uploaded dependency on column' ) - def test_0010_install_repositories( self ): - '''Install convert_chars with repository dependencies check box - this should install both convert_chars and column_maker.''' + def test_0020_install_convert_repository( self ): + '''Install convert_chars without repository dependencies into convert_chars tool panel section.''' self.galaxy_logout() self.galaxy_login( email=common.admin_email, username=common.admin_username ) self.install_repository( convert_repository_name, common.test_user_1_name, category_name, install_tool_dependencies=False, - install_repository_dependencies='Yes', - new_tool_panel_section='test_1080' ) + install_repository_dependencies=False, + new_tool_panel_section='convert_chars' ) installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, - common.test_user_1_name ) + common.test_user_1_name ) browse_strings_displayed = [ installed_convert_repository.name, installed_convert_repository.description, installed_convert_repository.tool_shed, @@ -101,13 +121,41 @@ installed_convert_repository.installed_changeset_revision, installed_column_repository.name, installed_column_repository.installed_changeset_revision, - 'Installed' ] - self.display_galaxy_browse_repositories_page( strings_displayed=strings_displayed ) - strings_displayed.append( 'Installed repository dependencies' ) + 'Missing repository dependencies' ] + self.display_galaxy_browse_repositories_page( strings_displayed=browse_strings_displayed ) self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) - def test_0015_deactivate_convert_repository( self ): - '''Deactivate convert_chars - this should display column_maker as installed but missing repository dependencies''' + def test_0025_install_column_repository( self ): + '''Install column maker with repository dependencies into column_maker tool panel section.''' + self.install_repository( column_repository_name, + common.test_user_1_name, + category_name, + install_repository_dependencies=True, + new_tool_panel_section='column_maker', + strings_displayed=[ 'install_repository_dependencies' ] ) + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + browse_strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.installed_changeset_revision ] + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_galaxy_browse_repositories_page( strings_displayed=browse_strings_displayed ) + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + def test_0030_deactivate_convert_repository( self ): + '''Deactivate convert_chars, verify that column_maker is installed and missing repository dependencies.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -123,8 +171,8 @@ 'Deactivated' ] self.display_installed_repository_manage_page( installed_column_repository, strings_displayed=strings_displayed ) - def test_0020_reactivate_convert_repository( self ): - '''Activate convert_chars - this should display both convert_chars and column_maker as installed with a green box''' + def test_0035_reactivate_convert_repository( self ): + '''Reactivate convert_chars, both convert_chars and column_maker should now show as installed.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -137,8 +185,8 @@ installed_column_repository.name, installed_column_repository.installed_changeset_revision, 'Installed repository dependencies' ] - def test_0025_deactivate_column_repository( self ): - '''Deactivate column_maker - this should display convert_chars installed but missing repository dependencies''' + def test_0040_deactivate_column_repository( self ): + '''Deactivate column_maker, verify that convert_chars is installed and missing repository dependencies.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -154,8 +202,8 @@ 'Deactivated' ] self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) - def test_0030_deactivate_convert_repository( self ): - '''Deactivate convert_chars - both convert_chars and column_maker are deactivated''' + def test_0045_deactivate_convert_repository( self ): + '''Deactivate convert_chars, verify that both convert_chars and column_maker are deactivated.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -168,8 +216,8 @@ installed_convert_repository.description, installed_convert_repository.installed_changeset_revision ] self.display_galaxy_browse_repositories_page( strings_not_displayed=strings_not_displayed ) - def test_0035_reactivate_column_repository( self ): - '''Activate column_maker - this should not automatically activate convert_chars, so column_maker should be displayed as installed but missing repository dependencies''' + def test_0050_reactivate_column_repository( self ): + '''Reactivate column_maker. This should not automatically reactivate convert_chars, so column_maker should be displayed as installed but missing repository dependencies.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -185,8 +233,8 @@ 'Deactivated' ] self.display_installed_repository_manage_page( installed_column_repository, strings_displayed=strings_displayed ) - def test_0040_reactivate_convert_repository( self ): - '''Activate convert_chars - this should display both convert_chars and column_maker as installed with a green box''' + def test_0055_reactivate_convert_repository( self ): + '''Activate convert_chars. Both convert_chars and column_maker should now show as installed.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -210,8 +258,8 @@ 'Installed repository dependencies' ] self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) - def test_0045_uninstall_column_repository( self ): - '''Uninstall column_maker - this should display convert_chars installed but missing repository dependencies''' + def test_0060_uninstall_column_repository( self ): + '''Uninstall column_maker. Verify that convert_chars is installed and missing repository dependencies, and column_maker was in the right tool panel section.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -227,7 +275,8 @@ 'Uninstalled' ] self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) - def test_0050_reinstall_column_repository( self ): + self.check_galaxy_repository_tool_panel_section( installed_column_repository, 'column_maker' ) + def test_0065_reinstall_column_repository( self ): '''Reinstall column_maker without repository dependencies, verify both convert_chars and column_maker are installed.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) @@ -252,8 +301,8 @@ 'Installed repository dependencies' ] self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) - def test_0055_uninstall_convert_repository( self ): - '''Uninstall convert_chars, verify column_maker installed but missing repository dependencies''' + def test_0070_uninstall_convert_repository( self ): + '''Uninstall convert_chars, verify column_maker installed but missing repository dependencies.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -269,8 +318,9 @@ 'Uninstalled' ] self.display_installed_repository_manage_page( installed_column_repository, strings_displayed=strings_displayed ) - def test_0060_uninstall_column_repository( self ): - '''Uninstall column_maker - both convert_chars and column_maker are uninstalled''' + self.check_galaxy_repository_tool_panel_section( installed_convert_repository, 'convert_chars' ) + def test_0075_uninstall_column_repository( self ): + '''Uninstall column_maker, verify that both convert_chars and column_maker are uninstalled.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, @@ -287,13 +337,16 @@ 'Uninstalled' ] self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) - def test_0065_reinstall_convert_repository( self ): - '''Reinstall convert_chars and check the handle repository dependencies check box - this should install both convert_chars and column_maker ( make sure )''' + def test_0080_reinstall_convert_repository( self ): + '''Reinstall convert_chars with repository dependencies, verify that this installs both convert_chars and column_maker.''' installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, common.test_user_1_name ) installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, common.test_user_1_name ) - self.reinstall_repository( installed_convert_repository, install_repository_dependencies=True ) + self.reinstall_repository( installed_convert_repository, + install_repository_dependencies=True, + no_changes=False, + new_tool_panel_section='convert_maker' ) strings_displayed = [ installed_column_repository.name, installed_column_repository.description, installed_column_repository.tool_shed, @@ -312,3 +365,16 @@ 'Installed repository dependencies' ] self.display_installed_repository_manage_page( installed_convert_repository, strings_displayed=strings_displayed ) + def test_0085_uninstall_all_repositories( self ): + '''Uninstall convert_chars and column_maker to verify that they are in the right tool panel sections.''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_column_repository, remove_from_disk=True ) + self.uninstall_repository( installed_convert_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_column_repository ) + test_db_util.ga_refresh( installed_convert_repository ) + self.check_galaxy_repository_tool_panel_section( installed_column_repository, 'convert_maker' ) + self.check_galaxy_repository_tool_panel_section( installed_convert_repository, 'convert_maker' ) + diff -r 0b2c782ca2d7a62f6a4e0781c8b729225806b442 -r 6ef9b038e329b6ee4e71ef1d1b1a22b9b2000ca1 test/tool_shed/functional/test_1085_repository_dependency_handling.py --- /dev/null +++ b/test/tool_shed/functional/test_1085_repository_dependency_handling.py @@ -0,0 +1,176 @@ +from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os, logging +import tool_shed.base.test_db_util as test_db_util + +column_repository_name = 'column_maker_1085' +column_repository_description = "Add column" +column_repository_long_description = "Compute an expression on every row" + +convert_repository_name = 'convert_chars_1085' +convert_repository_description = "Convert delimiters" +convert_repository_long_description = "Convert delimiters to tab" + +category_name = 'Test 1085 Advanced Circular Dependencies' +category_description = 'Test circular dependency features' + +log = logging.getLogger( __name__ ) + +class TestRepositoryDependencies( ShedTwillTestCase ): + '''Testing freebayes with tool data table entries, .loc files, and tool dependencies.''' + def test_0000_create_or_login_admin_user( self ): + """Create necessary user accounts and login as an admin user.""" + self.galaxy_logout() + self.galaxy_login( email=common.admin_email, username=common.admin_username ) + galaxy_admin_user = test_db_util.get_galaxy_user( common.admin_email ) + assert galaxy_admin_user is not None, 'Problem retrieving user with email %s from the database' % common.admin_email + galaxy_admin_user_private_role = test_db_util.get_galaxy_private_role( galaxy_admin_user ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + test_user_1 = test_db_util.get_user( common.test_user_1_email ) + assert test_user_1 is not None, 'Problem retrieving user with email %s from the database' % test_user_1_email + test_user_1_private_role = test_db_util.get_private_role( test_user_1 ) + self.logout() + self.login( email=common.admin_email, username=common.admin_username ) + admin_user = test_db_util.get_user( common.admin_email ) + assert admin_user is not None, 'Problem retrieving user with email %s from the database' % admin_email + admin_user_private_role = test_db_util.get_private_role( admin_user ) + def test_0005_create_and_populate_column_repository( self ): + """Create a category for this test suite and add repositories to it.""" + category = self.create_category( name=category_name, description=category_description ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=column_repository_name, + description=column_repository_description, + long_description=column_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + if self.repository_is_new( repository ): + self.upload_file( repository, + 'column_maker/column_maker.tar', + strings_displayed=[], + commit_message='Uploaded column_maker.tar.' ) + def test_0010_create_and_populate_convert_repository( self ): + self.logout() + self.login( email=common.admin_email, username=common.admin_username ) + category = self.create_category( name=category_name, description=category_description ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=convert_repository_name, + description=convert_repository_description, + long_description=convert_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + if self.repository_is_new( repository ): + self.upload_file( repository, + 'convert_chars/convert_chars.tar', + strings_displayed=[], + commit_message='Uploaded convert_chars.tar.' ) + def test_0015_create_and_upload_dependency_files( self ): + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_1085', additional_paths=[ 'column' ] ) + self.generate_repository_dependency_xml( [ convert_repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Column maker depends on the convert_chars repository.' ) + self.upload_file( column_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on convert' ) + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_1085', additional_paths=[ 'convert' ] ) + self.generate_repository_dependency_xml( [ column_repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Convert chars depends on the column_maker repository.' ) + self.upload_file( convert_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on column' ) + def test_0020_install_repositories( self ): + '''Install column_maker into column_maker tool panel section and install repository dependencies.''' + self.galaxy_logout() + self.galaxy_login( email=common.admin_email, username=common.admin_username ) + self.install_repository( column_repository_name, + common.test_user_1_name, + category_name, + install_tool_dependencies=False, + install_repository_dependencies=True, + new_tool_panel_section='column_maker', + strings_displayed=[ 'install_repository_dependencies' ] ) + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + browse_strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.installed_changeset_revision ] + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_galaxy_browse_repositories_page( strings_displayed=browse_strings_displayed ) + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0025_uninstall_column_repository( self ): + '''uninstall column_maker, verify same section''' + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_column_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_column_repository ) + self.check_galaxy_repository_tool_panel_section( installed_column_repository, 'column_maker' ) + def test_0030_uninstall_convert_repository( self ): + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_convert_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_convert_repository ) + self.check_galaxy_repository_tool_panel_section( installed_convert_repository, 'column_maker' ) + def test_0035_reinstall_column_repository( self ): + '''reinstall column_maker into new section 'new_column_maker' (no_changes = false), no dependencies''' + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.reinstall_repository( installed_column_repository, + install_tool_dependencies=False, + install_repository_dependencies=False, + new_tool_panel_section='new_column_maker', + no_changes=False ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + def test_0040_reinstall_convert_repository( self ): + '''reinstall convert_chars into new section 'new_convert_chars' (no_changes = false), no dependencies''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + self.reinstall_repository( installed_convert_repository, + install_tool_dependencies=False, + install_repository_dependencies=False, + new_tool_panel_section='new_convert_chars', + no_changes=False ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0045_uninstall_and_verify_tool_panel_sections( self ): + '''uninstall both and verify tool panel sections''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_column_repository, remove_from_disk=True ) + self.uninstall_repository( installed_convert_repository, remove_from_disk=True ) + test_db_util.ga_refresh( installed_convert_repository ) + test_db_util.ga_refresh( installed_column_repository ) + self.check_galaxy_repository_tool_panel_section( installed_column_repository, 'new_column_maker' ) + self.check_galaxy_repository_tool_panel_section( installed_convert_repository, 'new_convert_chars' ) diff -r 0b2c782ca2d7a62f6a4e0781c8b729225806b442 -r 6ef9b038e329b6ee4e71ef1d1b1a22b9b2000ca1 test/tool_shed/functional_tests.py --- a/test/tool_shed/functional_tests.py +++ b/test/tool_shed/functional_tests.py @@ -60,6 +60,7 @@ tool_sheds_conf_xml_template = '''<?xml version="1.0"?><tool_sheds><tool_shed name="Embedded tool shed for functional tests" url="http://${shed_url}:${shed_port}/"/> + <tool_shed name="Galaxy main tool shed" url="http://toolshed.g2.bx.psu.edu/"/></tool_sheds> ''' @@ -123,7 +124,7 @@ galaxy_tool_data_table_conf_file = os.environ.get( 'GALAXY_TEST_TOOL_DATA_TABLE_CONF', os.path.join( tool_shed_test_tmp_dir, 'tool_data_table_conf.xml' ) ) galaxy_tool_conf_file = os.environ.get( 'GALAXY_TEST_TOOL_CONF', os.path.join( tool_shed_test_tmp_dir, 'test_tool_conf.xml' ) ) galaxy_shed_tool_conf_file = os.environ.get( 'GALAXY_TEST_SHED_TOOL_CONF', os.path.join( tool_shed_test_tmp_dir, 'test_shed_tool_conf.xml' ) ) - galaxy_tool_sheds_conf_file = os.environ.get( 'GALAXY_TEST_SHED_TOOLS_CONF', os.path.join( tool_shed_test_tmp_dir, 'test_sheds_conf.xml' ) ) + galaxy_tool_sheds_conf_file = os.environ.get( 'GALAXY_TEST_TOOL_SHEDS_CONF', os.path.join( tool_shed_test_tmp_dir, 'test_sheds_conf.xml' ) ) if 'GALAXY_TEST_TOOL_DATA_PATH' in os.environ: tool_data_path = os.environ.get( 'GALAXY_TEST_TOOL_DATA_PATH' ) else: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Fix for correctly positioning tool shed repository tools in the proper tool panel location when the repository is being installed or reinstalled.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/857a60e69c7a/ changeset: 857a60e69c7a user: greg date: 2013-01-09 15:20:58 summary: Fix for correctly positioning tool shed repository tools in the proper tool panel location when the repository is being installed or reinstalled. affected #: 1 file diff -r be29d1334c08ec3e32290feab9a6da7bda9862ec -r 857a60e69c7aee92f67f37311a40a825b40a6481 lib/galaxy/util/shed_util.py --- a/lib/galaxy/util/shed_util.py +++ b/lib/galaxy/util/shed_util.py @@ -184,17 +184,17 @@ instance and when uninstalled repositories are being reinstalled. """ message = '' - # The following list will be maintained within this method to contain all created or updated tool shed repositories, including repository - # dependencies that may not be installed. + # The following list will be maintained within this method to contain all created or updated tool shed repositories, including repository dependencies + # that may not be installed. all_created_or_updated_tool_shed_repositories = [] - # There will be a one-to-one mapping between items in created_or_updated_tool_shed_repositories and tool_panel_section_keys. The following - # list will filter out repository dependencies that are not to be installed. + # There will be a one-to-one mapping between items in 3 lists: created_or_updated_tool_shed_repositories, tool_panel_section_keys and filtered_repo_info_dicts. + # The 3 lists will filter out repository dependencies that are not to be installed. created_or_updated_tool_shed_repositories = [] tool_panel_section_keys = [] - # Repositories will be filtered (e.g., if already installed, etc), so filter the associated repo_info_dicts accordingly. + # Repositories will be filtered (e.g., if already installed, if elected to not be installed, etc), so filter the associated repo_info_dicts accordingly. filtered_repo_info_dicts = [] - # Discover all repository dependencies and retrieve information for installing them. Even if the user elected to not install repository dependencies, - # we have to make sure all repository dependency objects exist so that the appropriate repository dependency relationships can be built. + # Discover all repository dependencies and retrieve information for installing them. Even if the user elected to not install repository dependencies we have + # to make sure all repository dependency objects exist so that the appropriate repository dependency relationships can be built. all_repo_info_dicts = get_required_repo_info_dicts( tool_shed_url, repo_info_dicts ) if not all_repo_info_dicts: # No repository dependencies were discovered so process the received repositories. @@ -215,9 +215,12 @@ tool_panel_section=tool_panel_section, new_tool_panel_section=new_tool_panel_section ) if reinstalling or install_repository_dependencies: + # If the user elected to install repository dependencies, all items in the all_repo_info_dicts list will be processed. However, if + # repository dependencies are not to be installed, only those items contained in the received repo_info_dicts list will be processed. if is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ) or install_repository_dependencies: if installed_tool_shed_repository.status in [ trans.model.ToolShedRepository.installation_status.ERROR, trans.model.ToolShedRepository.installation_status.UNINSTALLED ]: + # The current tool shed repository is not currently installed, so we can update it's record in the database. can_update = True name = installed_tool_shed_repository.name description = installed_tool_shed_repository.description @@ -225,13 +228,19 @@ metadata_dict = installed_tool_shed_repository.metadata dist_to_shed = installed_tool_shed_repository.dist_to_shed else: - # There is a repository already installed which is a dependency of the repository being reinstalled. + # The tool shed repository currently being processed is already installed or is in the process of being installed, so it's record + # in the database cannot be updated. can_update = False else: + # This block will be reached only if reinstalling is True, install_repository_dependencies is False and is_in_repo_info_dicts is False. + # The tool shed repository currently being processed must be a repository dependency that the user elected to not install, so it's + # record in the database cannot be updated. can_update = False else: + # This block will be reached only if reinstalling is False and install_repository_dependencies is False. This implies that the tool shed + # repository currently being processed has already been installed. if len( all_repo_info_dicts ) == 1: - # An attempt is being made to install a tool shed repository into a Galaxy instance when the same repository was previously installed. + # If only a single repository is being installed, return an informative message to the user. message += "Revision <b>%s</b> of tool shed repository <b>%s</b> owned by <b>%s</b> " % ( changeset_revision, name, repository_owner ) if installed_changeset_revision != changeset_revision: message += "was previously installed using changeset revision <b>%s</b>. " % installed_changeset_revision @@ -247,26 +256,24 @@ created_or_updated_tool_shed_repositories.append( installed_tool_shed_repository ) tool_panel_section_keys.append( tool_panel_section_key ) return created_or_updated_tool_shed_repositories, tool_panel_section_keys, all_repo_info_dicts, filtered_repo_info_dicts, message - elif is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ): - can_update = True else: + # We're in the process of installing multiple tool shed repositories into Galaxy. Since the repository currently being processed + # has already been installed, skip it and process the next repository in the list. can_update = False else: - # A tool shed repository is being installed into a Galaxy instance for the first time. We may have the case where a repository - # is being reinstalled where because the repository being newly installed here may be a dependency of the repository being reinstalled. + # A tool shed repository is being installed into a Galaxy instance for the first time, or we're attempting to install it or reinstall it resulted + # in an error. In the latter case, the repository record in the database has no metadata and it's status has been set to 'New'. In either case, + # the repository's database record may be updated. can_update = True installed_changeset_revision = changeset_revision metadata_dict = {} dist_to_shed = False if can_update: + # The database record for the tool shed repository currently being processed can be updated. if reinstalling or install_repository_dependencies: # Get the repository metadata to see where it was previously located in the tool panel. - installed_tool_shed_repository = suc.get_repository_for_dependency_relationship( app=trans.app, - tool_shed=tool_shed_url, - name=name, - owner=repository_owner, - changeset_revision=changeset_revision ) if installed_tool_shed_repository: + # The tool shed repository status is one of 'New', 'Uninstalled', or 'Error'. tool_section, new_tool_panel_section, tool_panel_section_key = \ handle_tool_panel_selection( trans=trans, metadata=installed_tool_shed_repository.metadata, @@ -274,6 +281,8 @@ tool_panel_section=tool_panel_section, new_tool_panel_section=new_tool_panel_section ) else: + # We're installing a new tool shed repository that does not yet have a database record. This repository is a repository dependency + # of a different repository being installed. if new_tool_panel_section: section_id = new_tool_panel_section.lower().replace( ' ', '_' ) tool_panel_section_key = 'section_%s' % str( section_id ) @@ -282,6 +291,7 @@ else: tool_panel_section_key = None else: + # We're installing a new tool shed repository that does not yet have a database record. if new_tool_panel_section: section_id = new_tool_panel_section.lower().replace( ' ', '_' ) tool_panel_section_key = 'section_%s' % str( section_id ) @@ -300,9 +310,11 @@ current_changeset_revision=changeset_revision, owner=repository_owner, dist_to_shed=False ) + # Add the processed tool shed repository to the list of all processed repositories maintained within this method. all_created_or_updated_tool_shed_repositories.append( tool_shed_repository ) - # Only append the tool_shed_repository to the list of created_or_updated_tool_shed_repositories if it is supposed to be installed. + # Only append the tool shed repository to the list of created_or_updated_tool_shed_repositories if it is supposed to be installed. if install_repository_dependencies or is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ): + # Keep the one-to-one mapping between items in 3 lists. created_or_updated_tool_shed_repositories.append( tool_shed_repository ) tool_panel_section_keys.append( tool_panel_section_key ) filtered_repo_info_dicts.append( encoding_util.tool_shed_encode( repo_info_dict ) ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Add missing import.
by Bitbucket 09 Jan '13

09 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/be29d1334c08/ changeset: be29d1334c08 user: jgoecks date: 2013-01-09 15:07:58 summary: Add missing import. affected #: 1 file diff -r 66bbee3c9c02b343c4ab621d39efa261b261f237 -r be29d1334c08ec3e32290feab9a6da7bda9862ec lib/galaxy/webapps/galaxy/controllers/visualization.py --- a/lib/galaxy/webapps/galaxy/controllers/visualization.py +++ b/lib/galaxy/webapps/galaxy/controllers/visualization.py @@ -5,6 +5,7 @@ from galaxy.model.item_attrs import UsesAnnotations, UsesItemRatings from galaxy.web.base.controller import BaseUIController, SharableMixin, UsesVisualizationMixin, VALID_SLUG_RE from galaxy.web.framework.helpers import time_ago, grids, iff +from galaxy import util from galaxy.util.json import from_json_string from galaxy.util.sanitize_html import sanitize_html from galaxy.visualization.genomes import decode_dbkey Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Handle repository dependency objects appropriately if the user elected to not install them.
by Bitbucket 08 Jan '13

08 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/66bbee3c9c02/ changeset: 66bbee3c9c02 user: greg date: 2013-01-08 22:30:14 summary: Handle repository dependency objects appropriately if the user elected to not install them. affected #: 2 files diff -r 2ec609faae49b4ab32190403669849f1af0b68f6 -r 66bbee3c9c02b343c4ab621d39efa261b261f237 lib/galaxy/util/shed_util.py --- a/lib/galaxy/util/shed_util.py +++ b/lib/galaxy/util/shed_util.py @@ -2,7 +2,7 @@ from galaxy.datatypes import checkers from galaxy.web import url_for from galaxy import util -from galaxy.util.json import from_json_string, to_json_string +from galaxy.util import json from galaxy.webapps.community.util import container_util import shed_util_common as suc import galaxy.tools @@ -184,19 +184,20 @@ instance and when uninstalled repositories are being reinstalled. """ message = '' - # There will be a one-to-one mapping between items in created_or_updated_tool_shed_repositories and tool_panel_section_keys. + # The following list will be maintained within this method to contain all created or updated tool shed repositories, including repository + # dependencies that may not be installed. + all_created_or_updated_tool_shed_repositories = [] + # There will be a one-to-one mapping between items in created_or_updated_tool_shed_repositories and tool_panel_section_keys. The following + # list will filter out repository dependencies that are not to be installed. created_or_updated_tool_shed_repositories = [] tool_panel_section_keys = [] # Repositories will be filtered (e.g., if already installed, etc), so filter the associated repo_info_dicts accordingly. filtered_repo_info_dicts = [] - if install_repository_dependencies: - # Discover all repository dependencies and retrieve information for installing them. - all_repo_info_dicts = get_required_repo_info_dicts( tool_shed_url, repo_info_dicts ) - if not all_repo_info_dicts: - # No repository dependencies were discovered so process the received repositories. - all_repo_info_dicts = [ rid for rid in repo_info_dicts ] - else: - # The user chose to not install repository dependencies, so process the received repositories. + # Discover all repository dependencies and retrieve information for installing them. Even if the user elected to not install repository dependencies, + # we have to make sure all repository dependency objects exist so that the appropriate repository dependency relationships can be built. + all_repo_info_dicts = get_required_repo_info_dicts( tool_shed_url, repo_info_dicts ) + if not all_repo_info_dicts: + # No repository dependencies were discovered so process the received repositories. all_repo_info_dicts = [ rid for rid in repo_info_dicts ] for repo_info_dict in all_repo_info_dicts: for name, repo_info_tuple in repo_info_dict.items(): @@ -214,16 +215,19 @@ tool_panel_section=tool_panel_section, new_tool_panel_section=new_tool_panel_section ) if reinstalling or install_repository_dependencies: - if installed_tool_shed_repository.status in [ trans.model.ToolShedRepository.installation_status.ERROR, - trans.model.ToolShedRepository.installation_status.UNINSTALLED ]: - can_update = True - name = installed_tool_shed_repository.name - description = installed_tool_shed_repository.description - installed_changeset_revision = installed_tool_shed_repository.installed_changeset_revision - metadata_dict = installed_tool_shed_repository.metadata - dist_to_shed = installed_tool_shed_repository.dist_to_shed + if is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ) or install_repository_dependencies: + if installed_tool_shed_repository.status in [ trans.model.ToolShedRepository.installation_status.ERROR, + trans.model.ToolShedRepository.installation_status.UNINSTALLED ]: + can_update = True + name = installed_tool_shed_repository.name + description = installed_tool_shed_repository.description + installed_changeset_revision = installed_tool_shed_repository.installed_changeset_revision + metadata_dict = installed_tool_shed_repository.metadata + dist_to_shed = installed_tool_shed_repository.dist_to_shed + else: + # There is a repository already installed which is a dependency of the repository being reinstalled. + can_update = False else: - # There is a repository already installed which is a dependency of the repository being reinstalled. can_update = False else: if len( all_repo_info_dicts ) == 1: @@ -243,14 +247,16 @@ created_or_updated_tool_shed_repositories.append( installed_tool_shed_repository ) tool_panel_section_keys.append( tool_panel_section_key ) return created_or_updated_tool_shed_repositories, tool_panel_section_keys, all_repo_info_dicts, filtered_repo_info_dicts, message + elif is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ): + can_update = True else: - can_update = True + can_update = False else: # A tool shed repository is being installed into a Galaxy instance for the first time. We may have the case where a repository # is being reinstalled where because the repository being newly installed here may be a dependency of the repository being reinstalled. can_update = True installed_changeset_revision = changeset_revision - metadata_dict={} + metadata_dict = {} dist_to_shed = False if can_update: if reinstalling or install_repository_dependencies: @@ -294,9 +300,14 @@ current_changeset_revision=changeset_revision, owner=repository_owner, dist_to_shed=False ) - created_or_updated_tool_shed_repositories.append( tool_shed_repository ) - tool_panel_section_keys.append( tool_panel_section_key ) - filtered_repo_info_dicts.append( encoding_util.tool_shed_encode( repo_info_dict ) ) + all_created_or_updated_tool_shed_repositories.append( tool_shed_repository ) + # Only append the tool_shed_repository to the list of created_or_updated_tool_shed_repositories if it is supposed to be installed. + if install_repository_dependencies or is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ): + created_or_updated_tool_shed_repositories.append( tool_shed_repository ) + tool_panel_section_keys.append( tool_panel_section_key ) + filtered_repo_info_dicts.append( encoding_util.tool_shed_encode( repo_info_dict ) ) + # Build repository dependency relationships even if the user chose to not install repository dependencies. + suc.build_repository_dependency_relationships( trans, all_repo_info_dicts, all_created_or_updated_tool_shed_repositories ) return created_or_updated_tool_shed_repositories, tool_panel_section_keys, all_repo_info_dicts, filtered_repo_info_dicts, message def create_repository_dict_for_proprietary_datatypes( tool_shed, name, owner, installed_changeset_revision, tool_dicts, converter_path=None, display_path=None ): return dict( tool_shed=tool_shed, @@ -679,6 +690,73 @@ tmp_url = suc.clean_repository_clone_url( repository_clone_url ) tool_shed = tmp_url.split( 'repos' )[ 0 ].rstrip( '/' ) return get_repository_owner( tmp_url ) +def get_repository_readme_and_dependencies_for_display( trans, tool_shed_url, repo_info_dict, includes_tool_dependencies ): + # If we're installing a single repository, see if it contains a readme or dependencies that we can display. + name = repo_info_dict.keys()[ 0 ] + repo_info_tuple = repo_info_dict[ name ] + description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, repository_dependencies, installed_td = \ + suc.get_repo_info_tuple_contents( repo_info_tuple ) + # Handle README files. + url = suc.url_join( tool_shed_url, + 'repository/get_readme_files?name=%s&owner=%s&changeset_revision=%s' % \ + ( name, repository_owner, changeset_revision ) ) + response = urllib2.urlopen( url ) + raw_text = response.read() + response.close() + readme_files_dict = json.from_json_string( raw_text ) + if repository_dependencies: + missing_td = {} + # Handle the scenario where a repository was installed, then uninstalled and an error occurred during the reinstallation process. + # In this case, a record for the repository will exist in the database with the status of 'New'. + repository = suc.get_repository_for_dependency_relationship( trans.app, tool_shed_url, name, repository_owner, changeset_revision ) + if repository and repository.metadata: + installed_rd, missing_rd = \ + get_installed_and_missing_repository_dependencies( trans, repository ) + else: + installed_rd, missing_rd = \ + get_installed_and_missing_repository_dependencies_for_new_install( trans, repository_dependencies ) + # Discover all repository dependencies and retrieve information for installing them. + required_repo_info_dicts = get_required_repo_info_dicts( tool_shed_url, util.listify( repo_info_dict ) ) + # Display tool dependencies defined for each of the repository dependencies. + if required_repo_info_dicts: + all_tool_dependencies = {} + for rid in required_repo_info_dicts: + for name, repo_info_tuple in rid.items(): + description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, repository_dependencies, rid_installed_td = \ + suc.get_repo_info_tuple_contents( repo_info_tuple ) + if rid_installed_td: + for td_key, td_dict in rid_installed_td.items(): + if td_key not in all_tool_dependencies: + all_tool_dependencies[ td_key ] = td_dict + if all_tool_dependencies: + if installed_td is None: + installed_td = {} + else: + # Move all tool dependencies to the missing_tool_dependencies container. + for td_key, td_dict in installed_td.items(): + if td_key not in missing_td: + missing_td[ td_key ] = td_dict + installed_td = {} + # Discover and categorize all tool dependencies defined for this repository's repository dependencies. + required_tool_dependencies, required_missing_tool_dependencies = \ + get_installed_and_missing_tool_dependencies_for_new_install( trans, all_tool_dependencies ) + if required_tool_dependencies: + if not includes_tool_dependencies: + includes_tool_dependencies = True + for td_key, td_dict in required_tool_dependencies.items(): + if td_key not in installed_td: + installed_td[ td_key ] = td_dict + if required_missing_tool_dependencies: + if not includes_tool_dependencies: + includes_tool_dependencies = True + for td_key, td_dict in required_missing_tool_dependencies.items(): + if td_key not in missing_td: + missing_td[ td_key ] = td_dict + else: + installed_rd = None + missing_rd = None + missing_td = None + return name, repository_owner, changeset_revision, readme_files_dict, includes_tool_dependencies, installed_rd, missing_rd, installed_td, missing_td def get_required_repo_info_dicts( tool_shed_url, repo_info_dicts ): """ Inspect the list of repo_info_dicts for repository dependencies and append a repo_info_dict for each of them to the list. All @@ -716,7 +794,7 @@ text = response.read() response.close() if text: - required_repo_info_dict = from_json_string( text ) + required_repo_info_dict = json.from_json_string( text ) required_repo_info_dicts = [] encoded_dict_strings = required_repo_info_dict[ 'repo_info_dicts' ] for encoded_dict_str in encoded_dict_strings: @@ -1015,6 +1093,18 @@ return False # Default to copying the file if none of the above are true. return True +def is_in_repo_info_dicts( repo_info_dict, repo_info_dicts ): + """Return True if the received repo_info_dict is contained in the list of received repo_info_dicts.""" + for name, repo_info_tuple in repo_info_dict.items(): + for rid in repo_info_dicts: + for rid_name, rid_repo_info_tuple in rid.items(): + if rid_name == name: + if len( rid_repo_info_tuple ) == len( repo_info_tuple ): + for item in rid_repo_info_tuple: + if item not in repo_info_tuple: + return False + return True + return False def load_installed_datatype_converters( app, installed_repository_dict, deactivate=False ): # Load or deactivate proprietary datatype converters app.datatypes_registry.load_datatype_converters( app.toolbox, installed_repository_dict=installed_repository_dict, deactivate=deactivate ) @@ -1073,7 +1163,7 @@ response = urllib2.urlopen( url ) raw_text = response.read() response.close() - readme_files_dict = from_json_string( raw_text ) + readme_files_dict = json.from_json_string( raw_text ) else: readme_files_dict = suc.build_readme_files_dict( repository.metadata, tool_path ) else: diff -r 2ec609faae49b4ab32190403669849f1af0b68f6 -r 66bbee3c9c02b343c4ab621d39efa261b261f237 lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py --- a/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py +++ b/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py @@ -1220,9 +1220,6 @@ message=message, status='error' ) ) if created_or_updated_tool_shed_repositories: - if install_repository_dependencies: - # Build repository dependency relationships. - suc.build_repository_dependency_relationships( trans, repo_info_dicts, created_or_updated_tool_shed_repositories ) # Handle contained tools. if includes_tools and ( new_tool_panel_section or tool_panel_section ): if new_tool_panel_section: @@ -1290,73 +1287,10 @@ if len( repo_info_dicts ) == 1: # If we're installing a single repository, see if it contains a readme or dependencies that we can display. repo_info_dict = repo_info_dicts[ 0 ] - name = repo_info_dict.keys()[ 0 ] - repo_info_tuple = repo_info_dict[ name ] - description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, repository_dependencies, tool_dependencies = \ - suc.get_repo_info_tuple_contents( repo_info_tuple ) - # Handle README files. - url = suc.url_join( tool_shed_url, - 'repository/get_readme_files?name=%s&owner=%s&changeset_revision=%s' % \ - ( name, repository_owner, changeset_revision ) ) - response = urllib2.urlopen( url ) - raw_text = response.read() - response.close() - readme_files_dict = json.from_json_string( raw_text ) - if repository_dependencies: - missing_tool_dependencies = {} - # Handle the scenario where a repository was installed, then uninstalled and an error occurred during the reinstallation process. - # In this case, a record for the repository will exist in the database with the status of 'New'. - repository = suc.get_repository_for_dependency_relationship( trans.app, tool_shed_url, name, repository_owner, changeset_revision ) - if repository and repository.metadata: - installed_repository_dependencies, missing_repository_dependencies = \ - shed_util.get_installed_and_missing_repository_dependencies( trans, repository ) - else: - installed_repository_dependencies, missing_repository_dependencies = \ - shed_util.get_installed_and_missing_repository_dependencies_for_new_install( trans, repository_dependencies ) - # Discover all repository dependencies and retrieve information for installing them. - required_repo_info_dicts = shed_util.get_required_repo_info_dicts( tool_shed_url, util.listify( repo_info_dict ) ) - # Display tool dependencies defined for each of the repository dependencies. - if required_repo_info_dicts: - all_tool_dependencies = {} - for rid in required_repo_info_dicts: - for name, repo_info_tuple in rid.items(): - description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, repository_dependencies, tool_dependencies = \ - suc.get_repo_info_tuple_contents( repo_info_tuple ) - if tool_dependencies: - for td_key, td_dict in tool_dependencies.items(): - if td_key not in all_tool_dependencies: - all_tool_dependencies[ td_key ] = td_dict - if all_tool_dependencies: - if tool_dependencies is None: - tool_dependencies = {} - else: - # Move all tool dependencies to the missing_tool_dependencies container. - for td_key, td_dict in tool_dependencies.items(): - if td_key not in missing_tool_dependencies: - missing_tool_dependencies[ td_key ] = td_dict - tool_dependencies = {} - # Discover and categorize all tool dependencies defined this this repository's repository dependencies. - required_tool_dependencies, required_missing_tool_dependencies = \ - shed_util.get_installed_and_missing_tool_dependencies_for_new_install( trans, all_tool_dependencies ) - if required_tool_dependencies: - if not includes_tool_dependencies: - includes_tool_dependencies = True - for td_key, td_dict in required_tool_dependencies.items(): - if td_key not in tool_dependencies: - tool_dependencies[ td_key ] = td_dict - if required_missing_tool_dependencies: - if not includes_tool_dependencies: - includes_tool_dependencies = True - for td_key, td_dict in required_missing_tool_dependencies.items(): - if td_key not in missing_tool_dependencies: - missing_tool_dependencies[ td_key ] = td_dict - else: - installed_repository_dependencies = None - missing_repository_dependencies = None - missing_tool_dependencies = None - required_repo_info_dicts = None - # Since we are installing a new repository, no tool dependencies will be considered "missing". Most of the repository contents - # are set to None since we don't yet know what they are. + name, repository_owner, changeset_revision, readme_files_dict, includes_tool_dependencies, \ + installed_repository_dependencies, missing_repository_dependencies, tool_dependencies, missing_tool_dependencies = \ + shed_util.get_repository_readme_and_dependencies_for_display( trans, tool_shed_url, repo_info_dict, includes_tool_dependencies ) + # Since we are installing a new repository, most of the repository contents are set to None since we don't yet know what they are. containers_dict = suc.build_repository_containers_for_galaxy( trans=trans, toolshed_base_url=tool_shed_url, repository_name=name, @@ -1374,7 +1308,7 @@ workflows=None, new_install=True, reinstalling=False ) - # We're handling 1 of 2 scenarios here: (1) we're installing a tool shed repository for the first time, so we're retrieved the list of installed + # We're handling 1 of 2 scenarios here: (1) we're installing a tool shed repository for the first time, so we've retrieved the list of installed # and missing repository dependencies from the database (2) we're handling the scenario where an error occurred during the installation process, # so we have a tool_shed_repository record in the database with associated repository dependency records. Since we have the repository # dependencies in either case, we'll merge the list of missing repository dependencies into the list of installed repository dependencies since @@ -1489,24 +1423,16 @@ repository_dependencies=repository_dependencies ) repo_info_dicts.append( repo_info_dict ) # Make sure all tool_shed_repository records exist. - if install_repository_dependencies: - created_or_updated_tool_shed_repositories, tool_panel_section_keys, repo_info_dicts, filtered_repo_info_dicts, message = \ - shed_util.create_repository_dependency_objects( trans, - tool_path, - tool_shed_url, - repo_info_dicts, - reinstalling=True, - install_repository_dependencies=install_repository_dependencies, - no_changes_checked=no_changes_checked, - tool_panel_section=tool_panel_section, - new_tool_panel_section=new_tool_panel_section ) - if len( created_or_updated_tool_shed_repositories ) > 1: - # Build repository dependency relationships. - suc.build_repository_dependency_relationships( trans, repo_info_dicts, created_or_updated_tool_shed_repositories ) - else: - filtered_repo_info_dicts = [ repo_info_dict for repo_info_dict in repo_info_dicts ] - created_or_updated_tool_shed_repositories = [ tool_shed_repository ] - tool_panel_section_keys.append( tool_panel_section_key ) + created_or_updated_tool_shed_repositories, tool_panel_section_keys, repo_info_dicts, filtered_repo_info_dicts, message = \ + shed_util.create_repository_dependency_objects( trans, + tool_path, + tool_shed_url, + repo_info_dicts, + reinstalling=True, + install_repository_dependencies=install_repository_dependencies, + no_changes_checked=no_changes_checked, + tool_panel_section=tool_panel_section, + new_tool_panel_section=new_tool_panel_section ) # Default the selected tool panel location for loading tools included in each newly installed required tool shed repository to the location # selected for the repository selected for reinstallation. for index, tps_key in enumerate( tool_panel_section_keys ): Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: When installing a new repository from the too shed repository to Galaxy or reinstalling an uninstalled repository, handle tool dependencies defined for repository dependencies defined for the repository being installed.
by Bitbucket 08 Jan '13

08 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/2ec609faae49/ changeset: 2ec609faae49 user: greg date: 2013-01-08 19:50:16 summary: When installing a new repository from the too shed repository to Galaxy or reinstalling an uninstalled repository, handle tool dependencies defined for repository dependencies defined for the repository being installed. affected #: 6 files diff -r 530fb4f8204f2106e11f419c381fd53d2319ce24 -r 2ec609faae49b4ab32190403669849f1af0b68f6 lib/galaxy/util/shed_util.py --- a/lib/galaxy/util/shed_util.py +++ b/lib/galaxy/util/shed_util.py @@ -653,6 +653,22 @@ tool_dependencies = None missing_tool_dependencies = None return tool_dependencies, missing_tool_dependencies +def get_installed_and_missing_tool_dependencies_for_new_install( trans, all_tool_dependencies ): + """Return the lists of installed tool dependencies and missing tool dependencies for a set of repositories being installed into Galaxy.""" + # FIXME: this method currently populates and returns only missing tool dependencies since tool dependencies defined for complex repository dependency + # relationships is not currently supported. This method should be enhanced to search for installed tool dependencies defined as complex repository + # dependency relationships when that feature is implemented. + if all_tool_dependencies: + tool_dependencies = {} + missing_tool_dependencies = {} + for td_key, val in all_tool_dependencies.items(): + # Since we have a new install, missing tool dependencies have never been installed. + val[ 'status' ] = trans.model.ToolDependency.installation_status.NEVER_INSTALLED + missing_tool_dependencies[ td_key ] = val + else: + tool_dependencies = None + missing_tool_dependencies = None + return tool_dependencies, missing_tool_dependencies def get_repository_owner( cleaned_repository_url ): items = cleaned_repository_url.split( 'repos' ) repo_path = items[ 1 ] @@ -667,7 +683,7 @@ """ Inspect the list of repo_info_dicts for repository dependencies and append a repo_info_dict for each of them to the list. All repository_dependencies entries in each of the received repo_info_dicts includes all required repositories, so only one pass through - this methid is required to retrieve all repository dependencies. + this method is required to retrieve all repository dependencies. """ all_repo_info_dicts = [] if repo_info_dicts: @@ -1035,7 +1051,7 @@ if k not in [ 'id', 'version', 'name' ]: return True return False -def populate_containers_dict_from_repository_metadata( trans, tool_shed_url, tool_path, repository, reinstalling=False ): +def populate_containers_dict_from_repository_metadata( trans, tool_shed_url, tool_path, repository, reinstalling=False, required_repo_info_dicts=None ): """ Retrieve necessary information from the received repository's metadata to populate the containers_dict for display. This method is called only from Galaxy (not the tool shed) when displaying repository dependencies for installed repositories and when displaying them for uninstalled @@ -1064,15 +1080,51 @@ readme_files_dict = None # Handle repository dependencies. installed_repository_dependencies, missing_repository_dependencies = get_installed_and_missing_repository_dependencies( trans, repository ) - # Handle tool dependencies. - all_tool_dependencies = metadata.get( 'tool_dependencies', None ) - installed_tool_dependencies, missing_tool_dependencies = get_installed_and_missing_tool_dependencies( trans, repository, all_tool_dependencies ) + # Handle the current repository's tool dependencies. + repository_tool_dependencies = metadata.get( 'tool_dependencies', None ) + repository_installed_tool_dependencies, repository_missing_tool_dependencies = \ + get_installed_and_missing_tool_dependencies( trans, repository, repository_tool_dependencies ) if reinstalling: - # All tool dependencies will be considered missing since we are reinstalling the repository. - if installed_tool_dependencies: - for td in installed_tool_dependencies: - missing_tool_dependencies.append( td ) - installed_tool_dependencies = None + installed_tool_dependencies = None + missing_tool_dependencies = None + if repository_installed_tool_dependencies is None: + repository_installed_tool_dependencies = {} + if repository_missing_tool_dependencies is None: + repository_missing_tool_dependencies = {} + if required_repo_info_dicts: + # Handle the tool dependencies defined for each of the repository's repository dependencies. + for rid in required_repo_info_dicts: + for name, repo_info_tuple in rid.items(): + description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, repository_dependencies, tool_dependencies = \ + suc.get_repo_info_tuple_contents( repo_info_tuple ) + if tool_dependencies: + tool_shed = suc.get_tool_shed_from_clone_url( repository_clone_url ) + required_repository = suc.get_repository_for_dependency_relationship( trans.app, tool_shed, name, repository_owner, changeset_revision ) + if not required_repository: + # The required_repository may have been installed with a different changeset revision. + required_repository, installed_changeset_revision = repository_was_previously_installed( trans, + tool_shed_url, + name, + repo_info_tuple, + clone_dir ) + if required_repository: + required_repository_installed_tool_dependencies, required_repository_missing_tool_dependencies = \ + get_installed_and_missing_tool_dependencies( trans, required_repository, tool_dependencies ) + if required_repository_installed_tool_dependencies: + for td_key, td_dict in required_repository_installed_tool_dependencies.items(): + if td_key not in repository_installed_tool_dependencies: + repository_installed_tool_dependencies[ td_key ] = td_dict + if required_repository_missing_tool_dependencies: + for td_key, td_dict in required_repository_missing_tool_dependencies.items(): + if td_key not in repository_missing_tool_dependencies: + repository_missing_tool_dependencies[ td_key ] = td_dict + if repository_installed_tool_dependencies: + installed_tool_dependencies = repository_installed_tool_dependencies + if repository_missing_tool_dependencies: + missing_tool_dependencies = repository_missing_tool_dependencies + else: + installed_tool_dependencies = repository_installed_tool_dependencies + missing_tool_dependencies = repository_missing_tool_dependencies # Handle valid tools. valid_tools = metadata.get( 'tools', None ) # Handle workflows. @@ -1092,7 +1144,8 @@ tool_dependencies=installed_tool_dependencies, valid_tools=valid_tools, workflows=workflows, - new_install=False ) + new_install=False, + reinstalling=reinstalling ) else: containers_dict = dict( datatypes=None, invalid_tools=None, diff -r 530fb4f8204f2106e11f419c381fd53d2319ce24 -r 2ec609faae49b4ab32190403669849f1af0b68f6 lib/galaxy/util/shed_util_common.py --- a/lib/galaxy/util/shed_util_common.py +++ b/lib/galaxy/util/shed_util_common.py @@ -78,7 +78,7 @@ return readme_files_dict def build_repository_containers_for_galaxy( trans, toolshed_base_url, repository_name, repository_owner, changeset_revision, repository, datatypes, invalid_tools, missing_repository_dependencies, missing_tool_dependencies, readme_files_dict, - repository_dependencies, tool_dependencies, valid_tools, workflows, new_install=False ): + repository_dependencies, tool_dependencies, valid_tools, workflows, new_install=False, reinstalling=False ): """Return a dictionary of containers for the received repository's dependencies and readme files for display during installation to Galaxy.""" containers_dict = dict( datatypes=None, invalid_tools=None, @@ -160,7 +160,8 @@ tool_dependencies, label=label, missing=False, - new_install=new_install ) + new_install=new_install, + reinstalling=reinstalling ) containers_dict[ 'tool_dependencies' ] = tool_dependencies_root_folder # Missing tool dependencies container. if missing_tool_dependencies: @@ -170,7 +171,8 @@ missing_tool_dependencies, label='Missing tool dependencies', missing=True, - new_install=new_install ) + new_install=new_install, + reinstalling=reinstalling ) containers_dict[ 'missing_tool_dependencies' ] = missing_tool_dependencies_root_folder # Valid tools container. if valid_tools: @@ -790,7 +792,7 @@ Return a dictionary that includes all of the information needed to install a repository into a local Galaxy instance. The dictionary will also contain the recursive list of repository dependencies defined for the repository, as well as the defined tool dependencies. - This method is called from Galaxy from two places: + This method is called from Galaxy in two places: 1. During the tool shed repository installation process (via the tool shed's get_repository_information() method)- in this case both the received repository and repository_metadata will be objects. 2. When a tool shed repository that was uninstalled from a Galaxy instance is being re-installed - in this case, both repository and @@ -2414,6 +2416,47 @@ removed = True error_message = '' return removed, error_message +def repository_dependencies_have_tool_dependencies( trans, repository_dependencies ): + """ + repository_dependencies': + {'http://localhost:9009__ESEP__emboss_6__ESEP__test__ESEP__92bedb60b0c9': + [['http://localhost:9009', 'emboss_datatypes', 'test', '27df73fe48a6']], + 'root_key': 'http://localhost:9009__ESEP__emboss__ESEP__test__ESEP__06d729cb3f34', + 'description': 'required to enable emboss 6 tools', + 'http://localhost:9009__ESEP__emboss__ESEP__test__ESEP__06d729cb3f34': + [['http://localhost:9009', 'emboss_6', 'test', '92bedb60b0c9']]}} + """ + rd_tups_processed = [] + for key, rd_tups in repository_dependencies.items(): + if key in [ 'root_key', 'description' ]: + continue + rd_tup = container_util.get_components_from_key( key ) + if rd_tup not in rd_tups_processed: + toolshed, name, owner, changeset_revision = rd_tup + repository = get_repository_by_name_and_owner( trans, name, owner ) + repository_metadata = get_repository_metadata_by_repository_id_changset_revision( trans, + trans.security.encode_id( repository.id ), + changeset_revision ) + if repository_metadata: + metadata = repository_metadata.metadata + if metadata: + if 'tool_dependencies' in metadata: + return True + rd_tups_processed.append( rd_tup ) + for rd_tup in rd_tups: + if rd_tup not in rd_tups_processed: + toolshed, name, owner, changeset_revision = rd_tup + repository = get_repository_by_name_and_owner( trans, name, owner ) + repository_metadata = get_repository_metadata_by_repository_id_changset_revision( trans, + trans.security.encode_id( repository.id ), + changeset_revision ) + if repository_metadata: + metadata = repository_metadata.metadata + if metadata: + if 'tool_dependencies' in metadata: + return True + rd_tups_processed.append( rd_tup ) + return False def reset_all_metadata_on_installed_repository( trans, id ): """Reset all metadata on a single tool shed repository installed into a Galaxy instance.""" repository = get_installed_tool_shed_repository( trans, id ) diff -r 530fb4f8204f2106e11f419c381fd53d2319ce24 -r 2ec609faae49b4ab32190403669849f1af0b68f6 lib/galaxy/webapps/community/controllers/repository.py --- a/lib/galaxy/webapps/community/controllers/repository.py +++ b/lib/galaxy/webapps/community/controllers/repository.py @@ -786,9 +786,9 @@ operation='preview_tools_in_changeset', repository_id=repository_id ) self.valid_repository_grid.operations = [ grids.GridOperation( "Preview and install", - url_args=url_args, - allow_multiple=False, - async_compatible=False ) ] + url_args=url_args, + allow_multiple=False, + async_compatible=False ) ] return self.valid_repository_grid( trans, **kwd ) def __build_allow_push_select_field( self, trans, current_push_list, selected_value='none' ): options = [] @@ -1410,7 +1410,10 @@ repo_info_dicts=repo_info_dicts ) @web.json def get_required_repo_info_dict( self, trans, encoded_str ): - """Retrive a list of dictionaries that each contain all of the information needed to install the list of repositories defined by encoded_str.""" + """ + Retrieve and return a dictionary that includes a list of dictionaries that each contain all of the information needed to install the list of + repositories defined by the received encoded_str. + """ encoded_required_repository_str = encoding_util.tool_shed_decode( encoded_str ) encoded_required_repository_tups = encoded_required_repository_str.split( encoding_util.encoding_sep2 ) decoded_required_repository_tups = [] @@ -1430,7 +1433,7 @@ return repo_info_dict @web.expose def get_tool_dependencies( self, trans, **kwd ): - """Handle a request from the InstallManager of a local Galaxy instance.""" + """Handle a request from a Galaxy instance.""" params = util.Params( kwd ) message = util.restore_text( params.get( 'message', '' ) ) status = params.get( 'status', 'done' ) diff -r 530fb4f8204f2106e11f419c381fd53d2319ce24 -r 2ec609faae49b4ab32190403669849f1af0b68f6 lib/galaxy/webapps/community/util/container_util.py --- a/lib/galaxy/webapps/community/util/container_util.py +++ b/lib/galaxy/webapps/community/util/container_util.py @@ -291,7 +291,7 @@ else: tools_root_folder = None return folder_id, tools_root_folder -def build_tool_dependencies_folder( trans, folder_id, tool_dependencies, label='Tool dependencies', missing=False, new_install=False ): +def build_tool_dependencies_folder( trans, folder_id, tool_dependencies, label='Tool dependencies', missing=False, new_install=False, reinstalling=False ): """Return a folder hierarchy containing tool dependencies.""" # The status will be displayed only if the received value for missing is True. When this is the case, we're in Galaxy (not the tool shed) # and the tool dependencies are not installed or are in an error state, so they are considered missing. The tool dependency status will be @@ -304,7 +304,9 @@ folder_id += 1 folder = Folder( id=folder_id, key='tool_dependencies', label=label, parent=tool_dependencies_root_folder ) if trans.webapp.name == 'galaxy': - if missing: + if reinstalling: + folder.description = "this repository's tools require handling of these dependencies" + elif missing and not reinstalling: folder.description = 'click the name to install the missing dependency' else: if new_install: diff -r 530fb4f8204f2106e11f419c381fd53d2319ce24 -r 2ec609faae49b4ab32190403669849f1af0b68f6 lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py --- a/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py +++ b/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py @@ -1024,6 +1024,8 @@ decoded_kwd = encoding_util.tool_shed_decode( encoded_kwd ) tsr_ids = decoded_kwd[ 'tool_shed_repository_ids' ] tool_panel_section_keys = decoded_kwd[ 'tool_panel_section_keys' ] + repo_info_dicts = decoded_kwd[ 'repo_info_dicts' ] + filtered_repo_info_dicts = [] filtered_tool_panel_section_keys = [] repositories_for_installation = [] for index, tsr_id in enumerate( tsr_ids ): @@ -1031,8 +1033,10 @@ if repository.status in [ trans.model.ToolShedRepository.installation_status.NEW, trans.model.ToolShedRepository.installation_status.UNINSTALLED ]: repositories_for_installation.append( repository ) + filtered_repo_info_dicts.append( repo_info_dicts[ index ] ) filtered_tool_panel_section_keys.append( tool_panel_section_keys[ index ] ) if repositories_for_installation: + decoded_kwd[ 'repo_info_dicts' ] = filtered_repo_info_dicts decoded_kwd[ 'tool_panel_section_keys' ] = filtered_tool_panel_section_keys self.install_tool_shed_repositories( trans, repositories_for_installation, reinstalling=reinstalling, **decoded_kwd ) else: @@ -1152,7 +1156,6 @@ return trans.show_error_message( message ) message = kwd.get( 'message', '' ) status = kwd.get( 'status', 'done' ) - includes_tools = util.string_as_bool( kwd.get( 'includes_tools', False ) ) tool_shed_url = kwd[ 'tool_shed_url' ] # Handle repository dependencies. includes_repository_dependencies = util.string_as_bool( kwd.get( 'includes_repository_dependencies', False ) ) @@ -1259,7 +1262,6 @@ shed_tool_conf=shed_tool_conf, status=status, tool_path=tool_path, - tool_panel_section_keys=tool_panel_section_keys, tool_shed_repository_ids=tsrids_list, tool_shed_url=tool_shed_url ) @@ -1300,9 +1302,10 @@ raw_text = response.read() response.close() readme_files_dict = json.from_json_string( raw_text ) - # Handle the scenario where a repository was installed, then uninstalled and an error occurred during the reinstallation process. - # In this case, a record for the repository will exist in the database with the status of 'New'. if repository_dependencies: + missing_tool_dependencies = {} + # Handle the scenario where a repository was installed, then uninstalled and an error occurred during the reinstallation process. + # In this case, a record for the repository will exist in the database with the status of 'New'. repository = suc.get_repository_for_dependency_relationship( trans.app, tool_shed_url, name, repository_owner, changeset_revision ) if repository and repository.metadata: installed_repository_dependencies, missing_repository_dependencies = \ @@ -1310,9 +1313,48 @@ else: installed_repository_dependencies, missing_repository_dependencies = \ shed_util.get_installed_and_missing_repository_dependencies_for_new_install( trans, repository_dependencies ) + # Discover all repository dependencies and retrieve information for installing them. + required_repo_info_dicts = shed_util.get_required_repo_info_dicts( tool_shed_url, util.listify( repo_info_dict ) ) + # Display tool dependencies defined for each of the repository dependencies. + if required_repo_info_dicts: + all_tool_dependencies = {} + for rid in required_repo_info_dicts: + for name, repo_info_tuple in rid.items(): + description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, repository_dependencies, tool_dependencies = \ + suc.get_repo_info_tuple_contents( repo_info_tuple ) + if tool_dependencies: + for td_key, td_dict in tool_dependencies.items(): + if td_key not in all_tool_dependencies: + all_tool_dependencies[ td_key ] = td_dict + if all_tool_dependencies: + if tool_dependencies is None: + tool_dependencies = {} + else: + # Move all tool dependencies to the missing_tool_dependencies container. + for td_key, td_dict in tool_dependencies.items(): + if td_key not in missing_tool_dependencies: + missing_tool_dependencies[ td_key ] = td_dict + tool_dependencies = {} + # Discover and categorize all tool dependencies defined this this repository's repository dependencies. + required_tool_dependencies, required_missing_tool_dependencies = \ + shed_util.get_installed_and_missing_tool_dependencies_for_new_install( trans, all_tool_dependencies ) + if required_tool_dependencies: + if not includes_tool_dependencies: + includes_tool_dependencies = True + for td_key, td_dict in required_tool_dependencies.items(): + if td_key not in tool_dependencies: + tool_dependencies[ td_key ] = td_dict + if required_missing_tool_dependencies: + if not includes_tool_dependencies: + includes_tool_dependencies = True + for td_key, td_dict in required_missing_tool_dependencies.items(): + if td_key not in missing_tool_dependencies: + missing_tool_dependencies[ td_key ] = td_dict else: installed_repository_dependencies = None missing_repository_dependencies = None + missing_tool_dependencies = None + required_repo_info_dicts = None # Since we are installing a new repository, no tool dependencies will be considered "missing". Most of the repository contents # are set to None since we don't yet know what they are. containers_dict = suc.build_repository_containers_for_galaxy( trans=trans, @@ -1324,13 +1366,14 @@ datatypes=None, invalid_tools=None, missing_repository_dependencies=missing_repository_dependencies, - missing_tool_dependencies=None, + missing_tool_dependencies=missing_tool_dependencies, readme_files_dict=readme_files_dict, repository_dependencies=installed_repository_dependencies, tool_dependencies=tool_dependencies, valid_tools=None, workflows=None, - new_install=True ) + new_install=True, + reinstalling=False ) # We're handling 1 of 2 scenarios here: (1) we're installing a tool shed repository for the first time, so we're retrieved the list of installed # and missing repository dependencies from the database (2) we're handling the scenario where an error occurred during the installation process, # so we have a tool_shed_repository record in the database with associated repository dependency records. Since we have the repository @@ -1338,6 +1381,7 @@ # each displayed repository dependency will display a status, whether installed or missing. containers_dict = suc.merge_missing_repository_dependencies_to_installed_container( containers_dict ) else: + # FIXME: support the intallation of repository dependencies and tool dependencies for a list of tool shed repositories being installed. containers_dict = dict( datatypes=None, invalid_tools=None, readme_files_dict=None, @@ -1395,6 +1439,8 @@ tool_panel_section_key = None tool_panel_section_keys = [] metadata = tool_shed_repository.metadata + # Keep track of tool dependencies define dfor the current repository or those defined for any of it's repository dependencies. + includes_tool_dependencies = tool_shed_repository.includes_tool_dependencies if tool_shed_repository.includes_tools: # Handle the selected tool panel location for loading tools included in the tool shed repository. tool_section, new_tool_panel_section, tool_panel_section_key = \ @@ -1422,8 +1468,7 @@ if isinstance( repo_info_dict, basestring ): repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) else: - # Entering this else block used to happen only if the tool_shed_repository does not include any valid tools. After repository dependencies - # were introduced, it may never happen, but we'll keep the block just in case. + # Entering this else block occurs only if the tool_shed_repository does not include any valid tools. if install_repository_dependencies: repository_dependencies = self.get_repository_dependencies( trans=trans, repository_id=repository_id, @@ -1462,13 +1507,13 @@ filtered_repo_info_dicts = [ repo_info_dict for repo_info_dict in repo_info_dicts ] created_or_updated_tool_shed_repositories = [ tool_shed_repository ] tool_panel_section_keys.append( tool_panel_section_key ) - # Defaulot the selected tool panel location for loading tools included in each newly installed required tool shed repository to the location + # Default the selected tool panel location for loading tools included in each newly installed required tool shed repository to the location # selected for the repository selected for reinstallation. for index, tps_key in enumerate( tool_panel_section_keys ): if tps_key is None: tool_panel_section_keys[ index ] = tool_panel_section_key encoded_repository_ids = [ trans.security.encode_id( r.id ) for r in created_or_updated_tool_shed_repositories ] - new_kwd = dict( includes_tool_dependencies=tool_shed_repository.includes_tool_dependencies, + new_kwd = dict( includes_tool_dependencies=includes_tool_dependencies, includes_tools=tool_shed_repository.includes_tools, install_tool_dependencies=install_tool_dependencies, repo_info_dicts=filtered_repo_info_dicts, @@ -1539,13 +1584,17 @@ repository_metadata=None, metadata=metadata, repository_dependencies=repository_dependencies ) + if includes_repository_dependencies: + # Discover all repository dependencies and retrieve information for installing them. + required_repo_info_dicts = shed_util.get_required_repo_info_dicts( tool_shed_url, util.listify( repo_info_dict ) ) + else: + required_repo_info_dicts = None # Get the location in the tool panel in which the tool was originally loaded. if 'tool_panel_section' in metadata: tool_panel_dict = metadata[ 'tool_panel_section' ] if tool_panel_dict: if shed_util.panel_entry_per_tool( tool_panel_dict ): - # TODO: Fix this to handle the case where the tools are distributed across in more than 1 ToolSection. The - # following assumes everything was loaded into 1 section (or no section) in the tool panel. + # The following forces everything to be loaded into 1 section (or no section) in the tool panel. tool_section_dicts = tool_panel_dict[ tool_panel_dict.keys()[ 0 ] ] tool_section_dict = tool_section_dicts[ 0 ] original_section_name = tool_section_dict[ 'name' ] @@ -1567,7 +1616,14 @@ message += "The tools contained in your <b>%s</b> repository were last loaded into the tool panel outside of any sections. " % tool_shed_repository.name message += "Uncheck the <b>No changes</b> check box and select a tool panel section to load the tools into that section. " status = 'warning' - containers_dict = shed_util.populate_containers_dict_from_repository_metadata( trans, tool_shed_url, tool_path, tool_shed_repository, reinstalling=True ) + # Populate the containers_dict from the metadata for the tool shed repository we're reinstalling, but make sure to include tool dependencies defined for + # all of the repository's repository dependencies. + containers_dict = shed_util.populate_containers_dict_from_repository_metadata( trans=trans, + tool_shed_url=tool_shed_url, + tool_path=tool_path, + repository=tool_shed_repository, + reinstalling=True, + required_repo_info_dicts=required_repo_info_dicts ) # Since we're reinstalling we'll merge the list of missing repository dependencies into the list of installed repository dependencies since each displayed # repository dependency will display a status, whether installed or missing. containers_dict = suc.merge_missing_repository_dependencies_to_installed_container( containers_dict ) diff -r 530fb4f8204f2106e11f419c381fd53d2319ce24 -r 2ec609faae49b4ab32190403669849f1af0b68f6 templates/admin/tool_shed_repository/common.mako --- a/templates/admin/tool_shed_repository/common.mako +++ b/templates/admin/tool_shed_repository/common.mako @@ -91,6 +91,7 @@ repository_dependencies_root_folder = containers_dict[ 'repository_dependencies' ] tool_dependencies_root_folder = containers_dict[ 'tool_dependencies' ] + missing_tool_dependencies_root_folder = containers_dict[ 'missing_tool_dependencies' ] env_settings_heaader_row_displayed = False package_header_row_displayed = False %> @@ -122,7 +123,7 @@ <div style="clear: both"></div></div> %endif - %if tool_dependencies_root_folder: + %if tool_dependencies_root_folder or missing_tool_dependencies_root_folder: %if install_tool_dependencies_check_box is not None: <div class="form-row"><label>Handle tool dependencies?</label> @@ -138,14 +139,26 @@ </div><div style="clear: both"></div> %endif - <div class="form-row"> - <p/> - <% row_counter = RowCounter() %> - <table cellspacing="2" cellpadding="2" border="0" width="100%" class="tables container-table" id="dependency_table"> - ${render_folder( tool_dependencies_root_folder, 0, parent=None, row_counter=row_counter, is_root_folder=True )} - </table> - <div style="clear: both"></div> - </div> + %if tool_dependencies_root_folder: + <div class="form-row"> + <p/> + <% row_counter = RowCounter() %> + <table cellspacing="2" cellpadding="2" border="0" width="100%" class="tables container-table" id="dependency_table"> + ${render_folder( tool_dependencies_root_folder, 0, parent=None, row_counter=row_counter, is_root_folder=True )} + </table> + <div style="clear: both"></div> + </div> + %endif + %if missing_tool_dependencies_root_folder: + <div class="form-row"> + <p/> + <% row_counter = RowCounter() %> + <table cellspacing="2" cellpadding="2" border="0" width="100%" class="tables container-table" id="dependency_table"> + ${render_folder( missing_tool_dependencies_root_folder, 0, parent=None, row_counter=row_counter, is_root_folder=True )} + </table> + <div style="clear: both"></div> + </div> + %endif %endif </%def> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: dan: Fix for grouping parameters value_from_basic when ignore_errors is True. Fixes issue seen in workflows and rerun where an invalid stored value (e.g. due to changing parameter types) is provided.
by Bitbucket 08 Jan '13

08 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/530fb4f8204f/ changeset: 530fb4f8204f user: dan date: 2013-01-08 16:41:32 summary: Fix for grouping parameters value_from_basic when ignore_errors is True. Fixes issue seen in workflows and rerun where an invalid stored value (e.g. due to changing parameter types) is provided. affected #: 1 file diff -r 94bfcccdaf0625448e623d1d88ba9a1b7785ee78 -r 530fb4f8204f2106e11f419c381fd53d2319ce24 lib/galaxy/tools/parameters/grouping.py --- a/lib/galaxy/tools/parameters/grouping.py +++ b/lib/galaxy/tools/parameters/grouping.py @@ -68,21 +68,25 @@ return rval def value_from_basic( self, value, app, ignore_errors=False ): rval = [] - for i, d in enumerate( value ): - rval_dict = {} - # If the special __index__ key is not set, create it (for backward - # compatibility) - rval_dict['__index__'] = d.get( '__index__', i ) - # Restore child inputs - for input in self.inputs.itervalues(): - if ignore_errors and input.name not in d: - # If we do not have a value, and are ignoring errors, we simply - # do nothing. There will be no value for the parameter in the - # conditional's values dictionary. - pass - else: - rval_dict[ input.name ] = input.value_from_basic( d[input.name], app, ignore_errors ) - rval.append( rval_dict ) + try: + for i, d in enumerate( value ): + rval_dict = {} + # If the special __index__ key is not set, create it (for backward + # compatibility) + rval_dict['__index__'] = d.get( '__index__', i ) + # Restore child inputs + for input in self.inputs.itervalues(): + if ignore_errors and input.name not in d: + # If we do not have a value, and are ignoring errors, we simply + # do nothing. There will be no value for the parameter in the + # conditional's values dictionary. + pass + else: + rval_dict[ input.name ] = input.value_from_basic( d[input.name], app, ignore_errors ) + rval.append( rval_dict ) + except Exception, e: + if not ignore_errors: + raise e return rval def visit_inputs( self, prefix, value, callback ): for i, d in enumerate( value ): @@ -441,24 +445,28 @@ return rval def value_from_basic( self, value, app, ignore_errors=False ): rval = dict() - current_case = rval['__current_case__'] = value['__current_case__'] - # Test param - if ignore_errors and self.test_param.name not in value: - # If ignoring errors, do nothing. However this is potentially very - # problematic since if we are missing the value of test param, - # the entire conditional is wrong. - pass - else: - rval[ self.test_param.name ] = self.test_param.value_from_basic( value[ self.test_param.name ], app, ignore_errors ) - # Inputs associated with current case - for input in self.cases[current_case].inputs.itervalues(): - if ignore_errors and input.name not in value: - # If we do not have a value, and are ignoring errors, we simply - # do nothing. There will be no value for the parameter in the - # conditional's values dictionary. + try: + current_case = rval['__current_case__'] = value['__current_case__'] + # Test param + if ignore_errors and self.test_param.name not in value: + # If ignoring errors, do nothing. However this is potentially very + # problematic since if we are missing the value of test param, + # the entire conditional is wrong. pass else: - rval[ input.name ] = input.value_from_basic( value[ input.name ], app, ignore_errors ) + rval[ self.test_param.name ] = self.test_param.value_from_basic( value[ self.test_param.name ], app, ignore_errors ) + # Inputs associated with current case + for input in self.cases[current_case].inputs.itervalues(): + if ignore_errors and input.name not in value: + # If we do not have a value, and are ignoring errors, we simply + # do nothing. There will be no value for the parameter in the + # conditional's values dictionary. + pass + else: + rval[ input.name ] = input.value_from_basic( value[ input.name ], app, ignore_errors ) + except Exception, e: + if not ignore_errors: + raise e return rval def visit_inputs( self, prefix, value, callback ): current_case = value['__current_case__'] Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Fix bug in bowtie2 wrapper that prevented wrapper from working with old versions of samtools.
by Bitbucket 08 Jan '13

08 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/94bfcccdaf06/ changeset: 94bfcccdaf06 user: jgoecks date: 2013-01-08 15:25:16 summary: Fix bug in bowtie2 wrapper that prevented wrapper from working with old versions of samtools. affected #: 1 file diff -r 989c789fbc43b0d9960655752c6ba396ac7618af -r 94bfcccdaf0625448e623d1d88ba9a1b7785ee78 tools/sr_mapping/bowtie2_wrapper.py --- a/tools/sr_mapping/bowtie2_wrapper.py +++ b/tools/sr_mapping/bowtie2_wrapper.py @@ -71,7 +71,7 @@ index_path = options.index_path # Build bowtie command; use view and sort to create sorted bam. - cmd = 'bowtie2 %s -x %s %s | samtools view -Su - | samtools sort -o - sorted > %s' + cmd = 'bowtie2 %s -x %s %s | samtools view -Su - | samtools sort -o - - > %s' # Set up reads. if options.single_paired == 'paired': Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: james_taylor: comment out two eggs from eggs.ini for now
by Bitbucket 07 Jan '13

07 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/989c789fbc43/ changeset: 989c789fbc43 user: james_taylor date: 2013-01-07 22:25:54 summary: comment out two eggs from eggs.ini for now affected #: 1 file diff -r d8606b61644a09719a8042efd2b23b7aa7869ff2 -r 989c789fbc43b0d9960655752c6ba396ac7618af eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -29,7 +29,7 @@ simplejson = 2.1.1 threadframe = 0.2 guppy = 0.1.8 -msgpack_python = 0.2.4 +; msgpack_python = 0.2.4 [eggs:noplatform] amqplib = 0.6.1 @@ -66,7 +66,7 @@ Babel = 0.9.4 wchartype = 0.1 Whoosh = 0.3.18 -fluent_logger = 0.3.3 +; fluent_logger = 0.3.3 ; extra version information [tags] Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: inithello: Revert fix from 8530:df20658ac499.
by Bitbucket 07 Jan '13

07 Jan '13

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/d8606b61644a/ changeset: d8606b61644a user: inithello date: 2013-01-07 22:20:23 summary: Revert fix from 8530:df20658ac499. affected #: 1 file diff -r 09cf284087021586ad08656b7ea444959c6c49bf -r d8606b61644a09719a8042efd2b23b7aa7869ff2 lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py --- a/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py +++ b/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py @@ -848,7 +848,8 @@ tool_section = trans.app.toolbox.tool_panel[ tool_panel_section_key ] else: tool_section = None - repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) + if isinstance( repo_info_dict, basestring ): + repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) # Clone each repository to the configured location. shed_util.update_tool_shed_repository_status( trans.app, tool_shed_repository, trans.model.ToolShedRepository.installation_status.CLONING ) repo_info_tuple = repo_info_dict[ tool_shed_repository.name ] @@ -1418,7 +1419,8 @@ repo_info_dicts = [] repo_info_dict = kwd.get( 'repo_info_dict', None ) if repo_info_dict: - repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) + if isinstance( repo_info_dict, basestring ): + repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) else: # Entering this else block used to happen only if the tool_shed_repository does not include any valid tools. After repository dependencies # were introduced, it may never happen, but we'll keep the block just in case. Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 7 new changesets
by Bitbucket 07 Jan '13

07 Jan '13

7 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/67379010fc77/ changeset: 67379010fc77 user: james_taylor date: 2012-10-02 17:03:48 summary: tracing: first pass trace logging to fluentd affected #: 9 files diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -29,6 +29,7 @@ simplejson = 2.1.1 threadframe = 0.2 guppy = 0.1.8 +msgpack_python = 0.2.2 [eggs:noplatform] amqplib = 0.6.1 @@ -65,6 +66,7 @@ Babel = 0.9.4 wchartype = 0.1 Whoosh = 0.3.18 +fluent_logger = 0.3.3 ; extra version information [tags] diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/app.py --- a/lib/galaxy/app.py +++ b/lib/galaxy/app.py @@ -28,6 +28,7 @@ self.config = config.Configuration( **kwargs ) self.config.check() config.configure_logging( self.config ) + self.configure_fluent_log() # Determine the database url if self.config.database_connection: db_url = self.config.database_connection @@ -53,7 +54,8 @@ db_url, self.config.database_engine_options, database_query_profiling_proxy = self.config.database_query_profiling_proxy, - object_store = self.object_store ) + object_store = self.object_store, + trace_logger=self.trace_logger ) # Manage installed tool shed repositories. self.installed_repository_manager = galaxy.tool_shed.InstalledRepositoryManager( self ) # Create an empty datatypes registry. @@ -143,6 +145,7 @@ self.job_stop_queue = self.job_manager.job_stop_queue # Initialize the external service types self.external_service_types = external_service_types.ExternalServiceTypesCollection( self.config.external_service_type_config_file, self.config.external_service_type_path, self ) + def shutdown( self ): self.job_manager.shutdown() self.object_store.shutdown() @@ -155,3 +158,10 @@ os.unlink( self.datatypes_registry.integrated_datatypes_configs ) except: pass + + def configure_fluent_log( self ): + if self.config.fluent_log: + from galaxy.util.log.fluent_log import FluentTraceLogger + self.trace_logger = FluentTraceLogger( 'galaxy', self.config.fluent_host, self.config.fluent_port ) + else: + self.trace_logger = None diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/config.py --- a/lib/galaxy/config.py +++ b/lib/galaxy/config.py @@ -232,6 +232,11 @@ for k, v in amqp_config: self.amqp[k] = v self.running_functional_tests = string_as_bool( kwargs.get( 'running_functional_tests', False ) ) + # Logging with fluentd + self.fluent_log = string_as_bool( kwargs.get( 'fluent_log', False ) ) + self.fluent_host = kwargs.get( 'fluent_host', 'localhost' ) + self.fluent_port = int( kwargs.get( 'fluent_port', 24224 ) ) + def __read_tool_job_config( self, global_conf_parser, section, key ): try: tool_runners_config = global_conf_parser.items( section ) diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/model/mapping.py --- a/lib/galaxy/model/mapping.py +++ b/lib/galaxy/model/mapping.py @@ -1868,7 +1868,7 @@ # Let this go, it could possibly work with db's we don't support log.error( "database_connection contains an unknown SQLAlchemy database dialect: %s" % dialect ) -def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None ): +def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None, trace_logger=None ): """Connect mappings to the database""" # Connect dataset to the file path Dataset.file_path = file_path @@ -1876,10 +1876,10 @@ Dataset.object_store = object_store # Load the appropriate db module load_egg_for_url( url ) - # Should we use the logging proxy? - if database_query_profiling_proxy: + # If metlog is enabled, do micrologging + if trace_logger: import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy - proxy = logging_connection_proxy.LoggingProxy() + proxy = logging_connection_proxy.TraceLoggerProxy( trace_logger ) else: proxy = None # Create the database engine diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/model/orm/logging_connection_proxy.py --- a/lib/galaxy/model/orm/logging_connection_proxy.py +++ b/lib/galaxy/model/orm/logging_connection_proxy.py @@ -18,13 +18,31 @@ rval = [] for frame, fname, line, funcname, _, _ in inspect.stack()[2:]: rval.append( "%s:%s@%d" % ( stripwd( fname ), funcname, line ) ) - return " > ".join( rval ) + return rval class LoggingProxy(ConnectionProxy): + """ + Logs SQL statements using standard logging module + """ def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): start = time.clock() rval = execute(cursor, statement, parameters, context) duration = time.clock() - start log.debug( "statement: %r parameters: %r executemany: %r duration: %r stack: %r", - statement, parameters, executemany, duration, pretty_stack() ) + statement, parameters, executemany, duration, " > ".join( pretty_stack() ) ) return rval + +class TraceLoggerProxy(ConnectionProxy): + """ + Logs SQL statements using a metlog client + """ + def __init__( self, trace_logger ): + self.trace_logger = trace_logger + def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): + start = time.clock() + rval = execute(cursor, statement, parameters, context) + duration = time.clock() - start + self.trace_logger.log( "sqlalchemy_query", + message="Query executed", statement=statement, parameters=parameters, + executemany=executemany, duration=duration, stack=pretty_stack() ) + return rval \ No newline at end of file diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/util/log/__init__.py --- /dev/null +++ b/lib/galaxy/util/log/__init__.py @@ -0,0 +1,5 @@ +class TraceLogger( object ): + def __init__( self, name ): + self.name = name + def log( **kwargs ): + raise TypeError( "Abstract Method" ) \ No newline at end of file diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/util/log/fluent_log.py --- /dev/null +++ b/lib/galaxy/util/log/fluent_log.py @@ -0,0 +1,37 @@ +import time +import threading + +import galaxy.eggs +galaxy.eggs.require( "fluent-logger" ) +galaxy.eggs.require( "msgpack_python" ) + + +from fluent.sender import FluentSender + + +class FluentTraceLogger( object ): + def __init__( self, name, host='localhost', port=24224 ): + self.lock = threading.Lock() + self.thread_local = threading.local() + self.name = name + self.sender = FluentSender( self.name, host=host, port=port ) + + def context_push( self, value ): + self.lock.acquire() + if not hasattr( self.thread_local, 'context' ): + self.thread_local.context = [] + self.thread_local.context.append( value ) + self.lock.release() + + def context_pop( self ): + self.lock.acquire() + self.thread_local.context.pop() + self.lock.release() + + def log( self, label, **kwargs ): + self.lock.acquire() + if not hasattr( self.thread_local, 'context' ): + self.thread_local.context = [] + self.lock.release() + kwargs['log_context'] = self.thread_local.context + self.sender.emit_with_time( label, int(time.time()), kwargs ) \ No newline at end of file diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py +++ b/lib/galaxy/web/framework/base.py @@ -8,6 +8,8 @@ import os.path import sys import tarfile +import threading +import uuid from Cookie import SimpleCookie @@ -68,6 +70,9 @@ self.mapper.explicit = False self.api_mapper = routes.Mapper() self.transaction_factory = DefaultWebTransaction + # Each request will have a unique id. Since we are assuming + # a threaded model for the moment we can store that here + self.request_id = threading.local() def add_ui_controller( self, controller_name, controller ): """ Add a controller class to this application. A controller class has @@ -106,12 +111,20 @@ # Create/compile the regular expressions for route mapping self.mapper.create_regs( self.controllers.keys() ) self.api_mapper.create_regs( self.api_controllers.keys() ) + def trace( self, **fields ): + if self.trace_logger: + self.trace_logger.log( "WebApplication", **fields ) def __call__( self, environ, start_response ): """ Call interface as specified by WSGI. Wraps the environment in user friendly objects, finds the appropriate method to handle the request and calls it. """ + # Immediately create request_id which we will use for logging + self.request_id = request_id = uuid.uuid1().hex + if self.trace_logger: + self.trace_logger.context_push( dict( request_id = request_id ) ) + self.trace( message= "Starting request" ) # Map url using routes path_info = environ.get( 'PATH_INFO', '' ) map = self.mapper.match( path_info, environ ) @@ -125,6 +138,7 @@ controllers = self.controllers if map == None: raise httpexceptions.HTTPNotFound( "No route for " + path_info ) + self.trace( path_info=path_info, map=map ) # Setup routes rc = routes.request_config() rc.mapper = mapper diff -r 0445cd851094b8bad61d2a96f399538f74e5db03 -r 67379010fc77b462178e3d580e660df220d20634 lib/galaxy/webapps/galaxy/buildapp.py --- a/lib/galaxy/webapps/galaxy/buildapp.py +++ b/lib/galaxy/webapps/galaxy/buildapp.py @@ -110,7 +110,13 @@ webapp.api_mapper.connect("import_workflow", "/api/workflows/upload", controller="workflows", action="import_new_workflow", conditions=dict(method=["POST"])) webapp.api_mapper.connect("workflow_dict", '/api/workflows/download/{workflow_id}', controller='workflows', action='workflow_dict', conditions=dict(method=['GET'])) + # Connect logger from app + if app.trace_logger: + webapp.trace_logger = app.trace_logger + + # Indicate that all configuration settings have been provided webapp.finalize_config() + # Wrap the webapp in some useful middleware if kwargs.get( 'middleware', True ): webapp = wrap_in_middleware( webapp, global_conf, **kwargs ) https://bitbucket.org/galaxy/galaxy-central/commits/33d256f3121a/ changeset: 33d256f3121a user: james_taylor date: 2013-01-03 22:54:14 summary: merge affected #: 9 files diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -29,6 +29,7 @@ simplejson = 2.1.1 threadframe = 0.2 guppy = 0.1.8 +msgpack_python = 0.2.2 [eggs:noplatform] amqplib = 0.6.1 @@ -65,6 +66,7 @@ Babel = 0.9.4 wchartype = 0.1 Whoosh = 0.3.18 +fluent_logger = 0.3.3 ; extra version information [tags] diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/app.py --- a/lib/galaxy/app.py +++ b/lib/galaxy/app.py @@ -29,6 +29,7 @@ self.config = config.Configuration( **kwargs ) self.config.check() config.configure_logging( self.config ) + self.configure_fluent_log() # Determine the database url if self.config.database_connection: db_url = self.config.database_connection @@ -54,7 +55,8 @@ db_url, self.config.database_engine_options, database_query_profiling_proxy = self.config.database_query_profiling_proxy, - object_store = self.object_store ) + object_store = self.object_store, + trace_logger=self.trace_logger ) # Manage installed tool shed repositories. self.installed_repository_manager = galaxy.tool_shed.InstalledRepositoryManager( self ) # Create an empty datatypes registry. @@ -149,6 +151,7 @@ self.job_stop_queue = self.job_manager.job_stop_queue # Initialize the external service types self.external_service_types = external_service_types.ExternalServiceTypesCollection( self.config.external_service_type_config_file, self.config.external_service_type_path, self ) + def shutdown( self ): self.job_manager.shutdown() self.object_store.shutdown() @@ -161,3 +164,10 @@ os.unlink( self.datatypes_registry.integrated_datatypes_configs ) except: pass + + def configure_fluent_log( self ): + if self.config.fluent_log: + from galaxy.util.log.fluent_log import FluentTraceLogger + self.trace_logger = FluentTraceLogger( 'galaxy', self.config.fluent_host, self.config.fluent_port ) + else: + self.trace_logger = None diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/config.py --- a/lib/galaxy/config.py +++ b/lib/galaxy/config.py @@ -261,6 +261,10 @@ self.api_folders = string_as_bool( kwargs.get( 'api_folders', False ) ) # This is for testing new library browsing capabilities. self.new_lib_browse = string_as_bool( kwargs.get( 'new_lib_browse', False ) ) + # Logging with fluentd + self.fluent_log = string_as_bool( kwargs.get( 'fluent_log', False ) ) + self.fluent_host = kwargs.get( 'fluent_host', 'localhost' ) + self.fluent_port = int( kwargs.get( 'fluent_port', 24224 ) ) def __read_tool_job_config( self, global_conf_parser, section, key ): try: diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/model/mapping.py --- a/lib/galaxy/model/mapping.py +++ b/lib/galaxy/model/mapping.py @@ -1950,7 +1950,7 @@ # Let this go, it could possibly work with db's we don't support log.error( "database_connection contains an unknown SQLAlchemy database dialect: %s" % dialect ) -def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None ): +def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None, trace_logger=None ): """Connect mappings to the database""" # Connect dataset to the file path Dataset.file_path = file_path @@ -1958,10 +1958,10 @@ Dataset.object_store = object_store # Load the appropriate db module load_egg_for_url( url ) - # Should we use the logging proxy? - if database_query_profiling_proxy: + # If metlog is enabled, do micrologging + if trace_logger: import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy - proxy = logging_connection_proxy.LoggingProxy() + proxy = logging_connection_proxy.TraceLoggerProxy( trace_logger ) else: proxy = None # Create the database engine diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/model/orm/logging_connection_proxy.py --- a/lib/galaxy/model/orm/logging_connection_proxy.py +++ b/lib/galaxy/model/orm/logging_connection_proxy.py @@ -18,13 +18,31 @@ rval = [] for frame, fname, line, funcname, _, _ in inspect.stack()[2:]: rval.append( "%s:%s@%d" % ( stripwd( fname ), funcname, line ) ) - return " > ".join( rval ) + return rval class LoggingProxy(ConnectionProxy): + """ + Logs SQL statements using standard logging module + """ def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): start = time.clock() rval = execute(cursor, statement, parameters, context) duration = time.clock() - start log.debug( "statement: %r parameters: %r executemany: %r duration: %r stack: %r", - statement, parameters, executemany, duration, pretty_stack() ) + statement, parameters, executemany, duration, " > ".join( pretty_stack() ) ) return rval + +class TraceLoggerProxy(ConnectionProxy): + """ + Logs SQL statements using a metlog client + """ + def __init__( self, trace_logger ): + self.trace_logger = trace_logger + def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): + start = time.clock() + rval = execute(cursor, statement, parameters, context) + duration = time.clock() - start + self.trace_logger.log( "sqlalchemy_query", + message="Query executed", statement=statement, parameters=parameters, + executemany=executemany, duration=duration, stack=pretty_stack() ) + return rval \ No newline at end of file diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/util/log/__init__.py --- /dev/null +++ b/lib/galaxy/util/log/__init__.py @@ -0,0 +1,5 @@ +class TraceLogger( object ): + def __init__( self, name ): + self.name = name + def log( **kwargs ): + raise TypeError( "Abstract Method" ) \ No newline at end of file diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/util/log/fluent_log.py --- /dev/null +++ b/lib/galaxy/util/log/fluent_log.py @@ -0,0 +1,37 @@ +import time +import threading + +import galaxy.eggs +galaxy.eggs.require( "fluent-logger" ) +galaxy.eggs.require( "msgpack_python" ) + + +from fluent.sender import FluentSender + + +class FluentTraceLogger( object ): + def __init__( self, name, host='localhost', port=24224 ): + self.lock = threading.Lock() + self.thread_local = threading.local() + self.name = name + self.sender = FluentSender( self.name, host=host, port=port ) + + def context_push( self, value ): + self.lock.acquire() + if not hasattr( self.thread_local, 'context' ): + self.thread_local.context = [] + self.thread_local.context.append( value ) + self.lock.release() + + def context_pop( self ): + self.lock.acquire() + self.thread_local.context.pop() + self.lock.release() + + def log( self, label, **kwargs ): + self.lock.acquire() + if not hasattr( self.thread_local, 'context' ): + self.thread_local.context = [] + self.lock.release() + kwargs['log_context'] = self.thread_local.context + self.sender.emit_with_time( label, int(time.time()), kwargs ) \ No newline at end of file diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py +++ b/lib/galaxy/web/framework/base.py @@ -8,6 +8,8 @@ import os.path import sys import tarfile +import threading +import uuid from Cookie import SimpleCookie @@ -68,6 +70,9 @@ self.mapper.explicit = False self.api_mapper = routes.Mapper() self.transaction_factory = DefaultWebTransaction + # Each request will have a unique id. Since we are assuming + # a threaded model for the moment we can store that here + self.request_id = threading.local() def add_ui_controller( self, controller_name, controller ): """ Add a controller class to this application. A controller class has @@ -106,12 +111,20 @@ # Create/compile the regular expressions for route mapping self.mapper.create_regs( self.controllers.keys() ) self.api_mapper.create_regs( self.api_controllers.keys() ) + def trace( self, **fields ): + if self.trace_logger: + self.trace_logger.log( "WebApplication", **fields ) def __call__( self, environ, start_response ): """ Call interface as specified by WSGI. Wraps the environment in user friendly objects, finds the appropriate method to handle the request and calls it. """ + # Immediately create request_id which we will use for logging + self.request_id = request_id = uuid.uuid1().hex + if self.trace_logger: + self.trace_logger.context_push( dict( request_id = request_id ) ) + self.trace( message= "Starting request" ) # Map url using routes path_info = environ.get( 'PATH_INFO', '' ) map = self.mapper.match( path_info, environ ) @@ -125,6 +138,7 @@ controllers = self.controllers if map == None: raise httpexceptions.HTTPNotFound( "No route for " + path_info ) + self.trace( path_info=path_info, map=map ) # Setup routes rc = routes.request_config() rc.mapper = mapper diff -r d7475647cbb6a1c70218049fadc24ed1651d845a -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e lib/galaxy/webapps/galaxy/buildapp.py --- a/lib/galaxy/webapps/galaxy/buildapp.py +++ b/lib/galaxy/webapps/galaxy/buildapp.py @@ -130,7 +130,13 @@ webapp.api_mapper.connect("import_workflow", "/api/workflows/upload", controller="workflows", action="import_new_workflow", conditions=dict(method=["POST"])) webapp.api_mapper.connect("workflow_dict", '/api/workflows/download/{workflow_id}', controller='workflows', action='workflow_dict', conditions=dict(method=['GET'])) + # Connect logger from app + if app.trace_logger: + webapp.trace_logger = app.trace_logger + + # Indicate that all configuration settings have been provided webapp.finalize_config() + # Wrap the webapp in some useful middleware if kwargs.get( 'middleware', True ): webapp = wrap_in_middleware( webapp, global_conf, **kwargs ) https://bitbucket.org/galaxy/galaxy-central/commits/33750f347be2/ changeset: 33750f347be2 user: james_taylor date: 2013-01-04 20:26:22 summary: update msgpack_python version for trace logging, don't log entire stack with each query affected #: 2 files diff -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e -r 33750f347be2aef72fe4e32d5c2197dd82f2b45a eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -29,7 +29,7 @@ simplejson = 2.1.1 threadframe = 0.2 guppy = 0.1.8 -msgpack_python = 0.2.2 +msgpack_python = 0.2.4 [eggs:noplatform] amqplib = 0.6.1 diff -r 33d256f3121a7f32aaa87b4e34a110fcea57a36e -r 33750f347be2aef72fe4e32d5c2197dd82f2b45a lib/galaxy/model/orm/logging_connection_proxy.py --- a/lib/galaxy/model/orm/logging_connection_proxy.py +++ b/lib/galaxy/model/orm/logging_connection_proxy.py @@ -44,5 +44,5 @@ duration = time.clock() - start self.trace_logger.log( "sqlalchemy_query", message="Query executed", statement=statement, parameters=parameters, - executemany=executemany, duration=duration, stack=pretty_stack() ) + executemany=executemany, duration=duration ) return rval \ No newline at end of file https://bitbucket.org/galaxy/galaxy-central/commits/8331f2af6a90/ changeset: 8331f2af6a90 user: james_taylor date: 2013-01-04 22:48:35 summary: trace logging: flatten context affected #: 2 files diff -r 33750f347be2aef72fe4e32d5c2197dd82f2b45a -r 8331f2af6a9092f4528fb01ca7baf3d7999838a0 lib/galaxy/util/log/fluent_log.py --- a/lib/galaxy/util/log/fluent_log.py +++ b/lib/galaxy/util/log/fluent_log.py @@ -1,3 +1,7 @@ +""" +Provides a `TraceLogger` implementation that logs to a fluentd collector +""" + import time import threading @@ -5,33 +9,31 @@ galaxy.eggs.require( "fluent-logger" ) galaxy.eggs.require( "msgpack_python" ) - from fluent.sender import FluentSender class FluentTraceLogger( object ): - def __init__( self, name, host='localhost', port=24224 ): - self.lock = threading.Lock() - self.thread_local = threading.local() - self.name = name - self.sender = FluentSender( self.name, host=host, port=port ) + def __init__( self, name, host='localhost', port=24224 ): + self.lock = threading.Lock() + self.thread_local = threading.local() + self.name = name + self.sender = FluentSender( self.name, host=host, port=port ) - def context_push( self, value ): - self.lock.acquire() - if not hasattr( self.thread_local, 'context' ): - self.thread_local.context = [] - self.thread_local.context.append( value ) - self.lock.release() + def context_set( self, key, value ): + self.lock.acquire() + if not hasattr( self.thread_local, 'context' ): + self.thread_local.context = {} + self.thread_local.context[key] = value + self.lock.release() - def context_pop( self ): - self.lock.acquire() - self.thread_local.context.pop() - self.lock.release() + def context_remove( self, key ): + self.lock.acquire() + del self.thread_local.context[key] + self.lock.release() - def log( self, label, **kwargs ): - self.lock.acquire() - if not hasattr( self.thread_local, 'context' ): - self.thread_local.context = [] - self.lock.release() - kwargs['log_context'] = self.thread_local.context - self.sender.emit_with_time( label, int(time.time()), kwargs ) \ No newline at end of file + def log( self, label, **kwargs ): + self.lock.acquire() + if hasattr( self.thread_local, 'context' ): + kwargs.update( self.thread_local.context ) + self.lock.release() + self.sender.emit_with_time( label, int(time.time()), kwargs ) \ No newline at end of file diff -r 33750f347be2aef72fe4e32d5c2197dd82f2b45a -r 8331f2af6a9092f4528fb01ca7baf3d7999838a0 lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py +++ b/lib/galaxy/web/framework/base.py @@ -123,8 +123,16 @@ # Immediately create request_id which we will use for logging self.request_id = request_id = uuid.uuid1().hex if self.trace_logger: - self.trace_logger.context_push( dict( request_id = request_id ) ) - self.trace( message= "Starting request" ) + self.trace_logger.context_set( "request_id", request_id ) + self.trace( message="Starting request" ) + try: + return self.handle_request( environ, start_response ) + finally: + self.trace( message="Handle request finished" ) + if self.trace_logger: + self.trace_logger.context_remove( "request_id" ) + + def handle_request( self, environ, start_response ): # Map url using routes path_info = environ.get( 'PATH_INFO', '' ) map = self.mapper.match( path_info, environ ) https://bitbucket.org/galaxy/galaxy-central/commits/26f38d9bd3ee/ changeset: 26f38d9bd3ee user: james_taylor date: 2013-01-07 21:07:16 summary: Restore support for logging connection proxy affected #: 1 file diff -r 8331f2af6a9092f4528fb01ca7baf3d7999838a0 -r 26f38d9bd3ee9902aa5661d046604f6fb2bb96b3 lib/galaxy/model/mapping.py --- a/lib/galaxy/model/mapping.py +++ b/lib/galaxy/model/mapping.py @@ -1958,8 +1958,12 @@ Dataset.object_store = object_store # Load the appropriate db module load_egg_for_url( url ) + # Should we use the logging proxy? + if database_query_profiling_proxy: + import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy + proxy = logging_connection_proxy.LoggingProxy() # If metlog is enabled, do micrologging - if trace_logger: + elif trace_logger: import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy proxy = logging_connection_proxy.TraceLoggerProxy( trace_logger ) else: https://bitbucket.org/galaxy/galaxy-central/commits/5622f8127e9d/ changeset: 5622f8127e9d user: james_taylor date: 2013-01-07 21:12:18 summary: tracing: fix for when trace logger is disabled affected #: 1 file diff -r 26f38d9bd3ee9902aa5661d046604f6fb2bb96b3 -r 5622f8127e9ddf52baba4ad901e32deacd378f7c lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py +++ b/lib/galaxy/web/framework/base.py @@ -73,6 +73,8 @@ # Each request will have a unique id. Since we are assuming # a threaded model for the moment we can store that here self.request_id = threading.local() + # Set if trace logging is enabled + self.trace_logger = None def add_ui_controller( self, controller_name, controller ): """ Add a controller class to this application. A controller class has @@ -114,6 +116,7 @@ def trace( self, **fields ): if self.trace_logger: self.trace_logger.log( "WebApplication", **fields ) + def __call__( self, environ, start_response ): """ Call interface as specified by WSGI. Wraps the environment in user https://bitbucket.org/galaxy/galaxy-central/commits/09cf28408702/ changeset: 09cf28408702 user: james_taylor date: 2013-01-07 21:12:47 summary: Automated merge with ssh://bitbucket.org/galaxy/galaxy-central affected #: 9 files diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf eggs.ini --- a/eggs.ini +++ b/eggs.ini @@ -29,6 +29,7 @@ simplejson = 2.1.1 threadframe = 0.2 guppy = 0.1.8 +msgpack_python = 0.2.4 [eggs:noplatform] amqplib = 0.6.1 @@ -65,6 +66,7 @@ Babel = 0.9.4 wchartype = 0.1 Whoosh = 0.3.18 +fluent_logger = 0.3.3 ; extra version information [tags] diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/app.py --- a/lib/galaxy/app.py +++ b/lib/galaxy/app.py @@ -29,6 +29,7 @@ self.config = config.Configuration( **kwargs ) self.config.check() config.configure_logging( self.config ) + self.configure_fluent_log() # Determine the database url if self.config.database_connection: db_url = self.config.database_connection @@ -54,7 +55,8 @@ db_url, self.config.database_engine_options, database_query_profiling_proxy = self.config.database_query_profiling_proxy, - object_store = self.object_store ) + object_store = self.object_store, + trace_logger=self.trace_logger ) # Manage installed tool shed repositories. self.installed_repository_manager = galaxy.tool_shed.InstalledRepositoryManager( self ) # Create an empty datatypes registry. @@ -149,6 +151,7 @@ self.job_stop_queue = self.job_manager.job_stop_queue # Initialize the external service types self.external_service_types = external_service_types.ExternalServiceTypesCollection( self.config.external_service_type_config_file, self.config.external_service_type_path, self ) + def shutdown( self ): self.job_manager.shutdown() self.object_store.shutdown() @@ -161,3 +164,10 @@ os.unlink( self.datatypes_registry.integrated_datatypes_configs ) except: pass + + def configure_fluent_log( self ): + if self.config.fluent_log: + from galaxy.util.log.fluent_log import FluentTraceLogger + self.trace_logger = FluentTraceLogger( 'galaxy', self.config.fluent_host, self.config.fluent_port ) + else: + self.trace_logger = None diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/config.py --- a/lib/galaxy/config.py +++ b/lib/galaxy/config.py @@ -261,6 +261,10 @@ self.api_folders = string_as_bool( kwargs.get( 'api_folders', False ) ) # This is for testing new library browsing capabilities. self.new_lib_browse = string_as_bool( kwargs.get( 'new_lib_browse', False ) ) + # Logging with fluentd + self.fluent_log = string_as_bool( kwargs.get( 'fluent_log', False ) ) + self.fluent_host = kwargs.get( 'fluent_host', 'localhost' ) + self.fluent_port = int( kwargs.get( 'fluent_port', 24224 ) ) def __read_tool_job_config( self, global_conf_parser, section, key ): try: diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/model/mapping.py --- a/lib/galaxy/model/mapping.py +++ b/lib/galaxy/model/mapping.py @@ -1950,7 +1950,7 @@ # Let this go, it could possibly work with db's we don't support log.error( "database_connection contains an unknown SQLAlchemy database dialect: %s" % dialect ) -def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None ): +def init( file_path, url, engine_options={}, create_tables=False, database_query_profiling_proxy=False, object_store=None, trace_logger=None ): """Connect mappings to the database""" # Connect dataset to the file path Dataset.file_path = file_path @@ -1962,6 +1962,10 @@ if database_query_profiling_proxy: import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy proxy = logging_connection_proxy.LoggingProxy() + # If metlog is enabled, do micrologging + elif trace_logger: + import galaxy.model.orm.logging_connection_proxy as logging_connection_proxy + proxy = logging_connection_proxy.TraceLoggerProxy( trace_logger ) else: proxy = None # Create the database engine diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/model/orm/logging_connection_proxy.py --- a/lib/galaxy/model/orm/logging_connection_proxy.py +++ b/lib/galaxy/model/orm/logging_connection_proxy.py @@ -18,13 +18,31 @@ rval = [] for frame, fname, line, funcname, _, _ in inspect.stack()[2:]: rval.append( "%s:%s@%d" % ( stripwd( fname ), funcname, line ) ) - return " > ".join( rval ) + return rval class LoggingProxy(ConnectionProxy): + """ + Logs SQL statements using standard logging module + """ def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): start = time.clock() rval = execute(cursor, statement, parameters, context) duration = time.clock() - start log.debug( "statement: %r parameters: %r executemany: %r duration: %r stack: %r", - statement, parameters, executemany, duration, pretty_stack() ) + statement, parameters, executemany, duration, " > ".join( pretty_stack() ) ) return rval + +class TraceLoggerProxy(ConnectionProxy): + """ + Logs SQL statements using a metlog client + """ + def __init__( self, trace_logger ): + self.trace_logger = trace_logger + def cursor_execute(self, execute, cursor, statement, parameters, context, executemany): + start = time.clock() + rval = execute(cursor, statement, parameters, context) + duration = time.clock() - start + self.trace_logger.log( "sqlalchemy_query", + message="Query executed", statement=statement, parameters=parameters, + executemany=executemany, duration=duration ) + return rval \ No newline at end of file diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/util/log/__init__.py --- /dev/null +++ b/lib/galaxy/util/log/__init__.py @@ -0,0 +1,5 @@ +class TraceLogger( object ): + def __init__( self, name ): + self.name = name + def log( **kwargs ): + raise TypeError( "Abstract Method" ) \ No newline at end of file diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/util/log/fluent_log.py --- /dev/null +++ b/lib/galaxy/util/log/fluent_log.py @@ -0,0 +1,39 @@ +""" +Provides a `TraceLogger` implementation that logs to a fluentd collector +""" + +import time +import threading + +import galaxy.eggs +galaxy.eggs.require( "fluent-logger" ) +galaxy.eggs.require( "msgpack_python" ) + +from fluent.sender import FluentSender + + +class FluentTraceLogger( object ): + def __init__( self, name, host='localhost', port=24224 ): + self.lock = threading.Lock() + self.thread_local = threading.local() + self.name = name + self.sender = FluentSender( self.name, host=host, port=port ) + + def context_set( self, key, value ): + self.lock.acquire() + if not hasattr( self.thread_local, 'context' ): + self.thread_local.context = {} + self.thread_local.context[key] = value + self.lock.release() + + def context_remove( self, key ): + self.lock.acquire() + del self.thread_local.context[key] + self.lock.release() + + def log( self, label, **kwargs ): + self.lock.acquire() + if hasattr( self.thread_local, 'context' ): + kwargs.update( self.thread_local.context ) + self.lock.release() + self.sender.emit_with_time( label, int(time.time()), kwargs ) \ No newline at end of file diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/web/framework/base.py --- a/lib/galaxy/web/framework/base.py +++ b/lib/galaxy/web/framework/base.py @@ -8,6 +8,8 @@ import os.path import sys import tarfile +import threading +import uuid from Cookie import SimpleCookie @@ -68,6 +70,11 @@ self.mapper.explicit = False self.api_mapper = routes.Mapper() self.transaction_factory = DefaultWebTransaction + # Each request will have a unique id. Since we are assuming + # a threaded model for the moment we can store that here + self.request_id = threading.local() + # Set if trace logging is enabled + self.trace_logger = None def add_ui_controller( self, controller_name, controller ): """ Add a controller class to this application. A controller class has @@ -106,12 +113,29 @@ # Create/compile the regular expressions for route mapping self.mapper.create_regs( self.controllers.keys() ) self.api_mapper.create_regs( self.api_controllers.keys() ) + def trace( self, **fields ): + if self.trace_logger: + self.trace_logger.log( "WebApplication", **fields ) + def __call__( self, environ, start_response ): """ Call interface as specified by WSGI. Wraps the environment in user friendly objects, finds the appropriate method to handle the request and calls it. """ + # Immediately create request_id which we will use for logging + self.request_id = request_id = uuid.uuid1().hex + if self.trace_logger: + self.trace_logger.context_set( "request_id", request_id ) + self.trace( message="Starting request" ) + try: + return self.handle_request( environ, start_response ) + finally: + self.trace( message="Handle request finished" ) + if self.trace_logger: + self.trace_logger.context_remove( "request_id" ) + + def handle_request( self, environ, start_response ): # Map url using routes path_info = environ.get( 'PATH_INFO', '' ) map = self.mapper.match( path_info, environ ) @@ -125,6 +149,7 @@ controllers = self.controllers if map == None: raise httpexceptions.HTTPNotFound( "No route for " + path_info ) + self.trace( path_info=path_info, map=map ) # Setup routes rc = routes.request_config() rc.mapper = mapper diff -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 -r 09cf284087021586ad08656b7ea444959c6c49bf lib/galaxy/webapps/galaxy/buildapp.py --- a/lib/galaxy/webapps/galaxy/buildapp.py +++ b/lib/galaxy/webapps/galaxy/buildapp.py @@ -130,7 +130,13 @@ webapp.api_mapper.connect("import_workflow", "/api/workflows/upload", controller="workflows", action="import_new_workflow", conditions=dict(method=["POST"])) webapp.api_mapper.connect("workflow_dict", '/api/workflows/download/{workflow_id}', controller='workflows', action='workflow_dict', conditions=dict(method=['GET'])) + # Connect logger from app + if app.trace_logger: + webapp.trace_logger = app.trace_logger + + # Indicate that all configuration settings have been provided webapp.finalize_config() + # Wrap the webapp in some useful middleware if kwargs.get( 'middleware', True ): webapp = wrap_in_middleware( webapp, global_conf, **kwargs ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 2 new changesets
by Bitbucket 07 Jan '13

07 Jan '13

2 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/commits/df20658ac499/ changeset: df20658ac499 user: inithello date: 2013-01-07 18:19:00 summary: Fix for encoded repo_info_dict being passed instead of decoded. affected #: 1 file diff -r ce62bf5a91f86d0c53764bda3df4975486d7512e -r df20658ac4991d970bfd9d63feae73850db84bb7 lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py --- a/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py +++ b/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py @@ -848,8 +848,7 @@ tool_section = trans.app.toolbox.tool_panel[ tool_panel_section_key ] else: tool_section = None - if isinstance( repo_info_dict, basestring ): - repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) + repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) # Clone each repository to the configured location. shed_util.update_tool_shed_repository_status( trans.app, tool_shed_repository, trans.model.ToolShedRepository.installation_status.CLONING ) repo_info_tuple = repo_info_dict[ tool_shed_repository.name ] @@ -1419,8 +1418,7 @@ repo_info_dicts = [] repo_info_dict = kwd.get( 'repo_info_dict', None ) if repo_info_dict: - if isinstance( repo_info_dict, basestring ): - repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) + repo_info_dict = encoding_util.tool_shed_decode( repo_info_dict ) else: # Entering this else block used to happen only if the tool_shed_repository does not include any valid tools. After repository dependencies # were introduced, it may never happen, but we'll keep the block just in case. @@ -1442,12 +1440,9 @@ repository_metadata=None, metadata=metadata, repository_dependencies=repository_dependencies ) - repo_info_dict = encoding_util.tool_shed_encode( repo_info_dict ) repo_info_dicts.append( repo_info_dict ) # Make sure all tool_shed_repository records exist. if install_repository_dependencies: - # This is a bit screwy because filtered_repo_info_dicts in this block is a list of tool_shed_encoded dictionaries, but - # in the associated else block, it is a list of unencoded dictionaries - not sure if this should be corrected... created_or_updated_tool_shed_repositories, tool_panel_section_keys, repo_info_dicts, filtered_repo_info_dicts, message = \ shed_util.create_repository_dependency_objects( trans, tool_path, https://bitbucket.org/galaxy/galaxy-central/commits/395bfeb484ae/ changeset: 395bfeb484ae user: inithello date: 2013-01-07 18:21:46 summary: Tool shed functional test enhancements - more circular repository dependency tests. affected #: 7 files diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/base/test_db_util.py --- a/test/tool_shed/base/test_db_util.py +++ b/test/tool_shed/base/test_db_util.py @@ -43,6 +43,10 @@ galaxy.model.ToolShedRepository.table.c.owner == owner, galaxy.model.ToolShedRepository.table.c.changeset_revision == changeset_revision ) ) \ .first() +def get_installed_repository_by_id( repository_id ): + return ga_session.query( galaxy.model.ToolShedRepository ) \ + .filter( galaxy.model.ToolShedRepository.table.c.id == repository_id ) \ + .first() def get_installed_repository_by_name_owner( repository_name, owner ): return ga_session.query( galaxy.model.ToolShedRepository ) \ .filter( and_( galaxy.model.ToolShedRepository.table.c.name == repository_name, @@ -53,6 +57,10 @@ if role.name == user.email and role.description == 'Private Role for %s' % user.email: return role raise AssertionError( "Private role not found for user '%s'" % user.email ) +def get_repository_by_id( repository_id ): + return sa_session.query( model.Repository ) \ + .filter( model.Repository.table.c.id == repository_id ) \ + .first() def get_user( email ): return sa_session.query( model.User ) \ .filter( model.User.table.c.email==email ) \ diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/base/twilltestcase.py --- a/test/tool_shed/base/twilltestcase.py +++ b/test/tool_shed/base/twilltestcase.py @@ -1,14 +1,16 @@ import galaxy.webapps.community.util.hgweb_config import galaxy.model as galaxy_model -import common, string, os, re, test_db_util, simplejson +import common, string, os, re, test_db_util, simplejson, logging, time import galaxy.util as util from base.twilltestcase import tc, from_json_string, TwillTestCase, security, urllib -from galaxy.tool_shed.encoding_util import tool_shed_encode +from galaxy.tool_shed.encoding_util import tool_shed_encode, tool_shed_decode from galaxy import eggs eggs.require('mercurial') from mercurial import hg, ui +log = logging.getLogger( __name__ ) + class ShedTwillTestCase( TwillTestCase ): def setUp( self ): # Security helper @@ -429,7 +431,7 @@ if workflow_name not in strings_displayed: strings_displayed.append( workflow_name ) self.check_for_strings( strings_displayed, strings_not_displayed ) - def initiate_installation_process( self ): + def initiate_installation_process( self, install_tool_dependencies=False, install_repository_dependencies=True ): html = self.last_page() # Since the installation process is by necessity asynchronous, we have to get the parameters to 'manually' initiate the # installation process. This regex will return the tool shed repository IDs in group(1), the encoded_kwd parameter in @@ -438,11 +440,20 @@ install_parameters = re.search( 'initiate_repository_installation$ "([^"]+)", "([^"]+)", "([^"]+)" $;', html ) if install_parameters: iri_ids = install_parameters.group(1) - encoded_kwd = install_parameters.group(2) + # In some cases, the returned iri_ids are of the form: "[u'<encoded id>', u'<encoded id>']" + # This ensures that non-hex characters are stripped out of the list, so that util.listify/decode_id will handle them correctly. + repository_ids = str( iri_ids ) + repository_ids = re.sub( '[^a-fA-F0-9,]+', '', repository_ids ) + decoded_kwd = tool_shed_decode( install_parameters.group(2) ) + if 'install_tool_dependencies' in decoded_kwd: + decoded_kwd[ 'install_tool_dependencies' ] = install_tool_dependencies + if 'install_repository_dependencies' in decoded_kwd: + decoded_kwd[ 'install_repository_dependencies' ] = install_repository_dependencies reinstalling = install_parameters.group(3) url = '/admin_toolshed/manage_repositories?operation=install&tool_shed_repository_ids=%s&encoded_kwd=%s&reinstalling=%s' % \ - ( iri_ids, encoded_kwd, reinstalling ) + ( ','.join( util.listify( repository_ids ) ), tool_shed_encode( decoded_kwd ), reinstalling ) self.visit_galaxy_url( url ) + return util.listify( repository_ids ) def install_repository( self, name, owner, category_name, install_tool_dependencies=False, changeset_revision=None, strings_displayed=[], strings_not_displayed=[], preview_strings_displayed=[], post_submit_strings_displayed=[], **kwd ): @@ -469,8 +480,8 @@ kwd[ 'shed_tool_conf' ] = self.shed_tool_conf self.submit_form( 1, 'select_tool_panel_section_button', **kwd ) self.check_for_strings( post_submit_strings_displayed, strings_not_displayed ) - self.initiate_installation_process() - self.wait_for_repository_installation( repository, changeset_revision ) + repository_ids = self.initiate_installation_process( install_tool_dependencies ) + self.wait_for_repository_installation( repository_ids ) def load_invalid_tool_page( self, repository, tool_xml, changeset_revision, strings_displayed=[], strings_not_displayed=[] ): url = '/repository/load_invalid_tool?repository_id=%s&tool_config=%s&changeset_revision=%s' % \ ( self.security.encode_id( repository.id ), tool_xml, changeset_revision ) @@ -510,12 +521,17 @@ self.visit_galaxy_url( url ) strings_displayed = [ installed_repository.name, 'repository has been activated' ] self.check_for_strings( strings_displayed, [] ) - def reinstall_repository( self, installed_repository ): - url = '/admin_toolshed/reinstall_repository?id=%s' % self.security.encode_id( installed_repository.id ) + def reinstall_repository( self, + installed_repository, + install_repository_dependencies='true', + install_tool_dependencies='false' ): + url = '/admin_toolshed/reselect_tool_panel_section?id=%s' % self.security.encode_id( installed_repository.id ) self.visit_galaxy_url( url ) - self.initiate_installation_process() - tool_shed_repository = test_db_util.get_repository_by_name_and_owner( installed_repository.name, installed_repository.owner ) - self.wait_for_repository_installation( tool_shed_repository, installed_repository.installed_changeset_revision ) + url = '/admin_toolshed/reinstall_repository?id=%s&install_repository_dependencies=%s&install_repository_dependencies=%s' % \ + ( self.security.encode_id( installed_repository.id ), install_repository_dependencies, install_repository_dependencies ) + self.visit_galaxy_url( url ) + repository_ids = self.initiate_installation_process( install_tool_dependencies, install_repository_dependencies ) + self.wait_for_repository_installation( repository_ids ) def repository_is_new( self, repository ): repo = hg.repository( ui.ui(), self.get_repo_path( repository ) ) tip_ctx = repo.changectx( repo.changelog.tip() ) @@ -636,22 +652,18 @@ def visit_galaxy_url( self, url ): url = '%s%s' % ( self.galaxy_url, url ) self.visit_url( url ) - def wait_for_repository_installation( self, repository, changeset_revision ): + def wait_for_repository_installation( self, repository_ids ): final_states = [ galaxy_model.ToolShedRepository.installation_status.ERROR, - galaxy_model.ToolShedRepository.installation_status.INSTALLED, - galaxy_model.ToolShedRepository.installation_status.UNINSTALLED, - galaxy_model.ToolShedRepository.installation_status.DEACTIVATED ] - repository_name = repository.name - owner = repository.user.username - if changeset_revision is None: - changeset_revision = self.get_repository_tip( repository ) - galaxy_repository = test_db_util.get_galaxy_repository_by_name_owner_changeset_revision( repository_name, owner, changeset_revision ) - timeout_counter = 0 - while galaxy_repository.status not in final_states: - ga_refresh( galaxy_repository ) - timeout_counter = timeout_counter + 1 - if timeout_counter > common.repository_installation_timeout: - raise AssertionError( 'Repository installation timed out, %d seconds elapsed, repository state is %s.' % \ - ( timeout_counter, repository.status ) ) - break - time.sleep( 1 ) + galaxy_model.ToolShedRepository.installation_status.INSTALLED ] + if repository_ids: + for repository_id in repository_ids: + galaxy_repository = test_db_util.get_installed_repository_by_id( self.security.decode_id( repository_id ) ) + timeout_counter = 0 + while galaxy_repository.status not in final_states: + test_db_util.ga_refresh( galaxy_repository ) + timeout_counter = timeout_counter + 1 + if timeout_counter > common.repository_installation_timeout: + raise AssertionError( 'Repository installation timed out, %d seconds elapsed, repository state is %s.' % \ + ( timeout_counter, repository.status ) ) + break + time.sleep( 1 ) diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/functional/test_0040_repository_circular_dependencies.py --- a/test/tool_shed/functional/test_0040_repository_circular_dependencies.py +++ b/test/tool_shed/functional/test_0040_repository_circular_dependencies.py @@ -2,8 +2,8 @@ import tool_shed.base.test_db_util as test_db_util freebayes_repository_name = 'freebayes_0040' -freebayes_repository_name_description = "Galaxy's freebayes tool" -freebayes_repository_name_long_description = "Long description of Galaxy's freebayes tool" +freebayes_repository_description = "Galaxy's freebayes tool" +freebayes_repository_long_description = "Long description of Galaxy's freebayes tool" filtering_repository_name = 'filtering_0040' filtering_repository_description = "Galaxy's filtering tool" @@ -31,8 +31,8 @@ self.logout() self.login( email=common.test_user_1_email, username=common.test_user_1_name ) repository = self.get_or_create_repository( name=freebayes_repository_name, - description=freebayes_repository_name_description, - long_description=freebayes_repository_name_long_description, + description=freebayes_repository_description, + long_description=freebayes_repository_long_description, owner=common.test_user_1_name, categories=[ 'test_0040_repository_circular_dependencies' ], strings_displayed=[] ) diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/functional/test_0080_advanced_circular_dependencies.py --- /dev/null +++ b/test/tool_shed/functional/test_0080_advanced_circular_dependencies.py @@ -0,0 +1,89 @@ +from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os +import tool_shed.base.test_db_util as test_db_util + +column_repository_name = 'column_maker_0080' +column_repository_description = "Add column" +column_repository_long_description = "Compute an expression on every row" + +convert_repository_name = 'convert_chars_0080' +convert_repository_description = "Convert delimiters" +convert_repository_long_description = "Convert delimiters to tab" + +category_name = 'Test 0080 Advanced Circular Dependencies' +category_description = 'Test circular dependency features' + +class TestRepositoryCircularDependencies( ShedTwillTestCase ): + '''Verify that the code correctly handles circular dependencies.''' + def test_0000_initiate_users( self ): + """Create necessary user accounts.""" + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + test_user_1 = test_db_util.get_user( common.test_user_1_email ) + assert test_user_1 is not None, 'Problem retrieving user with email %s from the database' % test_user_1_email + test_user_1_private_role = test_db_util.get_private_role( test_user_1 ) + self.logout() + self.login( email=common.admin_email, username=common.admin_username ) + admin_user = test_db_util.get_user( common.admin_email ) + assert admin_user is not None, 'Problem retrieving user with email %s from the database' % admin_email + admin_user_private_role = test_db_util.get_private_role( admin_user ) + def test_0005_initiate_category_repositories( self ): + """Create a category for this test suite and add repositories to it.""" + category = self.create_category( name=category_name, description=category_description ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=column_repository_name, + description=column_repository_description, + long_description=column_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + self.upload_file( repository, + 'column_maker/column_maker.tar', + strings_displayed=[], + commit_message='Uploaded column_maker.tar.' ) + repository = self.get_or_create_repository( name=convert_repository_name, + description=convert_repository_description, + long_description=convert_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + self.upload_file( repository, + 'convert_chars/convert_chars.tar', + strings_displayed=[], + commit_message='Uploaded convert_chars.tar.' ) + def test_0020_create_repository_dependencies( self ): + '''Upload a repository_dependencies.xml file that specifies the current revision of freebayes to the filtering_0040 repository.''' + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_0080', additional_paths=[ 'convert' ] ) + self.generate_repository_dependency_xml( [ convert_repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Column maker depends on the convert repository.' ) + self.upload_file( column_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on convert' ) + def test_0025_create_dependency_on_filtering( self ): + '''Upload a repository_dependencies.xml file that specifies the current revision of filtering to the freebayes_0040 repository.''' + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_0080', additional_paths=[ 'convert' ] ) + self.generate_repository_dependency_xml( [ column_repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Convert chars depends on the column_maker repository.' ) + self.upload_file( convert_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on column' ) + def test_0030_verify_repository_dependencies( self ): + '''Verify that each repository can depend on the other without causing an infinite loop.''' + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + self.check_repository_dependency( convert_repository, column_repository, self.get_repository_tip( column_repository ) ) + self.check_repository_dependency( column_repository, convert_repository, self.get_repository_tip( convert_repository ) ) + def test_0035_verify_repository_metadata( self ): + '''Verify that resetting the metadata does not change it.''' + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + for repository in [ column_repository, convert_repository ]: + self.verify_unchanged_repository_metadata( repository ) diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/functional/test_1080_advanced_circular_dependency_installation.py --- /dev/null +++ b/test/tool_shed/functional/test_1080_advanced_circular_dependency_installation.py @@ -0,0 +1,314 @@ +from tool_shed.base.twilltestcase import ShedTwillTestCase, common, os +import tool_shed.base.test_db_util as test_db_util + +column_repository_name = 'column_maker_0080' +column_repository_description = "Add column" +column_repository_long_description = "Compute an expression on every row" + +convert_repository_name = 'convert_chars_0080' +convert_repository_description = "Convert delimiters" +convert_repository_long_description = "Convert delimiters to tab" + +category_name = 'Test 0080 Advanced Circular Dependencies' +category_description = 'Test circular dependency features' + +class TestRepositoryDependencies( ShedTwillTestCase ): + '''Testing freebayes with tool data table entries, .loc files, and tool dependencies.''' + def test_0000_create_or_login_admin_user( self ): + """Create necessary user accounts and login as an admin user.""" + self.galaxy_logout() + self.galaxy_login( email=common.admin_email, username=common.admin_username ) + galaxy_admin_user = test_db_util.get_galaxy_user( common.admin_email ) + assert galaxy_admin_user is not None, 'Problem retrieving user with email %s from the database' % common.admin_email + galaxy_admin_user_private_role = test_db_util.get_galaxy_private_role( galaxy_admin_user ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + test_user_1 = test_db_util.get_user( common.test_user_1_email ) + assert test_user_1 is not None, 'Problem retrieving user with email %s from the database' % test_user_1_email + test_user_1_private_role = test_db_util.get_private_role( test_user_1 ) + self.logout() + self.login( email=common.admin_email, username=common.admin_username ) + admin_user = test_db_util.get_user( common.admin_email ) + assert admin_user is not None, 'Problem retrieving user with email %s from the database' % admin_email + admin_user_private_role = test_db_util.get_private_role( admin_user ) + def test_0005_initiate_test_data( self ): + """Create a category for this test suite and add repositories to it.""" + category = self.create_category( name=category_name, description=category_description ) + self.logout() + self.login( email=common.test_user_1_email, username=common.test_user_1_name ) + repository = self.get_or_create_repository( name=column_repository_name, + description=column_repository_description, + long_description=column_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + if self.repository_is_new( repository ): + self.upload_file( repository, + 'column_maker/column_maker.tar', + strings_displayed=[], + commit_message='Uploaded column_maker.tar.' ) + repository = self.get_or_create_repository( name=convert_repository_name, + description=convert_repository_description, + long_description=convert_repository_long_description, + owner=common.test_user_1_name, + category_id=self.security.encode_id( category.id ), + strings_displayed=[] ) + self.upload_file( repository, + 'convert_chars/convert_chars.tar', + strings_displayed=[], + commit_message='Uploaded convert_chars.tar.' ) + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_1080', additional_paths=[ 'convert' ] ) + self.generate_repository_dependency_xml( [ convert_repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Column maker depends on the convert repository.' ) + self.upload_file( column_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on convert' ) + convert_repository = test_db_util.get_repository_by_name_and_owner( convert_repository_name, common.test_user_1_name ) + column_repository = test_db_util.get_repository_by_name_and_owner( column_repository_name, common.test_user_1_name ) + repository_dependencies_path = self.generate_temp_path( 'test_1080', additional_paths=[ 'convert' ] ) + self.generate_repository_dependency_xml( [ column_repository ], + self.get_filename( 'repository_dependencies.xml', filepath=repository_dependencies_path ), + dependency_description='Convert chars depends on the column_maker repository.' ) + self.upload_file( convert_repository, + 'repository_dependencies.xml', + filepath=repository_dependencies_path, + commit_message='Uploaded dependency on column' ) + def test_0010_install_repositories( self ): + '''Install convert_chars with repository dependencies check box - this should install both convert_chars and column_maker.''' + self.galaxy_logout() + self.galaxy_login( email=common.admin_email, username=common.admin_username ) + self.install_repository( convert_repository_name, + common.test_user_1_name, + category_name, + install_tool_dependencies=False, + install_repository_dependencies='Yes', + new_tool_panel_section='test_1080' ) + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + browse_strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision ] + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Installed' ] + self.display_galaxy_browse_repositories_page( strings_displayed=strings_displayed ) + strings_displayed.append( 'Installed repository dependencies' ) + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0015_deactivate_convert_repository( self ): + '''Deactivate convert_chars - this should display column_maker as installed but missing repository dependencies''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_convert_repository, remove_from_disk=False ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Missing repository dependencies', + 'Deactivated' ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + def test_0020_reactivate_convert_repository( self ): + '''Activate convert_chars - this should display both convert_chars and column_maker as installed with a green box''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.reactivate_repository( installed_convert_repository ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + def test_0025_deactivate_column_repository( self ): + '''Deactivate column_maker - this should display convert_chars installed but missing repository dependencies''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_column_repository, remove_from_disk=False ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Missing repository dependencies', + 'Deactivated' ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0030_deactivate_convert_repository( self ): + '''Deactivate convert_chars - both convert_chars and column_maker are deactivated''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_convert_repository, remove_from_disk=False ) + strings_not_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.installed_changeset_revision ] + self.display_galaxy_browse_repositories_page( strings_not_displayed=strings_not_displayed ) + def test_0035_reactivate_column_repository( self ): + '''Activate column_maker - this should not automatically activate convert_chars, so column_maker should be displayed as installed but missing repository dependencies''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.reactivate_repository( installed_column_repository ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Missing repository dependencies', + 'Deactivated' ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + def test_0040_reactivate_convert_repository( self ): + '''Activate convert_chars - this should display both convert_chars and column_maker as installed with a green box''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.reactivate_repository( installed_convert_repository ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0045_uninstall_column_repository( self ): + '''Uninstall column_maker - this should display convert_chars installed but missing repository dependencies''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_column_repository, remove_from_disk=True ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Missing repository dependencies', + 'Uninstalled' ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0050_reinstall_column_repository( self ): + '''Reinstall column_maker without repository dependencies, verify both convert_chars and column_maker are installed.''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.reinstall_repository( installed_column_repository, install_repository_dependencies=False ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0055_uninstall_convert_repository( self ): + '''Uninstall convert_chars, verify column_maker installed but missing repository dependencies''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_convert_repository, remove_from_disk=True ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Missing repository dependencies', + 'Uninstalled' ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + def test_0060_uninstall_column_repository( self ): + '''Uninstall column_maker - both convert_chars and column_maker are uninstalled''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.uninstall_repository( installed_column_repository, remove_from_disk=True ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Missing repository dependencies', + 'Activate or reinstall repository', + 'Uninstalled' ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) + def test_0065_reinstall_convert_repository( self ): + '''Reinstall convert_chars and check the handle repository dependencies check box - this should install both convert_chars and column_maker ( make sure )''' + installed_convert_repository = test_db_util.get_installed_repository_by_name_owner( convert_repository_name, + common.test_user_1_name ) + installed_column_repository = test_db_util.get_installed_repository_by_name_owner( column_repository_name, + common.test_user_1_name ) + self.reinstall_repository( installed_convert_repository, install_repository_dependencies=True ) + strings_displayed = [ installed_column_repository.name, + installed_column_repository.description, + installed_column_repository.tool_shed, + installed_column_repository.installed_changeset_revision, + installed_convert_repository.name, + installed_convert_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_installed_repository_manage_page( installed_column_repository, + strings_displayed=strings_displayed ) + strings_displayed = [ installed_convert_repository.name, + installed_convert_repository.description, + installed_convert_repository.tool_shed, + installed_convert_repository.installed_changeset_revision, + installed_column_repository.name, + installed_column_repository.installed_changeset_revision, + 'Installed repository dependencies' ] + self.display_installed_repository_manage_page( installed_convert_repository, + strings_displayed=strings_displayed ) diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/test_data/column_maker/column_maker.tar Binary file test/tool_shed/test_data/column_maker/column_maker.tar has changed diff -r df20658ac4991d970bfd9d63feae73850db84bb7 -r 395bfeb484ae0f69bdd7be2115ed007b4c01dd90 test/tool_shed/test_data/convert_chars/convert_chars.tar Binary file test/tool_shed/test_data/convert_chars/convert_chars.tar has changed Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0