- galaxy-dev - lists.galaxyproject.org

new_file_path questions
by Geert Vandeweyer 02 Aug '14

02 Aug '14

Hi, I'm running into issues with our galaxy installation regarding the new_file_path setting: 1/ clean up : ======== Can I safely delete all contents in the tmp folder using a cronjob, or manually when galaxy is (not) running. I have gathered almost 700Gb and >72K files on data there apparently. I suppose they are remnants of crashed jobs 2/ Should it be accessible by all nodes ======================= Since I have that much data there, I suppose it does. However, it would make sense to use a seperate (local /tmp/ ?) folder for toolshed actions such as installations. As an example, I tried to install the package_perl from the test toolshed. On our production server, hosting the new_file_path on a NFS share with the 700GB / 72K files, this process ran for over 24 hours, then crashed the main process with the error below. On my development box (laptop, no network drives, single galaxy process), the package installed in under five minutes. Best, Geert ========== tool_shed.galaxy_install.install_manager ERROR 2014-08-01 05:10:20,656 Error installing tool dependency perl version 5.18.1. Traceback (most recent call last): File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 110, in install_and_build_package_via_fabric tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 94, in install_and_build_package initial_download=False ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py", line 32, in execute_step initial_download=initial_download ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py", line 510, in execute_step return_output=False ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/install_environment.py", line 148, in handle_command context.flush() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do return getattr(self.registry(), name)(*args, **kwargs) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush self._flush(objects) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1807, in _flush transaction.rollback(_capture_exception=True) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 386, in rollback transaction._rollback_impl() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 414, in _rollback_impl t[1].rollback() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2034, in rollback self._do_rollback() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2071, in _do_rollback self.connection._rollback_impl() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1265, in _rollback_impl self._handle_dbapi_exception(e, None, None, None, None) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1262, in _rollback_impl self.engine.dialect.do_rollback(self.connection) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/dialects/mysql/base.py", line 1910, in do_rollback connection.rollback() OperationalError: (OperationalError) (2006, 'MySQL server has gone away') None None tool_shed.util.common_install_util ERROR 2014-08-01 05:10:20,743 Error installing tool dependency perl version 5.18.1: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) 'UPDATE tool_dependency SET update_time=%s, status=%s, error_message=%s WHERE tool_dependency.id = %s' [{'status': 'Error', 'error_message': ' File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 110, in install_and_build_package_via_fabric\n tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 94, in install_and_build_package\n initial_download=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py", line 32, in execute_step\n initial_download=initial_download )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py", line 510, in execute_step\n return_output=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/install_environment.py", line 148, in handle_command\n context.flush()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do\n return getattr(self.registry(), name)(*args, **kwargs)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush\n self._flush(objects)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1807, in _flush\n transaction.rollback(_capture_exception=True)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 386, in rollback\n transaction._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 414, in _rollback_impl\n t[1].rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2034, in rollback\n self._do_rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2071, in _do_rollback\n self.connection._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1265, in _rollback_impl\n self._handle_dbapi_exception(e, None, None, None, None)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1262, in _rollback_impl\n self.engine.dialect.do_rollback(self.connection)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/dialects/mysql/base.py", line 1910, in do_rollback\n connection.rollback()\n\n(OperationalError) (2006, \'MySQL server has gone away\') None None', u'tool_dependency_id': 326L}] Traceback (most recent call last): File "/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py", line 554, in install_specified_tool_dependencies from_tool_migration_manager=from_tool_migration_manager ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 313, in install_package action_elem=None ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 176, in install_via_fabric tool_dependency = self.install_and_build_package_via_fabric( app, tool_shed_repository, tool_dependency, actions_dict ) File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 119, in install_and_build_package_via_fabric remove_installation_path=False ) File "/galaxy/galaxy-dist/lib/tool_shed/util/tool_dependency_util.py", line 423, in handle_tool_dependency_installation_error sa_session.flush() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do return getattr(self.registry(), name)(*args, **kwargs) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush self._flush(objects) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1789, in _flush flush_context.execute() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute rec.execute(self) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute uow File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj mapper, table, update) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements execute(statement, params) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1651, in _execute_context None, None) File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1645, in _execute_context conn = self._revalidate_connection() File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1044, in _revalidate_connection "Can't reconnect until invalid " StatementError: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) 'UPDATE tool_dependency SET update_time=%s, status=%s, error_message=%s WHERE tool_dependency.id = %s' [{'status': 'Error', 'error_message': ' File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 110, in install_and_build_package_via_fabric\n tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 94, in install_and_build_package\n initial_download=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py", line 32, in execute_step\n initial_download=initial_download )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py", line 510, in execute_step\n return_output=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/install_environment.py", line 148, in handle_command\n context.flush()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do\n return getattr(self.registry(), name)(*args, **kwargs)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush\n self._flush(objects)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1807, in _flush\n transaction.rollback(_capture_exception=True)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 386, in rollback\n transaction._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 414, in _rollback_impl\n t[1].rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2034, in rollback\n self._do_rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2071, in _do_rollback\n self.connection._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1265, in _rollback_impl\n self._handle_dbapi_exception(e, None, None, None, None)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1262, in _rollback_impl\n self.engine.dialect.do_rollback(self.connection)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/dialects/mysql/base.py", line 1910, in do_rollback\n connection.rollback()\n\n(OperationalError) (2006, \'MySQL server has gone away\') None None', u'tool_dependency_id': 326L}] 143.169.236.220 - - [31/Jul/2014:14:54:50 +0200] "POST /galaxy/admin_toolshed/manage_repositories HTTP/1.1" 500 - "http://biominavm-galaxy.biomina.be/galaxy/admin_toolshed/prepare_for_install" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36" Error - <class 'sqlalchemy.exc.InvalidRequestError'>: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) 'UPDATE tool_dependency SET update_time=%s, status=%s, error_message=%s WHERE tool_dependency.id = %s' [{'status': 'Error', 'error_message': ' File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 110, in install_and_build_package_via_fabric\n tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 94, in install_and_build_package\n initial_download=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py", line 32, in execute_step\n initial_download=initial_download )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py", line 510, in execute_step\n return_output=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/install_environment.py", line 148, in handle_command\n context.flush()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do\n return getattr(self.registry(), name)(*args, **kwargs)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush\n self._flush(objects)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1807, in _flush\n transaction.rollback(_capture_exception=True)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 386, in rollback\n transaction._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 414, in _rollback_impl\n t[1].rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2034, in rollback\n self._do_rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2071, in _do_rollback\n self.connection._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1265, in _rollback_impl\n self._handle_dbapi_exception(e, None, None, None, None)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1262, in _rollback_impl\n self.engine.dialect.do_rollback(self.connection)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/dialects/mysql/base.py", line 1910, in do_rollback\n connection.rollback()\n\n(OperationalError) (2006, \'MySQL server has gone away\') None None', u'tool_dependency_id': 326L}] URL: http://biominavm-galaxy.biomina.be/galaxy/admin_toolshed/manage_repositories File '/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 377 in decorator return func( self, trans, *args, **kwargs ) File '/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 724 in manage_repositories **decoded_kwd ) File '/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 377 in decorator return func( self, trans, *args, **kwargs ) File '/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 645 in install_tool_shed_repositories reinstalling=reinstalling ) File '/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/repository_util.py', line 629 in install_tool_shed_repository from_tool_migration_manager=False ) File '/galaxy/galaxy-dist/lib/tool_shed/util/common_install_util.py', line 566 in install_specified_tool_dependencies remove_installation_path=False ) File '/galaxy/galaxy-dist/lib/tool_shed/util/tool_dependency_util.py', line 400 in handle_tool_dependency_installation_error tool_shed_repository = tool_dependency.tool_shed_repository File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 168 in __get__ File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 453 in get File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 490 in _load_for_state File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 531 in _get_ident_for_use_get File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py', line 1646 in _get_state_attr_by_column File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 451 in get File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/state.py', line 285 in __call__ File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/mapper.py', line 1714 in _load_scalar_attributes File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2512 in _load_on_ident File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2184 in one File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2227 in __iter__ File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2240 in _execute_and_instances File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2231 in _connection_from_session File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py', line 777 in connection close_with_result=close_with_result) File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py', line 781 in _connection_for_bind return self.transaction._connection_for_bind(engine) File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py', line 289 in _connection_for_bind self._assert_is_active() File '/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py', line 213 in _assert_is_active % self._rollback_exception InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) 'UPDATE tool_dependency SET update_time=%s, status=%s, error_message=%s WHERE tool_dependency.id = %s' [{'status': 'Error', 'error_message': ' File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 110, in install_and_build_package_via_fabric\n tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py", line 94, in install_and_build_package\n initial_download=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py", line 32, in execute_step\n initial_download=initial_download )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py", line 510, in execute_step\n return_output=False )\n File "/galaxy/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/install_environment.py", line 148, in handle_command\n context.flush()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do\n return getattr(self.registry(), name)(*args, **kwargs)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush\n self._flush(objects)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1807, in _flush\n transaction.rollback(_capture_exception=True)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 386, in rollback\n transaction._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 414, in _rollback_impl\n t[1].rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2034, in rollback\n self._do_rollback()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2071, in _do_rollback\n self.connection._rollback_impl()\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1265, in _rollback_impl\n self._handle_dbapi_exception(e, None, None, None, None)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1262, in _rollback_impl\n self.engine.dialect.do_rollback(self.connection)\n File "/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/dialects/mysql/base.py", line 1910, in do_rollback\n connection.rollback()\n\n(OperationalError) (2006, \'MySQL server has gone away\') None None', u'tool_dependency_id': 326L}] CGI Variables ------------- CONTENT_LENGTH: '-1' CONTENT_TYPE: 'application/x-www-form-urlencoded; charset=UTF-8' HTTP_ACCEPT: 'text/html, */*; q=0.01' HTTP_ACCEPT_ENCODING: 'gzip,deflate,sdch' HTTP_ACCEPT_LANGUAGE: 'nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4' HTTP_CACHE_CONTROL: 'max-age=0' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: '__utma=94419599.1127508766.1403299856.1403299856.1405949987.2; __utmz=94419599.1403299856.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); galaxysession=6730c1c7f605dc205ea1a59142b8e9026339899a15206ea6665be6a34bb4fa5a635956369e96da7d' HTTP_HOST: 'biominavm-galaxy.biomina.be' HTTP_ORIGIN: 'http://biominavm-galaxy.biomina.be' HTTP_REFERER: 'http://biominavm-galaxy.biomina.be/galaxy/admin_toolshed/prepare_for_install' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36' HTTP_VIA: '1.1 biominavm-galaxy.biomina.be' HTTP_X_REQUESTED_WITH: 'XMLHttpRequest' PATH_INFO: '/admin_toolshed/manage_repositories' REMOTE_ADDR: '143.169.236.220' REQUEST_METHOD: 'POST' SCRIPT_NAME: '/galaxy' SERVER_NAME: 'biominavm-galaxy.biomina.be' SERVER_PORT: '8080' SERVER_PROTOCOL: 'HTTP/1.1' WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0xad8c1d0> is_api_request: False paste.cookies: (<SimpleCookie: __utma='94419599.1127508766.1403299856.1403299856.1405949987.2' __utmz='94419599.1403299856.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)' galaxysession='6730c1c7f605dc205ea1a59142b8e9026339899a15206ea6665be6a34bb4fa5a635956369e96da7d'>, '__utma=94419599.1127508766.1403299856.1403299856.1405949987.2; __utmz=94419599.1403299856.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); galaxysession=6730c1c7f605dc205ea1a59142b8e9026339899a15206ea6665be6a34bb4fa5a635956369e96da7d') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0xa8df9d0> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x2ca6450> paste.recursive.forward: <paste.recursive.Forwarder from /galaxy> paste.recursive.include: <paste.recursive.Includer from /galaxy> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy> paste.recursive.script_name: '/galaxy' paste.throw_errors: True request_id: 'dc296d0a18b111e484a100163602f2ed' webob._parsed_post_vars: (MultiDict([('operation', 'install'), ('tool_shed_repository_ids', "['dc4272dc4bc65f47']"), ('encoded_kwd', '8466491acfb960dfb39d0377c52cdd7698cac2ef:7b22737461747573223a2022646f6e65222c20226861735f7265706f7369746f72795f646570656e64656e63696573223a2066616c73652c2022696e636c756465735f746f6f6c735f666f725f646973706c61795f696e5f746f6f6c5f70616e656c223a2066616c73652c2022746f6f6c5f736865645f7265706f7369746f72795f696473223a205b2264633432373264633462633635663437225d2c2022736865645f746f6f6c5f636f6e66223a20222e2f736865645f746f6f6c5f636f6e662e786d6c222c2022696e7374616c6c5f7265706f7369746f72795f646570656e64656e63696573223a2066616c73652c2022746f6f6c5f70617468223a20222e2e2f736865645f746f6f6c73222c20227265706f5f696e666f5f6469637473223a205b7b227061636b6167655f7065726c5f355f3138223a205b22436f6e7461696e73206120746f6f6c20646570656e64656e637920646566696e6974696f6e207468617420646f776e6c6f61647320616e6420636f6d70696c65732076657273696f6e20352e31382e78206f662074686520746865207065726c206c61...t([('op...e')])>) webob._parsed_query_vars: (MultiDict([]), '') wsgi process: 'Multithreaded' ------------------------------------------------------------ -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandeweyer(a)ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer

2 1

TestToolShed failure, Exception: History in error state.
by Peter Cock 02 Aug '14

02 Aug '14

Hi all, I'm not sure when this started (having hardly looked at my Tool Shed test results since GCC2014), but I think this is a fairly recent problem with my BLAST RBH tests failing (which has held me back from posting this to the main Tool Shed). This could be some silly mistake in my tar-ball, but usually missing test files and the like get an explicit error. The tests are passing on my GitHub/TravisCI setup (using Twill and the API backend): e.g. https://travis-ci.org/peterjc/galaxy_blast/builds/30592097 Here is the current error (the same for the last few Test Tool Shed runs), https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast_rbh Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 108, in test_tool self.do_it( td ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 35, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 69, in _verify_outputs galaxy_interactor.verify_output( history, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 53, in verify_output self.wait_for_history( history_id, maxseconds ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 107, in wait_for_history self.twill_test_case.wait_for( lambda: not self.__history_ready( history_id ), maxseconds=maxseconds) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 2453, in wait_for result = func() File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 107, in <lambda> self.twill_test_case.wait_for( lambda: not self.__history_ready( history_id ), maxseconds=maxseconds) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 257, in __history_ready return self._state_ready( state, error_msg="History in error state." ) File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py", line 316, in _state_ready raise Exception( error_msg ) Exception: History in error state. Is a more detailed log available which might help debug this? Thanks, Peter As an aside, this looks like the Test Tool Shed is still using the Twill backend for the functional tests?

4 5

bx version conflict causes failure for some tools (e.g. subtract interval tool)
by Langhorst, Brad 01 Aug '14

01 Aug '14

see: https://biostar.usegalaxy.org/p/8095/ I can work around the problem by adding a stdio block to these tools, how should the underlying problem be solved? Brad -- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist

1 0

Error while installing tools from main toolshed: "HTTPerror : HTTP error 307 : temporary redirect"
by Perley, Danielle 01 Aug '14

01 Aug '14

Hi, I'm trying to install tools from the main toolshed, but I keep getting the same error regardless of the tool that I'm installing HTTPError: HTTP Error 307: Temporary Redirect URL: http://localhost:8080/admin_toolshed/prepare_for_install?tool_shed_url=http… Module weberror.evalexception.middleware:364 in respond view >> app_iter = self.application(environ, detect_start_response) Module paste.recursive:84 in __call__ view >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ view >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ view >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request view >> body = method( trans, **kwargs ) Module galaxy.web.framework:377 in decorator view >> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1116 in prepare_for_install view >> updating=updating ) Module tool_shed.util.common_install_util:110 in get_dependencies_for_repository view >> all_repo_info_dict = get_required_repo_info_dicts( trans, tool_shed_url, util.listify( repo_info_dict ) ) Module tool_shed.util.common_install_util:451 in get_required_repo_info_dicts view >> response = urllib2.urlopen( request ).read() Module urllib2:127 in urlopen view >> return _opener.open(url, data, timeout) Module urllib2:410 in open view >> response = meth(req, response) Module urllib2:523 in http_response view >> 'http', request, response, code, msg, hdrs) Module urllib2:442 in error view >> result = self._call_chain(*args) Module urllib2:382 in _call_chain view >> result = func(*args) Module urllib2:608 in http_error_302 view >> new = self.redirect_request(req, fp, code, msg, headers, newurl) Module urllib2:569 in redirect_request view >> raise HTTPError(req.get_full_url(), code, msg, headers, fp) HTTPError: HTTP Error 307: Temporary Redirect I've just installed Galaxy locally for the first time on a Mac OS 10.9.3. The only changes I've made to the universe_wsg.ini file are: setting host=0.0.0.0 (The error also occurred with host=127.0.0.1?) specifying a full path for Tool_depenendency_dir giving myself administrator privileges with admin_users=myemail any ideas? Danielle

1 0

Toolshed tool_dependency.xml git archive configure issue
by Dooley, Damion 01 Aug '14

01 Aug '14

Hi folks, Having a difficult time with coding the "configure" step in this new tool_dependency.xml file, namely that an error "/bin/sh: configure: command not found" results when I try to install my beta development tool from a local test toolshed (which is downloading and installing other tools just fine). <actions> <action type="shell_command">git clone https://github.com/myuser/mygitstuff.git .</action> <action type="shell_command">git reset --hard 1819482db93bf0025c2588b5264cc0ec8c61029c</action> <action type="shell_command">configure --disable-gui --prefix=$INSTALL_DIR</action> <action type="make_install"></action> <action type="move_directory_files"> <source_directory>bin</source_directory> <destination_directory>$INSTALL_DIR/bin</destination_directory> </action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> Any hints about why this isn't being found or what the /bin/sh needs to get it going? When I do a git reset --hard, the "configure" file is definitely there, with executable flag set. Any other permission oddities? I've also tried <action type="autoconf"></action> , which yeilds same error. (Skipping that line and doing <action type="make_install"></action> at least finds and runs "make", though it errors out for lack of input data). Regards, Damion Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

2 5

galaxy install nginx
by ngsflow＠hygenomics.com 01 Aug '14

01 Aug '14

dear sir: nginx postgress galaxy which one first install ? ngsflow(a)hygenomics.com

2 1

Announcement: Galaxy IPython Integration - interactive programming in Galaxy
by Björn Grüning 01 Aug '14

01 Aug '14

Hi all, we proudly present the first release of the Galaxy IPython project. Galaxy IPython is a visualization plugin which should enable Galaxy users with coding skills to easily process their data in the most flexible way. With this plugin, it is possible to analyse and post-process data without downloading datasets or entire histories. One of our aims was to make Galaxy more attractive and accessible to bioinformaticians and programmers, and we hope that this project will build some bridges. Screencast: http://www.youtube.com/watch?v=jQDyTuYnn1k Disclaimer: Even though the Ipython notebooks can be stored and reused, this plugin will break the Galaxy philosophy of reproducibility, I feel personally bad about that, but I also think it is a great opportunity to get more bioinformaticians into Galaxy, and to get Galaxy used more often as a teaching resource. By being able to teach not only about workflows but also about data analysis tasks often necessary with Bioinformatics, Galaxy will be significantly more useful in teaching environments. Keep in mind to write a nice Tool Shed Tool if you catch yourself using IPython in Galaxy to often for the same task. A few features we have up and running: Use IPython directly in the main window or in the Scratchbook Completely encapsulated IPython environment with matplotlib, biopython, pandas and friends already installed. IPython runs completely self-contained within a docker container, separate from your Galaxy data Easy access to datasets from your current history via pre-defined IPython functions Manipulate and plot data as you like and export your new files back into your Galaxy history Save IPython Notebooks across analysis sessions in your Galaxy history with the click of a button. View saved IPython Notebooks directly in HTML format, or re-open them to continue your analysis. Self-closing and self-cleaning IPython docker container Notebooks are secure, only accessible to the intended user Please follow the installation instruction on our project page under: https://github.com/bgruening/galaxy-ipython The underlying IPython Notebook (+Galaxy sugar) is stored under: https://github.com/bgruening/docker-ipython-notebook https://registry.hub.docker.com/u/bgruening/docker-ipython-notebook/ You can also install a ipynb datatype: https://github.com/bgruening/galaxytools/tree/master/datatypes/json https://testtoolshed.g2.bx.psu.edu/view/iuc/datatyp_ipynb Comments welcome! Happy research! Eric & Björn

1 0

Galaxy Security Vulnerability
by Nate Coraor 31 Jul '14

31 Jul '14

A security vulnerability was recently discovered by Inge Alexander Raknes that would allow a malicious person to execute arbitrary code on a Galaxy server. The vulnerability was in a method that uses Python "pickle" functionality to decode state information from tool forms. Because pickles can be used to instantiate arbitrary Python objects, tool states could be constructed to exploit this vulnerability. Because this vulnerability allows for arbitrary code execution, administrators are strongly encouraged to apply this fix IMMEDIATELY. We have tried to make it as easy and quick as possible for server administrators to update their Galaxy instances. The fix has been applied to every stable release from 2013.01.13 until the tip, so it is possible to get this fix on older releases without updating to a newer feature release. You can do this by identifying your current release and updating to the `latest_<release_date>` tag corresponding to your release. For example, if you are running release_2013.11.04 (or a subsequent commit to the stable branch of Galaxy between release_2013.11.04 and release_2014.02.10), you can update with: % hg pull % hg update latest_2013.11.04 For the changes to take effect, YOU MUST RESTART ALL GALAXY SERVER PROCESSES. If you do not want to pull any upstream changes, we have also created a standalone patch that fixes this problem, with multiple versions depending on your current Galaxy release: - pickle-2013.11.04.patch - This patch should apply cleanly (with offset/fuzz) to releases from 2013.11.04 up to the current stable tip. Available at: https://depot.galaxyproject.org/patch/pickle-2013.11.04.patch - pickle-2013.01.13.patch - This patch should apply cleanly (with offset/fuzz) to releases from 2013.01.13 up to 2013.08.12, and possibly older versions of Galaxy as well. Available at: https://depot.galaxyproject.org/patch/pickle-2013.01.13.patch If you happen to be running a very recent revision on the default branch or the newly created next-stable branch, a pickle-default.patch file exists at the same place. For older releases or instances with conflicting local modifications, manual application of the patch should not be difficult as it only includes a few small changes. To apply the patch, navigate to the root of your Galaxy directory, then run (replacing <url_to_patch> with the URL above that is correct for your release): % wget -O pickle.patch <url_to_patch> or: % curl -o pickle.patch <url_to_patch> and then: % patch -p1 < pickle.patch patching file lib/galaxy/util/__init__.py Hunk #1 succeeded at 575 with fuzz 2 (offset -113 lines). patching file lib/galaxy/webapps/galaxy/controllers/ucsc_proxy.py Again, for the changes to take effect, YOU MUST RESTART ALL GALAXY SERVER PROCESSES. The Galaxy Team would like to extend special thanks to Inge Alexander Raknes and colleagues, who privately disclosed the vulnerability, with a full analysis and proof of concept. Credit for the fix and subsequent testing goes to my fellow Galaxy Team members John Chilton and Dannon Baker. On behalf of the Galaxy Team, --nate

1 0

package_r_2_11_0 and ggplot2
by Evan Bollig 31 Jul '14

31 Jul '14

I want to install ggplot2 into package_r. The library ggplot2 is no longer available from the CRAN mirror for R v2.11.0. How do you guys handle this situation? Can I install another package_r and have packages that depend on package_r_2_11_0 use the newer version? -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 evan(a)msi.umn.edu boll0107(a)umn.edu

2 6

fastq-mcf parameter bug
by Michael Mason 31 Jul '14

31 Jul '14

First time posting so I hope I do this correctly. I am using galaxy locally and installed the ea-utils fastq-mcf (FastqMcf). When using the trimming options (not defaults) the -C option gets set to the -s option. I checked the fastq-mcf.xml wrapper and indeed there is an mistake on line 42. -s $trimming.sampcnt Should be -C $trimming.sampcnt I hope this helps, Cheers, Mike ________________________________ --CONFIDENTIALITY NOTICE--: The information contained in this email is intended for the exclusive use of the addressee and may contain confidential information. If you are not the intended recipient, you are hereby notified that any form of dissemination of this communication is strictly prohibited. www.benaroyaresearch.org

1 0