- galaxy-commits - lists.galaxyproject.org

commit/galaxy-central: carlfeberhard: small fix to twilltestcase when GALAXY_TEST_SAVE fails
by Bitbucket 16 Aug '12

16 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/01ed2f462dd7/ changeset: 01ed2f462dd7 user: carlfeberhard date: 2012-08-15 17:13:13 summary: small fix to twilltestcase when GALAXY_TEST_SAVE fails affected #: 1 file diff -r 7566c91c1ec65f90c52f173ff1a5442789028f93 -r 01ed2f462dd7709876458b031d786d277d1f72f3 test/base/twilltestcase.py --- a/test/base/twilltestcase.py +++ b/test/base/twilltestcase.py @@ -694,11 +694,19 @@ if filename is not None: local_name = self.get_filename( filename, shed_tool_id=shed_tool_id ) temp_name = self.makeTfname(fname = filename) - file( temp_name, 'wb' ).write(data) - if self.keepOutdir > '': - ofn = os.path.join(self.keepOutdir,os.path.basename(local_name)) - shutil.copy(temp_name,ofn) - log.debug('## GALAXY_TEST_SAVE=%s. saved %s' % (self.keepOutdir,ofn)) + file( temp_name, 'wb' ).write( data ) + + # if the server's env has GALAXY_TEST_SAVE, save the output file to that dir + if self.keepOutdir: + ofn = os.path.join( self.keepOutdir, os.path.basename( local_name ) ) + try: + shutil.copy( temp_name, ofn ) + except Exception, exc: + error_log_msg = ( 'TwillTestCase could not save output file %s to %s: ' % ( temp_name, ofn ) ) + error_log_msg += str( e ) + else: + log.debug('## GALAXY_TEST_SAVE=%s. saved %s' % ( self.keepOutdir, ofn ) ) + try: # have to nest try-except in try-finally to handle 2.4 try: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 3 new changesets
by Bitbucket 16 Aug '12

16 Aug '12

3 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/16726c4310a7/ changeset: 16726c4310a7 user: jmchilton date: 2012-05-04 07:12:43 summary: Augment run.sh to search for the optional environment variable GALAXY_UNIVERSE_CONFIG_DIR. If this is set, before Galaxy is launched the directory specified by the environment variable is searched for ini config files. These files are merged into a single configuration structure and written back out as universe.ini. This should allow for a more modular configuration of Galaxy. The files are merged in lexiographic order and later files can override properties specified ealier on. If a consistent naming scheme is used this can be really useful: As an example, one might specify the files as such: 000_galaxy_defaults.ini 000_default_sqlite_config.ini 100_institution_specific.ini 500_job_runner.ini 500_webapp_0.ini 500_webapp_1.ini 500_instance_specific.ini 500_available_postgres.ini affected #: 2 files diff -r 1eb32dbab2c1a0bc308eb4d3a069fa55ff6ecb83 -r 16726c4310a71be907ebdcbf850dcd7ce72797b4 run.sh --- a/run.sh +++ b/run.sh @@ -33,6 +33,10 @@ fi done +if [ -n "$GALAXY_UNIVERSE_CONFIG_DIR" ]; then + python ./scripts/build_universe_config.py "$GALAXY_UNIVERSE_CONFIG_DIR" +fi + # explicitly attempt to fetch eggs before running FETCH_EGGS=1 for arg in "$@"; do diff -r 1eb32dbab2c1a0bc308eb4d3a069fa55ff6ecb83 -r 16726c4310a71be907ebdcbf850dcd7ce72797b4 scripts/build_universe_config.py --- /dev/null +++ b/scripts/build_universe_config.py @@ -0,0 +1,26 @@ +from ConfigParser import ConfigParser +from os import listdir +from os.path import join +from re import match +from sys import argv + + +def merge(): + "Merges all .ini files in a specified directory into ./universe.ini" + if len(argv) < 2: + message = "%s: Must specify directory to merge configuration files from." % argv[0] + raise Exception(message) + conf_directory = argv[1] + conf_files = [f for f in listdir(conf_directory) if match(r'.*\.ini', f)] + conf_files.sort() + + parser = ConfigParser() + for conf_file in conf_files: + parser.read([join(conf_directory, conf_file)]) + ## TODO: Expand enviroment variables here, that would + ## also make Galaxy much easier to configure. + + parser.write(open("universe.ini", 'w')) + +if __name__ == '__main__': + merge() https://bitbucket.org/galaxy/galaxy-central/changeset/6f4b97fd803e/ changeset: 6f4b97fd803e user: jmchilton date: 2012-05-24 14:47:14 summary: Fix a typo. affected #: 1 file diff -r 16726c4310a71be907ebdcbf850dcd7ce72797b4 -r 6f4b97fd803e6ab97192d8454bc71bff145e5695 scripts/build_universe_config.py --- a/scripts/build_universe_config.py +++ b/scripts/build_universe_config.py @@ -20,7 +20,7 @@ ## TODO: Expand enviroment variables here, that would ## also make Galaxy much easier to configure. - parser.write(open("universe.ini", 'w')) + parser.write(open("universe_wsgi.ini", 'w')) if __name__ == '__main__': merge() https://bitbucket.org/galaxy/galaxy-central/changeset/7566c91c1ec6/ changeset: 7566c91c1ec6 user: natefoo date: 2012-08-15 16:57:14 summary: Merged in jmchilton/galaxy-central-configuration-directory (pull request #44) affected #: 2 files diff -r b2fd558879b99de125f4554ba5e15b726bd8431c -r 7566c91c1ec65f90c52f173ff1a5442789028f93 run.sh --- a/run.sh +++ b/run.sh @@ -36,6 +36,10 @@ fi done +if [ -n "$GALAXY_UNIVERSE_CONFIG_DIR" ]; then + python ./scripts/build_universe_config.py "$GALAXY_UNIVERSE_CONFIG_DIR" +fi + # explicitly attempt to fetch eggs before running FETCH_EGGS=1 for arg in "$@"; do diff -r b2fd558879b99de125f4554ba5e15b726bd8431c -r 7566c91c1ec65f90c52f173ff1a5442789028f93 scripts/build_universe_config.py --- /dev/null +++ b/scripts/build_universe_config.py @@ -0,0 +1,26 @@ +from ConfigParser import ConfigParser +from os import listdir +from os.path import join +from re import match +from sys import argv + + +def merge(): + "Merges all .ini files in a specified directory into ./universe.ini" + if len(argv) < 2: + message = "%s: Must specify directory to merge configuration files from." % argv[0] + raise Exception(message) + conf_directory = argv[1] + conf_files = [f for f in listdir(conf_directory) if match(r'.*\.ini', f)] + conf_files.sort() + + parser = ConfigParser() + for conf_file in conf_files: + parser.read([join(conf_directory, conf_file)]) + ## TODO: Expand enviroment variables here, that would + ## also make Galaxy much easier to configure. + + parser.write(open("universe_wsgi.ini", 'w')) + +if __name__ == '__main__': + merge() Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: 3 new changesets
by Bitbucket 16 Aug '12

16 Aug '12

3 new commits in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/70b29b2bf906/ changeset: 70b29b2bf906 user: jmchilton date: 2012-03-05 19:11:53 summary: Adding db_shell.py that can be used to simplify writing command line scripts against the Galaxy model/database layer or to make use of Galaxy models in an interactive shell. affected #: 2 files diff -r e58a87c91bc4bb471b9b6477ef638dc41c4aa4b9 -r 70b29b2bf906dce0dbf8775a7a18e3d6b5403ea4 scripts/db_shell.py --- /dev/null +++ b/scripts/db_shell.py @@ -0,0 +1,86 @@ +# This script allows easy access to Galaxy's database layer via the +# Galaxy models. For example:q +# % python -i scripts/db_shel.py +# >>> new_user = User("admin(a)gmail.com") +# >>> new_user.set_password +# >>> sa_session.add(new_user) +# >>> sa_session.commit() +# >>> sa_session.query(User).all() +# +# You can also use this script as a library, for instance see https://gist.github.com/1979583q +# TODO: This script overlaps alot wth manage_db.py and create_db.py, +# these should maybe be refactored to remove duplication. +import sys, os.path, logging + +new_path = [ os.path.join( os.getcwd(), "lib" ) ] +new_path.extend( sys.path[1:] ) # remove scripts/ from the path +sys.path = new_path + +from galaxy import eggs + +import pkg_resources +pkg_resources.require( "sqlalchemy-migrate" ) +pkg_resources.require( "SQLAlchemy" ) + +from ConfigParser import SafeConfigParser + +log = logging.getLogger( __name__ ) + +if sys.argv[-1] in [ 'community' ]: + # Need to pop the last arg so the command line args will be correct + # for sqlalchemy-migrate + webapp = sys.argv.pop() + config_file = 'community_wsgi.ini' + repo = 'lib/galaxy/webapps/community/model/migrate' +else: + # Poor man's optparse + config_file = 'universe_wsgi.ini' + if '-c' in sys.argv: + pos = sys.argv.index( '-c' ) + sys.argv.pop(pos) + config_file = sys.argv.pop( pos ) + if not os.path.exists( config_file ): + print "Galaxy config file does not exist (hint: use '-c config.ini' for non-standard locations): %s" % config_file + sys.exit( 1 ) + repo = 'lib/galaxy/model/migrate' + +cp = SafeConfigParser() +cp.read( config_file ) + +if cp.has_option( "app:main", "database_connection" ): + db_url = cp.get( "app:main", "database_connection" ) +elif cp.has_option( "app:main", "database_file" ): + db_url = "sqlite:///%s?isolation_level=IMMEDIATE" % cp.get( "app:main", "database_file" ) +else: + db_url = "sqlite:///./database/universe.sqlite?isolation_level=IMMEDIATE" + +dialect_to_egg = { + "sqlite" : "pysqlite>=2", + "postgres" : "psycopg2", + "mysql" : "MySQL_python" +} +dialect = ( db_url.split( ':', 1 ) )[0] +try: + egg = dialect_to_egg[dialect] + try: + pkg_resources.require( egg ) + log.debug( "%s egg successfully loaded for %s dialect" % ( egg, dialect ) ) + except: + # If the module is in the path elsewhere (i.e. non-egg), it'll still load. + log.warning( "%s egg not found, but an attempt will be made to use %s anyway" % ( egg, dialect ) ) +except KeyError: + # Let this go, it could possibly work with db's we don't support + log.error( "database_connection contains an unknown SQLAlchemy database dialect: %s" % dialect ) + +# Setup DB scripting environment +from sqlalchemy import * +from sqlalchemy.orm import * +from sqlalchemy.exc import * + +os.environ['PYTHONINSPECT'] = 'True' +engine = create_engine(db_url, echo=True) +db_session = scoped_session( sessionmaker( bind = engine ) ) +from galaxy.model.mapping import context as sa_session +sa_session.bind = engine +from galaxy.model import * + https://bitbucket.org/galaxy/galaxy-central/changeset/2eebd7dca84b/ changeset: 2eebd7dca84b user: jmchilton date: 2012-03-05 22:23:28 summary: Actually we don't want to set that environment variable, that make python behave like -i every execution. affected #: 1 file diff -r 70b29b2bf906dce0dbf8775a7a18e3d6b5403ea4 -r 2eebd7dca84be8b34b99b9bbb16ee8f75352986c scripts/db_shell.py --- a/scripts/db_shell.py +++ b/scripts/db_shell.py @@ -77,7 +77,6 @@ from sqlalchemy.orm import * from sqlalchemy.exc import * -os.environ['PYTHONINSPECT'] = 'True' engine = create_engine(db_url, echo=True) db_session = scoped_session( sessionmaker( bind = engine ) ) from galaxy.model.mapping import context as sa_session https://bitbucket.org/galaxy/galaxy-central/changeset/b2fd558879b9/ changeset: b2fd558879b9 user: natefoo date: 2012-08-15 16:54:50 summary: Merged in jmchilton/galaxy-central-preseeding (pull request #38) affected #: 2 files diff -r a6436bc768699a48f2d434c376ffeaf767ca71be -r b2fd558879b99de125f4554ba5e15b726bd8431c scripts/db_shell.py --- /dev/null +++ b/scripts/db_shell.py @@ -0,0 +1,85 @@ +# This script allows easy access to Galaxy's database layer via the +# Galaxy models. For example:q +# % python -i scripts/db_shel.py +# >>> new_user = User("admin(a)gmail.com") +# >>> new_user.set_password +# >>> sa_session.add(new_user) +# >>> sa_session.commit() +# >>> sa_session.query(User).all() +# +# You can also use this script as a library, for instance see https://gist.github.com/1979583q +# TODO: This script overlaps alot wth manage_db.py and create_db.py, +# these should maybe be refactored to remove duplication. +import sys, os.path, logging + +new_path = [ os.path.join( os.getcwd(), "lib" ) ] +new_path.extend( sys.path[1:] ) # remove scripts/ from the path +sys.path = new_path + +from galaxy import eggs + +import pkg_resources +pkg_resources.require( "sqlalchemy-migrate" ) +pkg_resources.require( "SQLAlchemy" ) + +from ConfigParser import SafeConfigParser + +log = logging.getLogger( __name__ ) + +if sys.argv[-1] in [ 'community' ]: + # Need to pop the last arg so the command line args will be correct + # for sqlalchemy-migrate + webapp = sys.argv.pop() + config_file = 'community_wsgi.ini' + repo = 'lib/galaxy/webapps/community/model/migrate' +else: + # Poor man's optparse + config_file = 'universe_wsgi.ini' + if '-c' in sys.argv: + pos = sys.argv.index( '-c' ) + sys.argv.pop(pos) + config_file = sys.argv.pop( pos ) + if not os.path.exists( config_file ): + print "Galaxy config file does not exist (hint: use '-c config.ini' for non-standard locations): %s" % config_file + sys.exit( 1 ) + repo = 'lib/galaxy/model/migrate' + +cp = SafeConfigParser() +cp.read( config_file ) + +if cp.has_option( "app:main", "database_connection" ): + db_url = cp.get( "app:main", "database_connection" ) +elif cp.has_option( "app:main", "database_file" ): + db_url = "sqlite:///%s?isolation_level=IMMEDIATE" % cp.get( "app:main", "database_file" ) +else: + db_url = "sqlite:///./database/universe.sqlite?isolation_level=IMMEDIATE" + +dialect_to_egg = { + "sqlite" : "pysqlite>=2", + "postgres" : "psycopg2", + "mysql" : "MySQL_python" +} +dialect = ( db_url.split( ':', 1 ) )[0] +try: + egg = dialect_to_egg[dialect] + try: + pkg_resources.require( egg ) + log.debug( "%s egg successfully loaded for %s dialect" % ( egg, dialect ) ) + except: + # If the module is in the path elsewhere (i.e. non-egg), it'll still load. + log.warning( "%s egg not found, but an attempt will be made to use %s anyway" % ( egg, dialect ) ) +except KeyError: + # Let this go, it could possibly work with db's we don't support + log.error( "database_connection contains an unknown SQLAlchemy database dialect: %s" % dialect ) + +# Setup DB scripting environment +from sqlalchemy import * +from sqlalchemy.orm import * +from sqlalchemy.exc import * + +engine = create_engine(db_url, echo=True) +db_session = scoped_session( sessionmaker( bind = engine ) ) +from galaxy.model.mapping import context as sa_session +sa_session.bind = engine +from galaxy.model import * + Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Eliminate the freebay specific functional test data from the distribution since it is now included in the freebayes repository in the tool shed.
by Bitbucket 16 Aug '12

16 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/a6436bc76869/ changeset: a6436bc76869 user: greg date: 2012-08-15 16:52:46 summary: Eliminate the freebay specific functional test data from the distribution since it is now included in the freebayes repository in the tool shed. affected #: 2 files diff -r 1ce9b9138a59da9cce64cb795e55154a70d51cb1 -r a6436bc768699a48f2d434c376ffeaf767ca71be test-data/variant_detection/freebayes/freebayes_out_1.output_trace --- a/test-data/variant_detection/freebayes/freebayes_out_1.output_trace +++ /dev/null @@ -1,8 +0,0 @@ -phiX174,1411,allele,phiX174,phiX174,A,60,100 -phiX174,1412,allele,phiX174,phiX174,G,60,100 -phiX174,1413,allele,phiX174,phiX174,C,60,100 -phiX174,1414,allele,phiX174,phiX174,G,60,100 -phiX174,1415,allele,phiX174,phiX174,C,60,100 -phiX174,1416,allele,phiX174,phiX174,C,60,100 -phiX174,1417,allele,phiX174,phiX174,G,60,100 -phiX174,1418,allele,phiX174,phiX174,T,60,100 diff -r 1ce9b9138a59da9cce64cb795e55154a70d51cb1 -r a6436bc768699a48f2d434c376ffeaf767ca71be test-data/variant_detection/freebayes/freebayes_out_1.vcf.contains --- a/test-data/variant_detection/freebayes/freebayes_out_1.vcf.contains +++ /dev/null @@ -1,2 +0,0 @@ -#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT A - Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jmchilton: Enable the lwr & dynamic job runners automatically. Neither require
by Bitbucket 16 Aug '12

16 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/1ce9b9138a59/ changeset: 1ce9b9138a59 user: jmchilton date: 2012-08-15 06:38:24 summary: Enable the lwr & dynamic job runners automatically. Neither require any special libraries and deployers must assign tools to use them before they have any affect on Galaxy (i.e. no subtle/implicit behaviors), so I don't anticipate any down side to doing this. Also clean up logic in lib/galaxy/jobs/handler.py a little to accommodate this. affected #: 2 files diff -r f557b7b05fdd701cbf99ee04f311bcadb1ae29c4 -r 1ce9b9138a59da9cce64cb795e55154a70d51cb1 lib/galaxy/jobs/handler.py --- a/lib/galaxy/jobs/handler.py +++ b/lib/galaxy/jobs/handler.py @@ -360,7 +360,7 @@ def __init__( self, app ): self.app = app self.job_runners = {} - start_job_runners = ["local"] + start_job_runners = ["local", "lwr", "dynamic"] if app.config.start_job_runners is not None: start_job_runners.extend( [ x.strip() for x in util.listify( app.config.start_job_runners ) ] ) if app.config.use_tasked_jobs: diff -r f557b7b05fdd701cbf99ee04f311bcadb1ae29c4 -r 1ce9b9138a59da9cce64cb795e55154a70d51cb1 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -922,16 +922,11 @@ if self_id in self.app.config.tool_handlers: self.job_handlers = self.app.config.tool_handlers[ self_id ] # Set job runner(s). Each runner is a dict with 'url' and, optionally, 'params'. - if self.app.config.start_job_runners is None: - # Jobs are always local regardless of tool config if no additional - # runners are started - self.job_runners = [ { "url" : "local:///" } ] - else: - # Set job runner to the cluster default - self.job_runners = [ { "url" : self.app.config.default_cluster_job_runner } ] - # Set custom runner(s) if they're defined. - if self_id in self.app.config.tool_runners: - self.job_runners = self.app.config.tool_runners[ self_id ] + # Set job runner to the cluster default + self.job_runners = [ { "url" : self.app.config.default_cluster_job_runner } ] + # Set custom runner(s) if they're defined. + if self_id in self.app.config.tool_runners: + self.job_runners = self.app.config.tool_runners[ self_id ] # Is this a 'hidden' tool (hidden in tool menu) self.hidden = util.xml_text(root, "hidden") if self.hidden: self.hidden = util.string_as_bool(self.hidden) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: scot...@gatech.edu: Reworked some stdout/stderr parsing to make errors and warnings more apparent.
by Bitbucket 15 Aug '12

15 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/f557b7b05fdd/ changeset: f557b7b05fdd user: scot...(a)gatech.edu date: 2012-08-15 00:58:58 summary: Reworked some stdout/stderr parsing to make errors and warnings more apparent. affected #: 2 files diff -r a7a0a5962648eb68a27ed9e460d1b4ea012c2302 -r f557b7b05fdd701cbf99ee04f311bcadb1ae29c4 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py +++ b/lib/galaxy/jobs/__init__.py @@ -309,9 +309,10 @@ return self.fail( job.info ) # Check the tool's stdout, stderr, and exit code for errors, but only - # if the job has not already been marked as having an error. + # if the job has not already been marked as having an error. + # The job's stdout and stderr will be set accordingly. if job.states.ERROR != job.state: - if ( self.check_tool_output( stdout, stderr, tool_exit_code ) ): + if ( self.check_tool_output( stdout, stderr, tool_exit_code, job )): job.state = job.states.OK else: job.state = job.states.ERROR @@ -335,7 +336,7 @@ log.warning( "finish(): %s not found, but %s is not empty, so it will be used instead" % ( dataset_path.false_path, dataset_path.real_path ) ) else: return self.fail( "Job %s's output dataset(s) could not be read" % job.id ) - job_context = ExpressionContext( dict( stdout = stdout, stderr = stderr ) ) + job_context = ExpressionContext( dict( stdout = job.stdout, stderr = job.stderr ) ) job_tool = self.app.toolbox.tools_by_id.get( job.tool_id, None ) @@ -430,12 +431,12 @@ # will now be seen by the user. self.sa_session.flush() # Save stdout and stderr - if len( stdout ) > 32768: + if len( job.stdout ) > 32768: log.error( "stdout for job %d is greater than 32K, only first part will be logged to database" % job.id ) - job.stdout = stdout[:32768] - if len( stderr ) > 32768: + job.stdout = job.stdout[:32768] + if len( job.stderr ) > 32768: log.error( "stderr for job %d is greater than 32K, only first part will be logged to database" % job.id ) - job.stderr = stderr[:32768] + job.stderr = job.stderr[:32768] # custom post process setup inp_data = dict( [ ( da.name, da.dataset ) for da in job.input_datasets ] ) out_data = dict( [ ( da.name, da.dataset ) for da in job.output_datasets ] ) @@ -457,7 +458,7 @@ # Call 'exec_after_process' hook self.tool.call_hook( 'exec_after_process', self.queue.app, inp_data=inp_data, out_data=out_data, param_dict=param_dict, - tool=self.tool, stdout=stdout, stderr=stderr ) + tool=self.tool, stdout=job.stdout, stderr=job.stderr ) job.command_line = self.command_line bytes = 0 @@ -477,7 +478,7 @@ if self.app.config.cleanup_job == 'always' or ( not stderr and self.app.config.cleanup_job == 'onsuccess' ): self.cleanup() - def check_tool_output( self, stdout, stderr, tool_exit_code ): + def check_tool_output( self, stdout, stderr, tool_exit_code, job ): """ Check the output of a tool - given the stdout, stderr, and the tool's exit code, return True if the tool exited succesfully and False @@ -487,8 +488,8 @@ Note that, if the tool did not define any exit code handling or any stdio/stderr handling, then it reverts back to previous behavior: if stderr contains anything, then False is returned. + Note that the job id is just for messages. """ - job = self.get_job() err_msg = "" # By default, the tool succeeded. This covers the case where the code # has a bug but the tool was ok, and it lets a workflow continue. @@ -497,10 +498,14 @@ try: # Check exit codes and match regular expressions against stdout and # stderr if this tool was configured to do so. + # If there is a regular expression for scanning stdout/stderr, + # then we assume that the tool writer overwrote the default + # behavior of just setting an error if there is *anything* on + # stderr. if ( len( self.tool.stdio_regexes ) > 0 or len( self.tool.stdio_exit_codes ) > 0 ): - # We will check the exit code ranges in the order in which - # they were specified. Each exit_code is a ToolStdioExitCode + # Check the exit code ranges in the order in which + # they were specified. Each exit_code is a StdioExitCode # that includes an applicable range. If the exit code was in # that range, then apply the error level and add in a message. # If we've reached a fatal error rule, then stop. @@ -508,24 +513,33 @@ for stdio_exit_code in self.tool.stdio_exit_codes: if ( tool_exit_code >= stdio_exit_code.range_start and tool_exit_code <= stdio_exit_code.range_end ): - if None != stdio_exit_code.desc: - err_msg += stdio_exit_code.desc - # TODO: Find somewhere to stick the err_msg - possibly to - # the source (stderr/stdout), possibly in a new db column. + # Tack on a generic description of the code + # plus a specific code description. For example, + # this might append "Job 42: Warning: Out of Memory\n". + # TODO: Find somewhere to stick the err_msg - + # possibly to the source (stderr/stdout), possibly + # in a new db column. + code_desc = stdio_exit_code.desc + if ( None == code_desc ): + code_desc = "" + tool_msg = ( "Job %s: %s: Exit code %d: %s" % ( + job.get_id_tag(), + galaxy.tools.StdioErrorLevel.desc( tool_exit_code ), + tool_exit_code, + code_desc ) ) + log.info( tool_msg ) + stderr = err_msg + stderr max_error_level = max( max_error_level, stdio_exit_code.error_level ) - if max_error_level >= galaxy.tools.StdioErrorLevel.FATAL: + if ( max_error_level >= + galaxy.tools.StdioErrorLevel.FATAL ): break - # If there is a regular expression for scanning stdout/stderr, - # then we assume that the tool writer overwrote the default - # behavior of just setting an error if there is *anything* on - # stderr. if max_error_level < galaxy.tools.StdioErrorLevel.FATAL: # We'll examine every regex. Each regex specifies whether # it is to be run on stdout, stderr, or both. (It is # possible for neither stdout nor stderr to be scanned, - # but those won't be scanned.) We record the highest + # but those regexes won't be used.) We record the highest # error level, which are currently "warning" and "fatal". # If fatal, then we set the job's state to ERROR. # If warning, then we still set the job's state to OK @@ -539,19 +553,32 @@ # Repeat the stdout stuff for stderr. # TODO: Collapse this into a single function. if ( regex.stdout_match ): - regex_match = re.search( regex.match, stdout ) + regex_match = re.search( regex.match, stdout, + re.IGNORECASE ) if ( regex_match ): - err_msg += self.regex_err_msg( regex_match, regex ) - max_error_level = max( max_error_level, regex.error_level ) - if max_error_level >= galaxy.tools.StdioErrorLevel.FATAL: + rexmsg = self.regex_err_msg( regex_match, regex) + log.info( "Job %s: %s" + % ( job.get_id_tag(), rexmsg ) ) + stdout = rexmsg + "\n" + stdout + max_error_level = max( max_error_level, + regex.error_level ) + if ( max_error_level >= + galaxy.tools.StdioErrorLevel.FATAL ): break - if ( regex.stderr_match ): - regex_match = re.search( regex.match, stderr ) + + if ( regex.stderr_match ): + regex_match = re.search( regex.match, stderr, + re.IGNORECASE ) if ( regex_match ): - err_msg += self.regex_err_msg( regex_match, regex ) - max_error_level = max( max_error_level, + rexmsg = self.regex_err_msg( regex_match, regex) + # DELETEME + log.info( "Job %s: %s" + % ( job.get_id_tag(), rexmsg ) ) + stderr = rexmsg + "\n" + stderr + max_error_level = max( max_error_level, regex.error_level ) - if max_error_level >= galaxy.tools.StdioErrorLevel.FATAL: + if ( max_error_level >= + galaxy.tools.StdioErrorLevel.FATAL ): break # If we encountered a fatal error, then we'll need to set the @@ -565,17 +592,26 @@ # default to the previous behavior: when there's anything on stderr # the job has an error, and the job is ok otherwise. else: - log.debug( "The tool did not define exit code or stdio handling; " + # TODO: Add in the tool and job id: + log.debug( "Tool did not define exit code or stdio handling; " + "checking stderr for success" ) if stderr: success = False else: success = True + # On any exception, return True. except: + tb = traceback.format_exc() log.warning( "Tool check encountered unexpected exception; " - + "assuming tool was successful" ) + + "assuming tool was successful: " + tb ) success = True + + # Store the modified stdout and stderr in the job: + if None != job: + job.stdout = stdout + job.stderr = stderr + return success def regex_err_msg( self, match, regex ): diff -r a7a0a5962648eb68a27ed9e460d1b4ea012c2302 -r f557b7b05fdd701cbf99ee04f311bcadb1ae29c4 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py +++ b/lib/galaxy/tools/__init__.py @@ -1316,6 +1316,8 @@ return_level = StdioErrorLevel.WARNING elif ( re.search( "fatal", err_level, re.IGNORECASE ) ): return_level = StdioErrorLevel.FATAL + else: + log.debug( "Error level %s did not match warning/fatal" % err_level ) except Exception, e: log.error( "Exception in parse_error_level " + str(sys.exc_info() ) ) Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: greg: Fix for recently introduced bug that breaks installation of tool shed repositories into a local Galaxy instance if the repository includes tools that use the ToolDataTableManage. Also, move processing of sample files and other config files included in installed tool shed repositories to disk rather than memory.
by Bitbucket 15 Aug '12

15 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/a7a0a5962648/ changeset: a7a0a5962648 user: greg date: 2012-08-14 22:24:54 summary: Fix for recently introduced bug that breaks installation of tool shed repositories into a local Galaxy instance if the repository includes tools that use the ToolDataTableManage. Also, move processing of sample files and other config files included in installed tool shed repositories to disk rather than memory. affected #: 4 files diff -r b4f52783c54ac175d306d5064ef5ab7bd5149ca9 -r a7a0a5962648eb68a27ed9e460d1b4ea012c2302 lib/galaxy/tool_shed/install_manager.py --- a/lib/galaxy/tool_shed/install_manager.py +++ b/lib/galaxy/tool_shed/install_manager.py @@ -136,27 +136,22 @@ repository_clone_url=repository_clone_url, relative_install_dir=relative_install_dir, repository_files_dir=None, - resetting_all_metadata_on_repository=False ) + resetting_all_metadata_on_repository=False, + webapp='galaxy' ) tool_shed_repository.metadata = metadata_dict self.app.sa_session.add( tool_shed_repository ) self.app.sa_session.flush() if 'tool_dependencies' in metadata_dict: # All tool_dependency objects must be created before the tools are processed even if no tool dependencies will be installed. - tool_dependencies = create_tool_dependency_objects( self.app, tool_shed_repository, tool_shed_repository.installed_changeset_revision, set_status=True ) + tool_dependencies = create_tool_dependency_objects( self.app, tool_shed_repository, relative_install_dir, set_status=True ) else: tool_dependencies = None if 'tools' in metadata_dict: - work_dir = tempfile.mkdtemp() repository_tools_tups = get_repository_tools_tups( self.app, metadata_dict ) if repository_tools_tups: sample_files = metadata_dict.get( 'sample_files', [] ) # Handle missing data table entries for tool parameters that are dynamically generated select lists. - repository_tools_tups = handle_missing_data_table_entry( self.app, - tool_shed_repository, - tool_shed_repository.installed_changeset_revision, - self.tool_path, - repository_tools_tups, - work_dir ) + repository_tools_tups = handle_missing_data_table_entry( self.app, relative_install_dir, self.tool_path, repository_tools_tups ) # Handle missing index files for tool parameters that are dynamically generated select lists. repository_tools_tups, sample_files_copied = handle_missing_index_file( self.app, self.tool_path, sample_files, repository_tools_tups ) # Copy remaining sample files included in the repository to the ~/tool-data directory of the local Galaxy instance. @@ -166,12 +161,8 @@ update_tool_shed_repository_status( self.app, tool_shed_repository, self.app.model.ToolShedRepository.installation_status.INSTALLING_TOOL_DEPENDENCIES ) - # Get the tool_dependencies.xml file from the repository. - tool_dependencies_config = get_config_from_repository( self.app, - 'tool_dependencies.xml', - tool_shed_repository, - tool_shed_repository.installed_changeset_revision, - work_dir ) + # Get the tool_dependencies.xml file from disk. + tool_dependencies_config = get_config_from_disk( 'tool_dependencies.xml', relative_install_dir ) installed_tool_dependencies = handle_tool_dependencies( app=self.app, tool_shed_repository=tool_shed_repository, tool_dependencies_config=tool_dependencies_config, @@ -189,10 +180,6 @@ self.migrated_tools_config, tool_panel_dict=tool_panel_dict_for_display, new_install=True ) - try: - shutil.rmtree( work_dir ) - except: - pass if 'datatypes' in metadata_dict: tool_shed_repository.status = self.app.model.ToolShedRepository.installation_status.LOADING_PROPRIETARY_DATATYPES if not tool_shed_repository.includes_datatypes: @@ -200,11 +187,7 @@ self.app.sa_session.add( tool_shed_repository ) self.app.sa_session.flush() work_dir = tempfile.mkdtemp() - datatypes_config = get_config_from_repository( self.app, - 'datatypes_conf.xml', - tool_shed_repository, - tool_shed_repository.installed_changeset_revision, - work_dir ) + datatypes_config = get_config_from_disk( 'datatypes_conf.xml', relative_install_dir ) # Load proprietary data types required by tools. The value of override is not important here since the Galaxy server will be started # after this installation completes. converter_path, display_path = alter_config_and_load_prorietary_datatypes( self.app, datatypes_config, relative_install_dir, override=False ) diff -r b4f52783c54ac175d306d5064ef5ab7bd5149ca9 -r a7a0a5962648eb68a27ed9e460d1b4ea012c2302 lib/galaxy/util/shed_util.py --- a/lib/galaxy/util/shed_util.py +++ b/lib/galaxy/util/shed_util.py @@ -248,7 +248,7 @@ except: pass return converter_path, display_path -def check_tool_input_params( app, repo_dir, tool_config_name, tool, sample_files ): +def check_tool_input_params( app, repo_dir, tool_config_name, tool, sample_files, webapp='galaxy' ): """ Check all of the tool's input parameters, looking for any that are dynamically generated using external data files to make sure the files exist. @@ -291,8 +291,9 @@ correction_msg = "This file refers to a file named <b>%s</b>. " % str( index_file ) correction_msg += "Upload a file named <b>%s.sample</b> to the repository to correct this error." % str( index_file_name ) invalid_files_and_errors_tups.append( ( tool_config_name, correction_msg ) ) - # Reset the tool_data_tables by loading the empty tool_data_table_conf.xml file. - reset_tool_data_tables( app ) + if webapp == 'community': + # Reset the tool_data_tables by loading the empty tool_data_table_conf.xml file. + reset_tool_data_tables( app ) return invalid_files_and_errors_tups def config_elems_to_xml_file( app, config_elems, config_filename, tool_path ): # Persist the current in-memory list of config_elems to a file named by the value of config_filename. @@ -427,16 +428,11 @@ sa_session.add( tool_shed_repository ) sa_session.flush() return tool_shed_repository -def create_tool_dependency_objects( app, tool_shed_repository, current_changeset_revision, set_status=True ): +def create_tool_dependency_objects( app, tool_shed_repository, relative_install_dir, set_status=True ): # Create or update a ToolDependency for each entry in tool_dependencies_config. This method is called when installing a new tool_shed_repository. tool_dependency_objects = [] - work_dir = tempfile.mkdtemp() # Get the tool_dependencies.xml file from the repository. - tool_dependencies_config = get_config_from_repository( app, - 'tool_dependencies.xml', - tool_shed_repository, - current_changeset_revision, - work_dir ) + tool_dependencies_config = get_config_from_disk( 'tool_dependencies.xml', relative_install_dir ) tree = ElementTree.parse( tool_dependencies_config ) root = tree.getroot() ElementInclude.include( root ) @@ -454,10 +450,6 @@ status=app.model.ToolDependency.installation_status.NEVER_INSTALLED, set_status=set_status ) tool_dependency_objects.append( tool_dependency ) - try: - shutil.rmtree( work_dir ) - except: - pass return tool_dependency_objects def generate_clone_url( trans, repository ): """Generate the URL for cloning a repository.""" @@ -548,7 +540,8 @@ if not can_generate_dependency_metadata: break return can_generate_dependency_metadata -def generate_metadata_for_changeset_revision( app, repository_clone_url, relative_install_dir=None, repository_files_dir=None, resetting_all_metadata_on_repository=False ): +def generate_metadata_for_changeset_revision( app, repository_clone_url, relative_install_dir=None, repository_files_dir=None, + resetting_all_metadata_on_repository=False, webapp='galaxy' ): """ Generate metadata for a repository using it's files on disk. To generate metadata for changeset revisions older than the repository tip, the repository will have been cloned to a temporary location and updated to a specified changeset revision to access that changeset revision's @@ -629,7 +622,7 @@ invalid_tool_configs.append( name ) invalid_file_tups.append( ( name, str( e ) ) ) if tool is not None: - invalid_files_and_errors_tups = check_tool_input_params( app, files_dir, name, tool, sample_files ) + invalid_files_and_errors_tups = check_tool_input_params( app, files_dir, name, tool, sample_files, webapp=webapp ) can_set_metadata = True for tup in invalid_files_and_errors_tups: if name in tup: @@ -664,7 +657,7 @@ metadata_dict = generate_tool_dependency_metadata( tool_dependencies_config, metadata_dict ) if invalid_tool_configs: metadata_dict [ 'invalid_tools' ] = invalid_tool_configs - if resetting_all_metadata_on_repository: + if webapp == 'community' and resetting_all_metadata_on_repository: # Reset the tool_data_tables by loading the empty tool_data_table_conf.xml file. reset_tool_data_tables( app ) # Reset the value of the app's tool_data_path to it's original value. @@ -1255,7 +1248,7 @@ return shed_url # The tool shed from which the repository was originally installed must no longer be configured in tool_sheds_conf.xml. return None -def handle_missing_data_table_entry( app, repository, changeset_revision, tool_path, repository_tools_tups, dir ): +def handle_missing_data_table_entry( app, relative_install_dir, tool_path, repository_tools_tups ): """ Inspect each tool to see if any have input parameters that are dynamically generated select lists that require entries in the tool_data_table_conf.xml file. This method is called only from Galaxy (not the tool shed) when a repository is being installed @@ -1269,7 +1262,7 @@ break if missing_data_table_entry: # The repository must contain a tool_data_table_conf.xml.sample file that includes all required entries for all tools in the repository. - sample_tool_data_table_conf = get_config_from_repository( app, 'tool_data_table_conf.xml.sample', repository, changeset_revision, dir ) + sample_tool_data_table_conf = get_config_from_disk( 'tool_data_table_conf.xml.sample', relative_install_dir ) # Add entries to the ToolDataTableManager's in-memory data_tables dictionary as well as the list of data_table_elems and the list of # data_table_elem_names. error, correction_msg = handle_sample_tool_data_table_conf_file( app, sample_tool_data_table_conf, persist=True ) @@ -1394,14 +1387,8 @@ def load_installed_datatypes( app, repository, relative_install_dir, deactivate=False ): # Load proprietary datatypes and return information needed for loading proprietary datatypes converters and display applications later. metadata = repository.metadata - work_dir = tempfile.mkdtemp() repository_dict = None - datatypes_config = get_config_from_repository( app, - 'datatypes_conf.xml', - repository, - repository.changeset_revision, - work_dir, - install_dir=relative_install_dir ) + datatypes_config = get_config_from_disk( 'datatypes_conf.xml', relative_install_dir ) if datatypes_config: converter_path, display_path = alter_config_and_load_prorietary_datatypes( app, datatypes_config, relative_install_dir, deactivate=deactivate ) if converter_path or display_path: @@ -1413,10 +1400,6 @@ tool_dicts=metadata.get( 'tools', [] ), converter_path=converter_path, display_path=display_path ) - try: - shutil.rmtree( work_dir ) - except: - pass return repository_dict def load_installed_display_applications( app, installed_repository_dict, deactivate=False ): # Load or deactivate proprietary datatype display applications diff -r b4f52783c54ac175d306d5064ef5ab7bd5149ca9 -r a7a0a5962648eb68a27ed9e460d1b4ea012c2302 lib/galaxy/web/controllers/admin_toolshed.py --- a/lib/galaxy/web/controllers/admin_toolshed.py +++ b/lib/galaxy/web/controllers/admin_toolshed.py @@ -525,13 +525,8 @@ # Get the tool_shed_repository from one of the tool_dependencies. message = '' tool_shed_repository = tool_dependencies[ 0 ].tool_shed_repository - work_dir = tempfile.mkdtemp() # Get the tool_dependencies.xml file from the repository. - tool_dependencies_config = get_config_from_repository( trans.app, - 'tool_dependencies.xml', - tool_shed_repository, - tool_shed_repository.changeset_revision, - work_dir ) + tool_dependencies_config = get_config_from_disk( 'tool_dependencies.xml', tool_shed_repository.repo_path( trans.app ) ) installed_tool_dependencies = handle_tool_dependencies( app=trans.app, tool_shed_repository=tool_shed_repository, tool_dependencies_config=tool_dependencies_config, @@ -539,10 +534,6 @@ for installed_tool_dependency in installed_tool_dependencies: if installed_tool_dependency.status == trans.app.model.ToolDependency.installation_status.ERROR: message += ' %s' % installed_tool_dependency.error_message - try: - shutil.rmtree( work_dir ) - except: - pass tool_dependency_ids = [ trans.security.encode_id( td.id ) for td in tool_dependencies ] if message: status = 'error' @@ -663,11 +654,7 @@ tool_shed_repository, trans.model.ToolShedRepository.installation_status.INSTALLING_TOOL_DEPENDENCIES ) # Get the tool_dependencies.xml file from the repository. - tool_dependencies_config = get_config_from_repository( trans.app, - 'tool_dependencies.xml', - tool_shed_repository, - tool_shed_repository.installed_changeset_revision, - work_dir ) + tool_dependencies_config = get_config_from_disk( 'tool_dependencies.xml', relative_install_dir ) installed_tool_dependencies = handle_tool_dependencies( app=trans.app, tool_shed_repository=tool_shed_repository, tool_dependencies_config=tool_dependencies_config, @@ -691,24 +678,19 @@ repository_clone_url=repository_clone_url, relative_install_dir=relative_install_dir, repository_files_dir=None, - resetting_all_metadata_on_repository=False ) + resetting_all_metadata_on_repository=False, + webapp='galaxy' ) tool_shed_repository.metadata = metadata_dict trans.sa_session.add( tool_shed_repository ) trans.sa_session.flush() if 'tool_dependencies' in metadata_dict and not reinstalling: - tool_dependencies = create_tool_dependency_objects( trans.app, tool_shed_repository, tool_shed_repository.installed_changeset_revision, set_status=True ) + tool_dependencies = create_tool_dependency_objects( trans.app, tool_shed_repository, relative_install_dir, set_status=True ) if 'tools' in metadata_dict: tool_panel_dict = generate_tool_panel_dict_for_new_install( metadata_dict[ 'tools' ], tool_section ) repository_tools_tups = get_repository_tools_tups( trans.app, metadata_dict ) if repository_tools_tups: # Handle missing data table entries for tool parameters that are dynamically generated select lists. - work_dir = tempfile.mkdtemp() - repository_tools_tups = handle_missing_data_table_entry( trans.app, - tool_shed_repository, - tool_shed_repository.changeset_revision, - tool_path, - repository_tools_tups, - work_dir ) + repository_tools_tups = handle_missing_data_table_entry( trans.app, relative_install_dir, tool_path, repository_tools_tups ) # Handle missing index files for tool parameters that are dynamically generated select lists. sample_files = metadata_dict.get( 'sample_files', [] ) repository_tools_tups, sample_files_copied = handle_missing_index_file( trans.app, tool_path, sample_files, repository_tools_tups ) @@ -723,22 +705,13 @@ shed_tool_conf=shed_tool_conf, tool_panel_dict=tool_panel_dict, new_install=True ) - try: - shutil.rmtree( work_dir ) - except: - pass if 'datatypes' in metadata_dict: tool_shed_repository.status = trans.model.ToolShedRepository.installation_status.LOADING_PROPRIETARY_DATATYPES if not tool_shed_repository.includes_datatypes: tool_shed_repository.includes_datatypes = True trans.sa_session.add( tool_shed_repository ) trans.sa_session.flush() - work_dir = tempfile.mkdtemp() - datatypes_config = get_config_from_repository( trans.app, - 'datatypes_conf.xml', - tool_shed_repository, - tool_shed_repository.changeset_revision, - work_dir ) + datatypes_config = get_config_from_disk( 'datatypes_conf.xml', relative_install_dir ) # Load data types required by tools. converter_path, display_path = alter_config_and_load_prorietary_datatypes( trans.app, datatypes_config, relative_install_dir, override=False ) if converter_path or display_path: @@ -756,10 +729,6 @@ if display_path: # Load proprietary datatype display applications trans.app.datatypes_registry.load_display_applications( installed_repository_dict=repository_dict ) - try: - shutil.rmtree( work_dir ) - except: - pass @web.expose @web.require_admin def manage_repository( self, trans, **kwd ): @@ -1507,7 +1476,7 @@ trans.sa_session.flush() # Create tool_dependency records if necessary. if 'tool_dependencies' in metadata_dict: - tool_dependencies = create_tool_dependency_objects( trans.app, repository, repository.changeset_revision, set_status=False ) + tool_dependencies = create_tool_dependency_objects( trans.app, repository, relative_install_dir, set_status=False ) message = "The installed repository named '%s' has been updated to change set revision '%s'. " % ( name, latest_changeset_revision ) # See if any tool dependencies can be installed. shed_tool_conf, tool_path, relative_install_dir = get_tool_panel_config_tool_path_install_dir( trans.app, repository ) diff -r b4f52783c54ac175d306d5064ef5ab7bd5149ca9 -r a7a0a5962648eb68a27ed9e460d1b4ea012c2302 lib/galaxy/webapps/community/controllers/common.py --- a/lib/galaxy/webapps/community/controllers/common.py +++ b/lib/galaxy/webapps/community/controllers/common.py @@ -795,7 +795,8 @@ repository_clone_url=repository_clone_url, relative_install_dir=repo_dir, repository_files_dir=work_dir, - resetting_all_metadata_on_repository=True ) + resetting_all_metadata_on_repository=True, + webapp='community' ) if current_metadata_dict: if not metadata_changeset_revision and not metadata_dict: # We're at the first change set in the change log. @@ -885,7 +886,8 @@ repository_clone_url=repository_clone_url, relative_install_dir=repo_dir, repository_files_dir=None, - resetting_all_metadata_on_repository=False ) + resetting_all_metadata_on_repository=False, + webapp='community' ) if metadata_dict: downloadable = is_downloadable( metadata_dict ) repository_metadata = None Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: natefoo: Reclaim ownership of job's working dir prior to reading stdout/stderr/ec files to avoid permissions problems.
by Bitbucket 15 Aug '12

15 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/b4f52783c54a/ changeset: b4f52783c54a user: natefoo date: 2012-08-14 21:22:29 summary: Reclaim ownership of job's working dir prior to reading stdout/stderr/ec files to avoid permissions problems. affected #: 1 file diff -r 0ef82aa1c39cfb192ed6d1c0a1ed3a1a9e103485 -r b4f52783c54ac175d306d5064ef5ab7bd5149ca9 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.py +++ b/lib/galaxy/jobs/runners/drmaa.py @@ -339,6 +339,8 @@ # By default, the exit code is 0, which typically indicates success. exit_code = 0 exit_code_str = "0" + # To ensure that files below are readable, ownership must be reclaimed first + drm_job_state.job_wrapper.reclaim_ownership() while which_try < (self.app.config.retry_job_output_collection + 1): try: ofh = file(ofile, "r") Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: carlfeberhard: tool-data change to fix bwa_wrapper.test_tool_000001; backbone history.js start
by Bitbucket 15 Aug '12

15 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/0ef82aa1c39c/ changeset: 0ef82aa1c39c user: carlfeberhard date: 2012-08-14 19:15:48 summary: tool-data change to fix bwa_wrapper.test_tool_000001; backbone history.js start affected #: 4 files diff -r 40ad570cf570d4f2f2dcbf87c5eb3c212efe6f1c -r 0ef82aa1c39cfb192ed6d1c0a1ed3a1a9e103485 static/scripts/mvc/history.js --- /dev/null +++ b/static/scripts/mvc/history.js @@ -0,0 +1,145 @@ +/* +TODO: + as always: where does the model end and the view begin? + HistoryPanel + HistoryCollection: (collection of histories: 'Saved Histories') + +CASES: + logged-in/NOT +*/ +//============================================================================== +var HistoryItem = BaseModel.extend({ + // a single history structure + // from: http://localhost:8080/api/histories/f2db41e1fa331b3e/contents/f2db41e1fa331… + /* + { + "data_type": "fastq", + "deleted": false, + "download_url": "/datasets/f2db41e1fa331b3e/display?to_ext=fastq", + "file_size": 226297533, + "genome_build": "?", + "id": "f2db41e1fa331b3e", + "metadata_data_lines": null, + "metadata_dbkey": "?", + "metadata_sequences": null, + "misc_blurb": "215.8 MB", + "misc_info": "uploaded fastq file", + "model_class": "HistoryDatasetAssociation", + "name": "LTCF-2-19_GTGAAA_L001_R1_001.fastq", + "state": "ok", + "visible": true + } + */ + + display : function(){}, + edit_attr : function(){}, + delete : function(){}, + download : function(){}, + details : function(){}, + rerun : function(){}, + tags : function(){}, + annotations : function(){}, + peek : function(){}, +}); + +//.............................................................................. +var HistoryItemView = BaseView.extend({ + // view for History model used in HistoryPanelView + tagName : "div", + className : "historyItemContainer", + + icons : { + display : 'path to icon', + edit_attr : 'path to icon', + delete : 'path to icon', + download : 'path to icon', + details : 'path to icon', + rerun : 'path to icon', + tags : 'path to icon', + annotations : 'path to icon', + }, + + render : function(){ + this.$el.append( 'div' ) + }, + +}); + + + +//============================================================================== +var History = Backbone.Collection.extend({ + // a collection of HistoryItems + + // from: http://localhost:8080/api/histories/f2db41e1fa331b3e + /* + { + "contents_url": "/api/histories/f2db41e1fa331b3e/contents", + "id": "f2db41e1fa331b3e", + "name": "one", + "state": "ok", + "state_details": { + "discarded": 0, + "empty": 0, + "error": 0, + "failed_metadata": 0, + "new": 0, + "ok": 4, + "queued": 0, + "running": 0, + "setting_metadata": 0, + "upload": 0 + } + } + */ + + // from: http://localhost:8080/api/histories/f2db41e1fa331b3e/contents + // (most are replicated in HistoryItem) + /* + [ + { + "id": "f2db41e1fa331b3e", + "name": "LTCF-2-19_GTGAAA_L001_R1_001.fastq", + "type": "file", + "url": "/api/histories/f2db41e1fa331b3e/contents/f2db41e1fa331b3e" + }, + { + "id": "f597429621d6eb2b", + "name": "LTCF-2-19_GTGAAA_L001_R2_001.fastq", + "type": "file", + "url": "/api/histories/f2db41e1fa331b3e/contents/f597429621d6eb2b" + }, + { + "id": "1cd8e2f6b131e891", + "name": "FASTQ Groomer on data 1", + "type": "file", + "url": "/api/histories/f2db41e1fa331b3e/contents/1cd8e2f6b131e891" + }, + { + "id": "ebfb8f50c6abde6d", + "name": "FASTQ Groomer on data 2", + "type": "file", + "url": "/api/histories/f2db41e1fa331b3e/contents/ebfb8f50c6abde6d" + }, + { + "id": "33b43b4e7093c91f", + "name": "Sa.04-02981.fasta", + "type": "file", + "url": "/api/histories/f2db41e1fa331b3e/contents/33b43b4e7093c91f" + } + ] + */ +}); + +//.............................................................................. +var HistoryCollectionView = BaseView.extend({ + // view for the HistoryCollection (as per current right hand panel) + tagName : "body", + className : "historyCollection", + + render : function(){ + + }, + +}); + diff -r 40ad570cf570d4f2f2dcbf87c5eb3c212efe6f1c -r 0ef82aa1c39cfb192ed6d1c0a1ed3a1a9e103485 templates/base_panels.mako --- a/templates/base_panels.mako +++ b/templates/base_panels.mako @@ -54,13 +54,17 @@ var galaxy_paths = new GalaxyPaths({ root_path: '${h.url_for( "/" )}', image_path: '${h.url_for( "/static/images" )}', + tool_url: '${h.url_for( controller="/api/tools" )}', + history_url: '${h.url_for( controller="/api/histories" )}', + data_url: '${h.url_for( controller="/tracks", action="data" )}', raw_data_url: '${h.url_for( controller="/tracks", action="raw_data" )}', converted_datasets_state_url: '${h.url_for( controller="/tracks", action="converted_datasets_state" )}', dataset_state_url: '${h.url_for( controller="/tracks", action="dataset_state" )}', + sweepster_url: '${h.url_for( controller="/tracks", action="sweepster" )}', + visualization_url: '${h.url_for( controller="/visualization", action="save" )}', - sweepster_url: '${h.url_for( controller="/tracks", action="sweepster" )}' }); </script></%def> diff -r 40ad570cf570d4f2f2dcbf87c5eb3c212efe6f1c -r 0ef82aa1c39cfb192ed6d1c0a1ed3a1a9e103485 test-data/bwa_wrapper_out2.sam --- a/test-data/bwa_wrapper_out2.sam +++ b/test-data/bwa_wrapper_out2.sam @@ -1,30 +1,30 @@ -seq1 16 phiX174 322 25 36M * 0 0 GATATTTTAAAGGAGCGTGGATTACTATCTGAGTCC B&&I13A$G$*%$IIIIIII9(.+5$IIIIIII#II XT:A:U NM:i:2 X0:i:1 XM:i:2 XO:i:0 XG:i:0 MD:Z:2C8A24 -seq10 0 phiX174 4149 37 17M1D19M * 0 0 ATTCTTTCTTTTCGTATCAGGGCGTTGAGTTCGATA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:5G11^G19 -seq11 0 phiX174 4072 37 18M1D18M * 0 0 GCATTTCTACTCCTTCTCATCCCCAATGCTTGGCTT II#IIIIIII$5+.(9IIIIIII$%*$G$A31I&&B XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:12T5^A18 +seq1 16 phiX174 322 25 36M * 0 0 GATATTTTAAAGGAGCGTGGATTACTATCTGAGTCC B&&I13A$G$*%$IIIIIII9(.+5$IIIIIII#II XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:2 XO:i:0 XG:i:0 MD:Z:2C8A24 +seq2 0 phiX174 141 37 36M * 0 0 ATTCGACCTATCCTTGCGCAGCTCGAGAAGCTCTTA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 X1:i:0 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 +seq3 0 phiX174 505 37 36M * 0 0 GTAACAAAGTTTGGATTGCTACTGACCGCTCTCGTG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 X1:i:0 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 +seq4 4 * 0 0 * * 0 0 AGCCGCTCGTCTTTTATGTAGGTGGTCAACCATTTT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII +seq5 0 phiX174 4985 25 36M * 0 0 CAGTTATATGGCTTTTGGTTTCTATGTGGCTTAATA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:2 XO:i:0 XG:i:0 MD:Z:13G17A4 +seq6 0 phiX174 925 37 11M1D25M * 0 0 AGGCGCTCGTCTTGGTATGTAGGTGGTCAACAATTT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:1 X0:i:1 X1:i:0 XM:i:0 XO:i:1 XG:i:1 MD:Z:11^T25 +seq7 0 phiX174 943 37 13M1I22M * 0 0 TGTAGGTGGTCAACCAATTTTAATTGCAGGGGCTTC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:1 X0:i:1 X1:i:0 XM:i:0 XO:i:1 XG:i:1 MD:Z:35 +seq8 4 * 0 0 * * 0 0 ACACCCGTCCTTTACGTCATGCGCTCTATTCTCTGG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII +seq9 0 phiX174 2596 37 16M1I19M * 0 0 GCCGCTATTCAGGTTGTTTTCTGTTGGTGCTGATAT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:7A27 +seq10 0 phiX174 4149 37 17M1D19M * 0 0 ATTCTTTCTTTTCGTATCAGGGCGTTGAGTTCGATA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:5G11^G19 +seq11 0 phiX174 4072 37 18M1D18M * 0 0 GCATTTCTACTCCTTCTCATCCCCAATGCTTGGCTT II#IIIIIII$5+.(9IIIIIII$%*$G$A31I&&B XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:12T5^A18 seq12 4 * 0 0 * * 0 0 CGCGCTTCGATAAAAATGGGATTGGCGTTTCCAACC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII seq13 4 * 0 0 * * 0 0 ATTTCTACTCTTTCTCATCCCCAATCCTTGCCTTCC IIIIIIIIIIIIIIIIIIIIIAAIIIIIIIIIIIII -seq14 0 phiX174 3998 37 21M1D15M * 0 0 CCCTTTTGAATGTCACGCTGATATTTTGACTTTGAG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:5C15^T15 +seq14 0 phiX174 3998 37 21M1D15M * 0 0 CCCTTTTGAATGTCACGCTGATATTTTGACTTTGAG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:5C15^T15 seq15 4 * 0 0 * * 0 0 CCAACTTACCAAGGTGGGTTACGAAACGCGACGCCG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII seq16 4 * 0 0 * * 0 0 TCAGGGTATTAAAAGAGATTATTTTTCTCCAGCCAC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII -seq17 0 phiX174 3034 37 19M1D17M * 0 0 GTGATGTGCTTGCTACCGAAACAATACTTTAGGCAT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:19^T9G7 +seq17 0 phiX174 3034 37 19M1D17M * 0 0 GTGATGTGCTTGCTACCGAAACAATACTTTAGGCAT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:19^T9G7 seq18 4 * 0 0 * * 0 0 TCAATCCCCCATGCTTGGCCGTTCCATAAGCAGATG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII seq19 4 * 0 0 * * 0 0 TTCCTGCGCTTAATGCTTGAGCGTCCTGGTGCTGAT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII -seq2 0 phiX174 141 37 36M * 0 0 ATTCGACCTATCCTTGCGCAGCTCGAGAAGCTCTTA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 -seq20 0 phiX174 1082 37 36M * 0 0 CTTATTACCATTTCAACTACTCCGGTTATCGCTGGC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 -seq21 0 phiX174 1344 37 15M1D21M * 0 0 CTGATACCAATAAAACCCTAAGCATTTGGTTCAGGG II#IIIIIII$5+.(9IIIIIII$%*$G$A31I&&B XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:15^T13T7 +seq20 0 phiX174 1082 37 36M * 0 0 CTTATTACCATTTCAACTACTCCGGTTATCGCTGGC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 X1:i:0 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 +seq21 0 phiX174 1344 37 15M1D21M * 0 0 CTGATACCAATAAAACCCTAAGCATTTGGTTCAGGG II#IIIIIII$5+.(9IIIIIII$%*$G$A31I&&B XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:15^T13T7 seq22 4 * 0 0 * * 0 0 AATCAAACTTACCAAGGGGTTACGACGCGACGCCGT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII seq23 4 * 0 0 * * 0 0 TGTGCTTCCCCAACTTGATTTAATAACCCTATAGAC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII -seq24 0 phiX174 4084 37 17M1I18M * 0 0 TTTCTCAATCCCCAATGCCTTGGCTTCCCTAAGCAG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:27A7 -seq25 0 phiX174 520 37 16M1I19M * 0 0 TTGCTACTGACCGCTCTTCGTGCTCGTTGCTGCGTT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:26C8 -seq26 0 phiX174 1976 37 36M * 0 0 CCGCGTGAAATTTCTATGAAGGATGTTTTCCGTTCT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 -seq27 0 phiX174 2598 37 20M1I15M * 0 0 CGCTAATCAAGTTGTTTCTGTTTGGTGCTGATATTG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:9G25 +seq24 0 phiX174 4084 37 17M1I18M * 0 0 TTTCTCAATCCCCAATGCCTTGGCTTCCCTAAGCAG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:27A7 +seq25 0 phiX174 520 37 16M1I19M * 0 0 TTGCTACTGACCGCTCTTCGTGCTCGTTGCTGCGTT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:26C8 +seq26 0 phiX174 1976 37 36M * 0 0 CCGCGTGAAATTTCTATGAAGGATGTTTTCCGTTCT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 X1:i:0 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 +seq27 0 phiX174 2598 37 20M1I15M * 0 0 CGCTAATCAAGTTGTTTCTGTTTGGTGCTGATATTG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:9G25 seq28 4 * 0 0 * * 0 0 AAAGAGATTATTTGTCGGTCCAGCCACTAAAGTGAG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII seq29 4 * 0 0 * * 0 0 CAAATTAATGCGCGCTTCGATAATGATTGGGGTATC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII -seq3 0 phiX174 505 37 36M * 0 0 GTAACAAAGTTTGGATTGCTACTGACCGCTCTCGTG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:0 X0:i:1 XM:i:0 XO:i:0 XG:i:0 MD:Z:36 -seq30 0 phiX174 4091 37 18M1I17M * 0 0 ATCCCCTATGCTTGGCTTACCATAAGCAGATGGATA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:6A28 -seq4 4 * 0 0 * * 0 0 AGCCGCTCGTCTTTTATGTAGGTGGTCAACCATTTT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII -seq5 0 phiX174 4985 25 36M * 0 0 CAGTTATATGGCTTTTGGTTTCTATGTGGCTTAATA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:2 XO:i:0 XG:i:0 MD:Z:13G17A4 -seq6 0 phiX174 925 37 11M1D25M * 0 0 AGGCGCTCGTCTTGGTATGTAGGTGGTCAACAATTT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:1 X0:i:1 XM:i:0 XO:i:1 XG:i:1 MD:Z:11^T25 -seq7 0 phiX174 943 37 13M1I22M * 0 0 TGTAGGTGGTCAACCAATTTTAATTGCAGGGGCTTC IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:1 X0:i:1 XM:i:0 XO:i:1 XG:i:1 MD:Z:35 -seq8 4 * 0 0 * * 0 0 ACACCCGTCCTTTACGTCATGCGCTCTATTCTCTGG IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII -seq9 0 phiX174 2596 37 16M1I19M * 0 0 GCCGCTATTCAGGTTGTTTTCTGTTGGTGCTGATAT IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 XM:i:1 XO:i:1 XG:i:1 MD:Z:7A27 +seq30 0 phiX174 4091 37 18M1I17M * 0 0 ATCCCCTATGCTTGGCTTACCATAAGCAGATGGATA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XT:A:U NM:i:2 X0:i:1 X1:i:0 XM:i:1 XO:i:1 XG:i:1 MD:Z:6A28 diff -r 40ad570cf570d4f2f2dcbf87c5eb3c212efe6f1c -r 0ef82aa1c39cfb192ed6d1c0a1ed3a1a9e103485 test/base/twilltestcase.py --- a/test/base/twilltestcase.py +++ b/test/base/twilltestcase.py @@ -729,6 +729,7 @@ self.verify_extra_files_content( extra_files, elem.get( 'id' ), shed_tool_id=shed_tool_id ) except AssertionError, err: errmsg = 'History item %s different than expected, difference (using %s):\n' % ( hid, compare ) + errmsg += "( %s v. %s )\n" % ( local_name, temp_name ) errmsg += str( err ) raise AssertionError( errmsg ) finally: Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0

commit/galaxy-central: jgoecks: Include fusions output in Tophat2 wrapper.
by Bitbucket 15 Aug '12

15 Aug '12

1 new commit in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/40ad570cf570/ changeset: 40ad570cf570 user: jgoecks date: 2012-08-14 18:17:01 summary: Include fusions output in Tophat2 wrapper. affected #: 1 file diff -r be81990d148a81bc2896c3c543d2ff70ad9c3c67 -r 40ad570cf570d4f2f2dcbf87c5eb3c212efe6f1c tools/ngs_rna/tophat2_wrapper.xml --- a/tools/ngs_rna/tophat2_wrapper.xml +++ b/tools/ngs_rna/tophat2_wrapper.xml @@ -296,6 +296,9 @@ </inputs><outputs> + <data format="tabular" name="fusions" label="${tool.name} on ${on_string}: fusions" from_work_dir="tophat_out/fusions.out"> + <filter>(params['fusion_search']['do_search'] == 'Yes')</filter> + </data><data format="bed" name="insertions" label="${tool.name} on ${on_string}: insertions" from_work_dir="tophat_out/insertions.bed"><actions><conditional name="refGenomeSource.genomeSource"> Repository URL: https://bitbucket.org/galaxy/galaxy-central/ -- This is a commit notification from bitbucket.org. You are receiving this because you have the service enabled, addressing the recipient of this email.

1 0