- galaxy-dev - lists.galaxyproject.org

[hg] galaxy 1730: Handle exceptions when building the app (witho...
by Greg Von Kuster 05 Feb '09

05 Feb '09

details: http://www.bx.psu.edu/hg/galaxy/rev/0424f713d58d changeset: 1730:0424f713d58d user: Nate Coraor <nate(a)bx.psu.edu> date: Tue Feb 03 14:14:02 2009 -0500 description: Handle exceptions when building the app (without a try block, you have to manually kill the process if there's an exception during startup) 1 file(s) affected in this change: lib/galaxy/web/buildapp.py diffs (19 lines): diff -r 743237dd0e65 -r 0424f713d58d lib/galaxy/web/buildapp.py --- a/lib/galaxy/web/buildapp.py Mon Feb 02 18:57:07 2009 -0500 +++ b/lib/galaxy/web/buildapp.py Tue Feb 03 14:14:02 2009 -0500 @@ -51,8 +51,13 @@ if 'app' in kwargs: app = kwargs.pop( 'app' ) else: - from galaxy.app import UniverseApplication - app = UniverseApplication( global_conf = global_conf, **kwargs ) + try: + from galaxy.app import UniverseApplication + app = UniverseApplication( global_conf = global_conf, **kwargs ) + except: + import traceback, sys + traceback.print_exc() + sys.exit( 1 ) atexit.register( app.shutdown ) # Create the universe WSGI application webapp = galaxy.web.framework.WebApplication( app, session_cookie='galaxysession' )

1 0

[hg] galaxy 1732: Removed lzo requirement from two tools- getInd...
by Greg Von Kuster 05 Feb '09

05 Feb '09

details: http://www.bx.psu.edu/hg/galaxy/rev/4d4de72c57c8 changeset: 1732:4d4de72c57c8 user: guru date: Tue Feb 03 15:06:21 2009 -0500 description: Removed lzo requirement from two tools- getIndels and qualityFilter. Also added validator to the regression tools. 5 file(s) affected in this change: tools/regVariation/best_regression_subsets.xml tools/regVariation/getIndels.py tools/regVariation/linear_regression.xml tools/regVariation/quality_filter.py tools/regVariation/rcve.xml diffs (94 lines): diff -r e26741c8c642 -r 4d4de72c57c8 tools/regVariation/best_regression_subsets.xml --- a/tools/regVariation/best_regression_subsets.xml Tue Feb 03 14:15:09 2009 -0500 +++ b/tools/regVariation/best_regression_subsets.xml Tue Feb 03 15:06:21 2009 -0500 @@ -13,7 +13,9 @@ <inputs> <param format="tabular" name="input1" type="data" label="Select data" help="Query missing? See TIP below."/> <param name="response_col" label="Response column (Y)" type="data_column" data_ref="input1" /> - <param name="predictor_cols" label="Predictor columns (X)" type="data_column" data_ref="input1" multiple="true" /> + <param name="predictor_cols" label="Predictor columns (X)" type="data_column" data_ref="input1" multiple="true" > + <validator type="no_options" message="Please select at least one column."/> + </param> </inputs> <outputs> <data format="input" name="out_file1" metadata_source="input1" /> diff -r e26741c8c642 -r 4d4de72c57c8 tools/regVariation/getIndels.py --- a/tools/regVariation/getIndels.py Tue Feb 03 14:15:09 2009 -0500 +++ b/tools/regVariation/getIndels.py Tue Feb 03 15:06:21 2009 -0500 @@ -1,7 +1,7 @@ #!/usr/bin/python """ -Estimate INDELs for pait-wise alignments. +Estimate INDELs for pair-wise alignments. usage: %prog maf_input out_file1 out_file2 """ @@ -10,22 +10,10 @@ from galaxy import eggs import pkg_resources pkg_resources.require( "bx-python" ) -pkg_resources.require( "lrucache" ) -try: - pkg_resources.require("numpy") - pkg_resources.require( "python-lzo" ) -except: - pass +pkg_resources.require("numpy") import psyco_full import sys -import os, os.path -from UserDict import DictMixin -import bx.wiggle -from bx.binned_array import BinnedArray, FileBinnedArray -from bx.bitset import * -from bx.bitset_builders import * -from fpconst import isNaN from bx.cookbook import doc_optparse from galaxy.tools.exception_handling import * import bx.align.maf diff -r e26741c8c642 -r 4d4de72c57c8 tools/regVariation/linear_regression.xml --- a/tools/regVariation/linear_regression.xml Tue Feb 03 14:15:09 2009 -0500 +++ b/tools/regVariation/linear_regression.xml Tue Feb 03 15:06:21 2009 -0500 @@ -12,7 +12,9 @@ <inputs> <param format="tabular" name="input1" type="data" label="Select data" help="Query missing? See TIP below."/> <param name="response_col" label="Response column (Y)" type="data_column" data_ref="input1" /> - <param name="predictor_cols" label="Predictor columns (X)" type="data_column" data_ref="input1" multiple="true" /> + <param name="predictor_cols" label="Predictor columns (X)" type="data_column" data_ref="input1" multiple="true" > + <validator type="no_options" message="Please select at least one column."/> + </param> </inputs> <outputs> <data format="input" name="out_file1" metadata_source="input1" /> diff -r e26741c8c642 -r 4d4de72c57c8 tools/regVariation/quality_filter.py --- a/tools/regVariation/quality_filter.py Tue Feb 03 14:15:09 2009 -0500 +++ b/tools/regVariation/quality_filter.py Tue Feb 03 15:06:21 2009 -0500 @@ -12,11 +12,7 @@ import pkg_resources pkg_resources.require( "bx-python" ) pkg_resources.require( "lrucache" ) -try: - pkg_resources.require("numpy") - pkg_resources.require( "python-lzo" ) -except: - pass +pkg_resources.require("numpy") import psyco_full import sys diff -r e26741c8c642 -r 4d4de72c57c8 tools/regVariation/rcve.xml --- a/tools/regVariation/rcve.xml Tue Feb 03 14:15:09 2009 -0500 +++ b/tools/regVariation/rcve.xml Tue Feb 03 15:06:21 2009 -0500 @@ -11,7 +11,9 @@ <inputs> <param format="tabular" name="input1" type="data" label="Select data" help="Query missing? See TIP below."/> <param name="response_col" label="Response column (Y)" type="data_column" data_ref="input1" /> - <param name="predictor_cols" label="Predictor columns (X)" type="data_column" data_ref="input1" multiple="true" /> + <param name="predictor_cols" label="Predictor columns (X)" type="data_column" data_ref="input1" multiple="true"> + <validator type="no_options" message="Please select at least one column."/> + </param> </inputs> <outputs> <data format="input" name="out_file1" metadata_source="input1" />

1 0

[hg] galaxy 1733: Fixes to get functional tests working.
by Greg Von Kuster 05 Feb '09

05 Feb '09

details: http://www.bx.psu.edu/hg/galaxy/rev/a4214de3752e changeset: 1733:a4214de3752e user: guru date: Tue Feb 03 18:34:27 2009 -0500 description: Fixes to get functional tests working. 4 file(s) affected in this change: test-data/branchlength_out.dat tools/hyphy/hyphy_branch_lengths_wrapper.xml tools/regVariation/getIndels.py tools/regVariation/quality_filter.py diffs (95 lines): diff -r 4d4de72c57c8 -r a4214de3752e test-data/branchlength_out.dat --- a/test-data/branchlength_out.dat Tue Feb 03 15:06:21 2009 -0500 +++ b/test-data/branchlength_out.dat Tue Feb 03 18:34:27 2009 -0500 @@ -7,27 +7,27 @@ 1 Node4 4.93827e-31 0 0.23542 1 canFam1.chr14(+):42826409-42826418|canFam1_0 4.93827e-31 0 0.23542 1 Total Tree 3.45679e-30 2.60643e-34 2.54701e-20 -2 hg17.chr7(+):26907310-26907326|hg17_1 1.48148e-31 0 0.0915385 -2 panTro1.chr6(+):28037328-28037344|panTro1_1 1.48148e-31 0 0.0915385 -2 Node1 6.69055e-19 0 0.0915385 -2 mm5.chr6(+):52104031-52104047|mm5_1 5.00368e-19 0 0.0915385 -2 rn3.chr4(+):80734404-80734420|rn3_1 0.0435526 0.00241411 0.219651 -2 Node4 1.48148e-31 0 0.0915385 -2 canFam1.chr14(+):42826418-42826434|canFam1_1 1.76058e-19 0 0.0915385 -2 Total Tree 0.0435526 0.00241411 0.219651 -3 hg17.chr7(+):26907326-26907338|hg17_2 1.70011e-31 0 0.143452 -3 panTro1.chr6(+):28037344-28037356|panTro1_2 1.70011e-31 0 0.143452 -3 Node1 1.27931e-17 0 0.143452 -3 mm5.chr6(+):52104047-52104059|mm5_2 1.70011e-31 0 0.143452 -3 rn3.chr4(+):80734420-80734432|rn3_2 0 0 0.143452 -3 Node4 0 0 0.143452 -3 canFam1.chr14(+):42826434-42826446|canFam1_2 1.70011e-31 0 0.143452 -3 Total Tree 1.27931e-17 9.64599e-22 9.4261e-08 -4 hg17.chr7(+):26907338-26907654|hg17_3 1.98974e-31 0 0.00662576 -4 panTro1.chr6(+):28037356-28037672|panTro1_3 1.98974e-31 0 0.00662576 -4 Node1 0.014806 0.00246541 0.0381554 -4 mm5.chr6(+):52104059-52104375|mm5_3 0.0275935 0.0109456 0.0548103 -4 rn3.chr4(+):80734432-80734748|rn3_3 0.0217688 0.00736645 0.0466551 -4 Node4 0.0451597 0.0224663 0.0792997 -4 canFam1.chr14(+):42826446-42826762|canFam1_3 0.0320496 0.013941 0.0605175 -4 Total Tree 0.141378 0.100447 0.192467 +2 hg17.chr7(+):26907310-26907326|hg17_1 1.90313e-31 0 0.13592 +2 panTro1.chr6(+):28037328-28037344|panTro1_1 1.90313e-31 0 0.13592 +2 Node1 4.10285e-19 0 0.13592 +2 mm5.chr6(+):52104031-52104047|mm5_1 2.02063e-19 0 0.13592 +2 rn3.chr4(+):80734404-80734420|rn3_1 0.069102 0.00372954 0.421893 +2 Node4 1.90313e-31 0 0.13592 +2 canFam1.chr14(+):42826418-42826434|canFam1_1 1.13513e-19 0 0.13592 +2 Total Tree 0.069102 0.00372954 0.421893 +3 hg17.chr7(+):26907326-26907338|hg17_2 8.84162e-30 0 0.167446 +3 panTro1.chr6(+):28037344-28037356|panTro1_2 8.84162e-30 0 0.167446 +3 Node1 0 0 0.167446 +3 mm5.chr6(+):52104047-52104059|mm5_2 0 0 0.24858 +3 rn3.chr4(+):80734420-80734432|rn3_2 0.201494 0.0309976 0.7364 +3 Node4 0.31691 0.0691821 1.08477 +3 canFam1.chr14(+):42826434-42826446|canFam1_2 0 0 0.167446 +3 Total Tree 0.518404 0.168855 1.26753 +4 hg17.chr7(+):26907338-26907654|hg17_3 1.11429e-19 0 0.0061265 +4 panTro1.chr6(+):28037356-28037672|panTro1_3 1.11866e-19 0 0.0061265 +4 Node1 0.0583134 0.0318643 0.0951333 +4 mm5.chr6(+):52104059-52104375|mm5_3 0.0357704 0.0167845 0.0644066 +4 rn3.chr4(+):80734432-80734748|rn3_3 0.0320354 0.0141331 0.0594834 +4 Node4 0.0770295 0.0463471 0.11848 +4 canFam1.chr14(+):42826446-42826762|canFam1_3 0.0575307 0.0316345 0.0936592 +4 Total Tree 0.260679 0.205255 0.325684 diff -r 4d4de72c57c8 -r a4214de3752e tools/hyphy/hyphy_branch_lengths_wrapper.xml --- a/tools/hyphy/hyphy_branch_lengths_wrapper.xml Tue Feb 03 15:06:21 2009 -0500 +++ b/tools/hyphy/hyphy_branch_lengths_wrapper.xml Tue Feb 03 18:34:27 2009 -0500 @@ -1,4 +1,4 @@ -<?xml version="1.1.0"?> +<?xml version="1.2.0"?> <tool name="Branch Lengths" id="hyphy_branch_lengths_wrapper1"> <description>Estimation</description> diff -r 4d4de72c57c8 -r a4214de3752e tools/regVariation/getIndels.py --- a/tools/regVariation/getIndels.py Tue Feb 03 15:06:21 2009 -0500 +++ b/tools/regVariation/getIndels.py Tue Feb 03 18:34:27 2009 -0500 @@ -10,8 +10,10 @@ from galaxy import eggs import pkg_resources pkg_resources.require( "bx-python" ) -pkg_resources.require("numpy") - +try: + pkg_resources.require("numpy") +except: + pass import psyco_full import sys from bx.cookbook import doc_optparse diff -r 4d4de72c57c8 -r a4214de3752e tools/regVariation/quality_filter.py --- a/tools/regVariation/quality_filter.py Tue Feb 03 15:06:21 2009 -0500 +++ b/tools/regVariation/quality_filter.py Tue Feb 03 18:34:27 2009 -0500 @@ -12,7 +12,10 @@ import pkg_resources pkg_resources.require( "bx-python" ) pkg_resources.require( "lrucache" ) -pkg_resources.require("numpy") +try: + pkg_resources.require("numpy") +except: + pass import psyco_full import sys

1 0

[hg] galaxy 1731: Stop jobs if any of their output datasets grow...
by Greg Von Kuster 05 Feb '09

05 Feb '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e26741c8c642 changeset: 1731:e26741c8c642 user: Nate Coraor <nate(a)bx.psu.edu> date: Tue Feb 03 14:15:09 2009 -0500 description: Stop jobs if any of their output datasets grow larger than a defined limit 5 file(s) affected in this change: lib/galaxy/config.py lib/galaxy/jobs/__init__.py lib/galaxy/jobs/runners/local.py lib/galaxy/jobs/runners/pbs.py lib/galaxy/tools/__init__.py diffs (309 lines): diff -r 0424f713d58d -r e26741c8c642 lib/galaxy/config.py --- a/lib/galaxy/config.py Tue Feb 03 14:14:02 2009 -0500 +++ b/lib/galaxy/config.py Tue Feb 03 14:15:09 2009 -0500 @@ -47,6 +47,7 @@ self.cluster_files_directory = os.path.abspath( kwargs.get( "cluster_files_directory", "database/pbs" ) ) self.job_working_directory = resolve_path( kwargs.get( "job_working_directory", "database/job_working_directory" ), self.root ) self.outputs_to_working_directory = string_as_bool( kwargs.get( 'outputs_to_working_directory', False ) ) + self.output_size_limit = int( kwargs.get( 'output_size_limit', 0 ) ) self.admin_pass = kwargs.get('admin_pass',"galaxy") self.sendmail_path = kwargs.get('sendmail_path',"/usr/sbin/sendmail") self.mailing_join_addr = kwargs.get('mailing_join_addr',"galaxy-user-join(a)bx.psu.edu") diff -r 0424f713d58d -r e26741c8c642 lib/galaxy/jobs/__init__.py --- a/lib/galaxy/jobs/__init__.py Tue Feb 03 14:14:02 2009 -0500 +++ b/lib/galaxy/jobs/__init__.py Tue Feb 03 14:15:09 2009 -0500 @@ -305,6 +305,7 @@ # and job recovery fail. self.working_directory = \ os.path.join( self.app.config.job_working_directory, str( self.job_id ) ) + self.output_paths = None def get_param_dict( self ): """ @@ -342,7 +343,7 @@ incoming['userId'] = userId incoming['userEmail'] = userEmail # Build params, done before hook so hook can use - param_dict = self.tool.build_param_dict( incoming, inp_data, out_data, self.working_directory ) + param_dict = self.tool.build_param_dict( incoming, inp_data, out_data, self.get_output_fnames() ) # Certain tools require tasks to be completed prior to job execution # ( this used to be performed in the "exec_before_job" hook, but hooks are deprecated ). if self.tool.tool_type is not None: @@ -371,7 +372,7 @@ self.param_dict = param_dict self.extra_filenames = extra_filenames return extra_filenames - + def fail( self, message, exception=False ): """ Indicate job failure by setting state and message on all output @@ -385,14 +386,14 @@ # Do this first in case we generate a traceback below if exception: job.traceback = traceback.format_exc() - for dataset_assoc in job.output_datasets: - if self.app.config.outputs_to_working_directory: - false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.dataset.id ) ) + if self.app.config.outputs_to_working_directory: + for dataset_path in self.get_output_fnames(): try: - shutil.move( false_path, dataset_assoc.dataset.file_name ) - log.debug( "fail(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) + shutil.move( dataset_path.false_path, dataset_path.real_path ) + log.debug( "fail(): Moved %s to %s" % ( dataset_path.false_path, dataset_path.real_path ) ) except ( IOError, OSError ), e: log.error( "fail(): Missing output file in working directory: %s" % e ) + for dataset_assoc in job.output_datasets: dataset = dataset_assoc.dataset dataset.refresh() dataset.state = dataset.states.ERROR @@ -452,15 +453,15 @@ job.state = "error" else: job.state = 'ok' + if self.app.config.outputs_to_working_directory: + for dataset_path in self.get_output_fnames(): + try: + shutil.move( dataset_path.false_path, dataset_path.real_path ) + log.debug( "finish(): Moved %s to %s" % ( dataset_path.false_path, dataset_path.real_path ) ) + except ( IOError, OSError ): + self.fail( "Job %s's output dataset(s) could not be read" % job.id ) + return for dataset_assoc in job.output_datasets: - if self.app.config.outputs_to_working_directory: - false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % dataset_assoc.dataset.dataset.id ) ) - try: - shutil.move( false_path, dataset_assoc.dataset.file_name ) - log.debug( "finish(): Moved %s to %s" % ( false_path, dataset_assoc.dataset.file_name ) ) - except ( IOError, OSError ): - self.fail( "The job's output dataset(s) could not be read" ) - return for dataset in dataset_assoc.dataset.dataset.history_associations: #need to update all associated output hdas, i.e. history was shared with job running dataset.blurb = 'done' dataset.peek = 'no peek' @@ -519,7 +520,7 @@ for fname in self.extra_filenames: os.remove( fname ) if self.working_directory is not None: - os.rmdir( self.working_directory ) + shutil.rmtree( self.working_directory ) except: log.exception( "Unable to cleanup job %d" % self.job_id ) @@ -543,9 +544,36 @@ return filenames def get_output_fnames( self ): + if self.output_paths is not None: + return self.output_paths + + class DatasetPath( object ): + def __init__( self, real_path, false_path = None ): + self.real_path = real_path + self.false_path = false_path + def __str__( self ): + if false_path is None: + return self.real_path + else: + return self.false_path + job = model.Job.get( self.job_id ) - return [ da.dataset.file_name for da in job.output_datasets ] + if self.app.config.outputs_to_working_directory: + self.output_paths = [] + for name, data in [ ( da.name, da.dataset ) for da in job.output_datasets ]: + false_path = os.path.abspath( os.path.join( self.working_directory, "galaxy_dataset_%d.dat" % data.id ) ) + self.output_paths.append( DatasetPath( data.file_name, false_path ) ) + else: + self.output_paths = [ DatasetPath( da.dataset.file_name ) for da in job.output_datasets ] + return self.output_paths + def check_output_sizes( self ): + sizes = [] + output_paths = self.get_output_fnames() + for outfile in [ str( o ) for o in output_paths ]: + sizes.append( ( outfile, os.stat( outfile ).st_size ) ) + return sizes + class DefaultJobDispatcher( object ): def __init__( self, app ): self.app = app diff -r 0424f713d58d -r e26741c8c642 lib/galaxy/jobs/runners/local.py --- a/lib/galaxy/jobs/runners/local.py Tue Feb 03 14:14:02 2009 -0500 +++ b/lib/galaxy/jobs/runners/local.py Tue Feb 03 14:15:09 2009 -0500 @@ -4,6 +4,7 @@ import threading from galaxy import model +from galaxy.datatypes.data import nice_size import os, errno from time import sleep @@ -68,9 +69,31 @@ env = env, preexec_fn = os.setpgrp ) job_wrapper.set_runner( 'local:///', proc.pid ) + if self.app.config.output_size_limit > 0: + sleep_time = 1 + while proc.poll() is None: + for outfile, size in job_wrapper.check_output_sizes(): + if size > self.app.config.output_size_limit: + # Error the job immediately + job_wrapper.fail( 'Job output grew too large (greater than %s), please try different job parameters or' \ + % nice_size( self.app.config.output_size_limit ) ) + log.warning( 'Terminating job %s due to output %s growing larger than %s limit' \ + % ( job_wrapper.job_id, os.path.basename( outfile ), nice_size( self.app.config.output_size_limit ) ) ) + # Then kill it + os.killpg( proc.pid, 15 ) + sleep( 1 ) + if proc.poll() is None: + os.killpg( proc.pid, 9 ) + proc.wait() # reap + log.debug( 'Job %s (pid %s) terminated' % ( job_wrapper.job_id, proc.pid ) ) + return + sleep( sleep_time ) + if sleep_time < 8: + # So we don't stat every second + sleep_time *= 2 stdout = proc.stdout.read() stderr = proc.stderr.read() - proc.wait() + proc.wait() # reap log.debug('execution finished: %s' % command_line) except Exception, exc: job_wrapper.fail( "failure running job", exception=True ) diff -r 0424f713d58d -r e26741c8c642 lib/galaxy/jobs/runners/pbs.py --- a/lib/galaxy/jobs/runners/pbs.py Tue Feb 03 14:14:02 2009 -0500 +++ b/lib/galaxy/jobs/runners/pbs.py Tue Feb 03 14:15:09 2009 -0500 @@ -2,6 +2,8 @@ from Queue import Queue, Empty from galaxy import model +from galaxy.datatypes.data import nice_size + from paste.deploy.converters import asbool import pkg_resources @@ -60,6 +62,7 @@ self.ofile = None self.efile = None self.runner_url = None + self.check_count = 0 class PBSJobRunner( object ): """ @@ -71,6 +74,8 @@ # Check if PBS was importable, fail if not if pbs is None: raise Exception( "PBSJobRunner requires pbs-python which was not found" ) + if app.config.pbs_application_server and app.config.outputs_to_working_directory: + raise Exception( "pbs_application_server (file staging) and outputs_to_working_directory options are mutually exclusive" ) self.app = app # 'watched' and 'queue' are both used to keep track of jobs to watch. # 'queue' is used to add new watched jobs, and can be called from @@ -132,6 +137,8 @@ self.queue_job( obj ) elif op == 'finish': self.finish_job( obj ) + elif op == 'fail_oversize_job': + self.fail_oversize_job( obj ) except: log.exception( "Uncaught exception %sing job" % op ) @@ -173,8 +180,9 @@ if self.app.config.pbs_application_server: pbs_ofile = self.app.config.pbs_application_server + ':' + ofile pbs_efile = self.app.config.pbs_application_server + ':' + efile - stagein = self.get_stage_in_out( job_wrapper.get_input_fnames() + job_wrapper.get_output_fnames(), symlink=True ) - stageout = self.get_stage_in_out( job_wrapper.get_output_fnames() ) + output_files = [ str( o ) for o in job_wrapper.get_output_fnames() ] + stagein = self.get_stage_in_out( job_wrapper.get_input_fnames() + output_files, symlink=True ) + stageout = self.get_stage_in_out( output_files ) job_attrs = pbs.new_attropl(5) job_attrs[0].name = pbs.ATTR_o job_attrs[0].value = pbs_ofile @@ -298,6 +306,20 @@ if state == "R" and not pbs_job_state.running: pbs_job_state.running = True pbs_job_state.job_wrapper.change_state( "running" ) + if self.app.config.output_size_limit > 0 and state == "R" and (pbs_job_state.check_count % 10) == 0: + # Every 10th time a job is checked, check the size of its outputs. + fail = False + for outfile, size in pbs_job_state.job_wrapper.check_output_sizes(): + if size > self.app.config.output_size_limit: + pbs_job_state.fail_message = 'Job output grew too large (greater than %s), please try different job parameters or' \ + % nice_size( self.app.config.output_size_limit ) + log.warning( '(%s/%s) Dequeueing job due to output %s growing larger than %s limit' \ + % ( galaxy_job_id, job_id, os.path.basename( outfile ), nice_size( self.app.config.output_size_limit ) ) ) + self.work_queue.put( ( 'fail_oversize_job', pbs_job_state ) ) + fail = True + break + if fail: + continue pbs_job_state.old_state = state new_watched.append( pbs_job_state ) else: @@ -329,6 +351,7 @@ pbs_server_name = self.determine_pbs_server( pbs_job_state.runner_url ) if pbs_server_name not in servers: servers.append( pbs_server_name ) + pbs_job_state.check_count += 1 for pbs_server_name in servers: c = pbs.pbs_connect( pbs_server_name ) if c <= 0: @@ -386,6 +409,14 @@ # clean up the pbs files self.cleanup( ( ofile, efile, job_file ) ) + def fail_oversize_job( self, pbs_job_state ): + """ + Seperated out so we can use the worker threads for it. + """ + self.stop_job( self.app.model.Job.get( pbs_job_state.job_wrapper.job_id ) ) + pbs_job_state.job_wrapper.fail( pbs_job_state.fail_message ) + self.cleanup( ( pbs_job_state.ofile, pbs_job_state.efile, pbs_job_state.job_file ) ) + def cleanup( self, files ): if not asbool( self.app.config.get( 'debug', False ) ): for file in files: @@ -431,7 +462,7 @@ return pbs.pbs_deljob( c, str( job.job_runner_external_id ), 'NULL' ) pbs.pbs_disconnect( c ) - log.debug( "(%s/%s) Removed from PBS queue at user's request" % ( job.id, job.job_runner_external_id ) ) + log.debug( "(%s/%s) Removed from PBS queue before job completion" % ( job.id, job.job_runner_external_id ) ) def recover( self, job, job_wrapper ): """Recovers jobs stuck in the queued/running state when Galaxy started""" diff -r 0424f713d58d -r e26741c8c642 lib/galaxy/tools/__init__.py --- a/lib/galaxy/tools/__init__.py Tue Feb 03 14:14:02 2009 -0500 +++ b/lib/galaxy/tools/__init__.py Tue Feb 03 14:15:09 2009 -0500 @@ -1045,7 +1045,7 @@ input.validate( value, None ) input_values[ input.name ] = value - def build_param_dict( self, incoming, input_datasets, output_datasets, working_directory = None ): + def build_param_dict( self, incoming, input_datasets, output_datasets, output_paths ): """ Build the dictionary of parameters for substituting into the command line. Each value is wrapped in a `InputValueWrapper`, which allows @@ -1098,10 +1098,14 @@ param_dict[ "_CHILD___%s___%s" % ( name, child.designation ) ] = DatasetFilenameWrapper( child ) for name, data in output_datasets.items(): # Write outputs to the working directory (for security purposes) if desired. - if self.app.config.outputs_to_working_directory and working_directory is not None: - false_path = os.path.abspath( os.path.join( working_directory, "galaxy_dataset_%d.dat" % data.dataset.id ) ) - param_dict[name] = DatasetFilenameWrapper( data, false_path = false_path ) - open( false_path, 'w' ).close() + if self.app.config.outputs_to_working_directory: + try: + false_path = [ dp.false_path for dp in output_paths if dp.real_path == data.file_name ][0] + param_dict[name] = DatasetFilenameWrapper( data, false_path = false_path ) + open( false_path, 'w' ).close() + except IndexError: + log.warning( "Unable to determine alternate path for writing job outputs, outputs will be written to their real paths" ) + param_dict[name] = DatasetFilenameWrapper( data ) else: param_dict[name] = DatasetFilenameWrapper( data ) # Provide access to a path to store additional files

1 0

[hg] galaxy 1729: Corrected a bug in branch lengths batchfile, a...
by Greg Von Kuster 03 Feb '09

03 Feb '09

details: http://www.bx.psu.edu/hg/galaxy/rev/743237dd0e65 changeset: 1729:743237dd0e65 user: guru date: Mon Feb 02 18:57:07 2009 -0500 description: Corrected a bug in branch lengths batchfile, as per Sergei's suggestions. 1 file(s) affected in this change: lib/galaxy/tools/util/hyphy_util.py diffs (306 lines): diff -r cabfdc9dea12 -r 743237dd0e65 lib/galaxy/tools/util/hyphy_util.py --- a/lib/galaxy/tools/util/hyphy_util.py Fri Jan 30 11:37:51 2009 -0500 +++ b/lib/galaxy/tools/util/hyphy_util.py Mon Feb 02 18:57:07 2009 -0500 @@ -368,143 +368,281 @@ BranchLengthsMF = """ VERBOSITY_LEVEL = -1; + fscanf (PROMPT_FOR_FILE, "Lines", inLines); + + _linesIn = Columns (inLines); + + /*---------------------------------------------------------*/ + + _currentGene = 1; + _currentState = 0; + geneSeqs = ""; + geneSeqs * 128; + + for (l=0; l<_linesIn; l=l+1) + { + if (Abs(inLines[l]) == 0) + { + if (_currentState == 1) + { + geneSeqs * 0; + DataSet ds = ReadFromString (geneSeqs); + _processAGene (_currentGene); - geneSeqs * 128; + + geneSeqs * 128; + _currentGene = _currentGene + 1; + } + } + else + { + if (_currentState == 0) + { + _currentState = 1; + } + geneSeqs * inLines[l]; + geneSeqs * "\\n"; + } + } + + if (_currentState == 1) + { + geneSeqs * 0; + if (Abs(geneSeqs)) + { + DataSet ds = ReadFromString (geneSeqs); + _processAGene (_currentGene); + } + } + + fprintf (resultFile,CLOSE_FILE); + + /*---------------------------------------------------------*/ + + function _processAGene (_geneID) + { + DataSetFilter filteredData = CreateFilter (ds,1); + if (_currentGene == 1) + { + SelectTemplateModel (filteredData); + - SetDialogPrompt ("Tree file"); + + SetDialogPrompt ("Tree file"); + fscanf (PROMPT_FOR_FILE, "Tree", givenTree); + fscanf (stdin, "String", resultFile); + + /* do sequence to branch map */ + + validNames = {}; + taxonNameMap = {}; + + for (k=0; k<TipCount(givenTree); k=k+1) + { + validNames[TipName(givenTree,k)&&1] = 1; + } + + for (k=0; k<BranchCount(givenTree); k=k+1) + { + thisName = BranchName(givenTree,k); + taxonNameMap[thisName&&1] = thisName; + } + + storeValidNames = validNames; + fprintf (resultFile,CLEAR_FILE,KEEP_OPEN,"Block\\tBranch\\tLength\\tLowerBound\\tUpperBound\\n"); + } + else + { + + HarvestFrequencies (vectorOfFrequencies, filteredData, 1,1,1); + validNames = storeValidNames; + } + + for (k=0; k<ds.species; k=k+1) + { + GetString (thisName, ds,k); + shortName = (thisName^{{"\\\\..+",""}})&&1; + if (validNames[shortName]) + { + taxonNameMap[shortName] = thisName; + validNames - (shortName); + SetParameter (ds,k,shortName); + } + else + { + fprintf (resultFile,"ERROR:", thisName, " could not be matched to any of the leaves in tree ", givenTree,"\\n"); + return 0; + } + } + + /* */ + + LikelihoodFunction lf = (filteredData,givenTree); + + Optimize (res,lf); + - Optimize (res,lf); + + timer = Time(0)-timer; + - timer = Time(0)-timer; + + branchNames = BranchName (givenTree,-1); + + branchLengths = BranchLength (givenTree,-1); + - branchNames = BranchName (givenTree,-1); - branchLengths = BranchLength (givenTree,-1); + + + for (k=0; k<Columns(branchNames)-1; k=k+1) + + { + + COVARIANCE_PARAMETER = "givenTree."+branchNames[k]+".t"; + + COVARIANCE_PRECISION = 0.95; + + CovarianceMatrix (cmx,lf); + + if (k==0) + + { + + /* compute a scaling factor */ + + ExecuteCommands ("givenTree."+branchNames[0]+".t=1"); + + scaleFactor = BranchLength (givenTree,0); + + ExecuteCommands ("givenTree."+branchNames[0]+".t="+cmx[0][1]); + + } + + fprintf (resultFile,_geneID,"\\t",taxonNameMap[branchNames[k]&&1],"\\t",branchLengths[k],"\\t",scaleFactor*cmx[0][0],"\\t",scaleFactor*cmx[0][2],"\\n"); + + } + - for (k=0; k<Columns(branchNames)-1; k=k+1) - { - COVARIANCE_PARAMETER = "givenTree."+branchNames[k]+".t"; - COVARIANCE_PRECISION = 0.95; - CovarianceMatrix (cmx,lf); - if (k==0) - { - /* compute a scaling factor */ - ExecuteCommands ("givenTree."+branchNames[0]+".t=1"); - scaleFactor = BranchLength (givenTree,0); - ExecuteCommands ("givenTree."+branchNames[0]+".t="+cmx[0][1]); - } - fprintf (resultFile,_geneID,"\\t",taxonNameMap[branchNames[k]&&1],"\\t",branchLengths[k],"\\t",scaleFactor*cmx[0][0],"\\t",scaleFactor*cmx[0][2],"\\n"); - } - + ttl = (branchLengths*(Transpose(branchLengths["1"])))[0]; + global treeScaler = 1; + ReplicateConstraint ("this1.?.t:=treeScaler*this2.?.t__",givenTree,givenTree); + COVARIANCE_PARAMETER = "treeScaler"; + COVARIANCE_PRECISION = 0.95; + CovarianceMatrix (cmx,lf); + fprintf (resultFile,_geneID,"\\tTotal Tree\\t",ttl,"\\t",ttl*cmx[0][0],"\\t",ttl*cmx[0][2],"\\n"); + ClearConstraints (givenTree); + return 0; + } """

1 0

Heatmap plotting tool
by Assaf Gordon 02 Feb '09

02 Feb '09

Hello, I'd like to offer a heatmap plotting tool. Similar to the XY-Plotting tool (and even uses the same R_wrapper.sh script), but generates a heatmap with user customizable colors. To run this tool (assuming XY-Plotting tool already works), you'll need to install the "gplots" package, as explained here: http://enotacoes.wordpress.com/2007/11/16/easy-guide-to-drawing-heat-maps-t… Comments are welcomed, Gordon.

1 0

[hg] galaxy 1727: Add fetch_eggs option to pull for other platfo...
by Nate Coraor 30 Jan '09

30 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e1e119b0bd17 changeset: 1727:e1e119b0bd17 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Jan 30 11:37:18 2009 -0500 description: Add fetch_eggs option to pull for other platforms (e.g. cluster nodes) 2 file(s) affected in this change: scripts/fetch_eggs.py scripts/get_platforms.py diffs (47 lines): diff -r 9a06f9910524 -r e1e119b0bd17 scripts/fetch_eggs.py --- a/scripts/fetch_eggs.py Tue Jan 27 17:12:04 2009 -0500 +++ b/scripts/fetch_eggs.py Fri Jan 30 11:37:18 2009 -0500 @@ -1,9 +1,13 @@ """ -usage: fetch_eggs.py [egg_name] +usage: fetch_eggs.py [egg_name] [platform] With no arguments, fetches all eggs necessary according to the settings in universe_wsgi.ini. egg_name - Fetch only this egg (as defined in eggs.ini) or 'all' for all eggs (even those not required by your settings). + platform - Fetch eggs for a specific platform (if not provided, fetch + eggs for *this* platform). Useful for fetching eggs for cluster + nodes which are of a different architecture than the head node. + Platform name can be determined with the get_platforms.py script. """ import os, sys, logging @@ -17,6 +21,8 @@ from galaxy.eggs import * c = Crate() +if len( sys.argv ) == 3: + c.platform = { 'peak' : sys.argv[2].rsplit('-',1)[0], 'galaxy' : sys.argv[2] } c.parse() try: if len( sys.argv ) == 1: diff -r 9a06f9910524 -r e1e119b0bd17 scripts/get_platforms.py --- a/scripts/get_platforms.py Tue Jan 27 17:12:04 2009 -0500 +++ b/scripts/get_platforms.py Fri Jan 30 11:37:18 2009 -0500 @@ -1,9 +1,12 @@ #!/usr/bin/env python -import sys +import os, sys assert sys.version_info[:2] >= ( 2, 4 ) -from eggs import get_full_platform, get_noplatform -print get_noplatform() -print get_full_platform() +lib = os.path.abspath( os.path.join( os.path.dirname( __file__ ), "..", "lib" ) ) +sys.path.append( lib ) + +from galaxy.eggs import get_platform +print get_platform() +print get_platform( True )

1 0

[hg] galaxy 1728: Fix annotation profiler interpreter to be pyth...
by Nate Coraor 30 Jan '09

30 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/cabfdc9dea12 changeset: 1728:cabfdc9dea12 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Jan 30 11:37:51 2009 -0500 description: Fix annotation profiler interpreter to be python, not python2.4 1 file(s) affected in this change: tools/annotation_profiler/annotation_profiler.xml diffs (11 lines): diff -r e1e119b0bd17 -r cabfdc9dea12 tools/annotation_profiler/annotation_profiler.xml --- a/tools/annotation_profiler/annotation_profiler.xml Fri Jan 30 11:37:18 2009 -0500 +++ b/tools/annotation_profiler/annotation_profiler.xml Fri Jan 30 11:37:51 2009 -0500 @@ -1,6 +1,6 @@ <tool id="Annotation_Profiler_0" name="Profile Annotations" Version="1.0.0"> <description>for a set of genomic intervals</description> - <command interpreter="python2.4">annotation_profiler_for_interval.py -i $input1 -c ${input1.metadata.chromCol} -s ${input1.metadata.startCol} -e ${input1.metadata.endCol} -o $out_file1 $keep_empty -p /depot/data2/galaxy/annotation_profiler/$dbkey $summary -l ${GALAXY_DATA_INDEX_DIR}/shared/ucsc/chrom/${dbkey}.len -b 3 -t $table_names</command> + <command interpreter="python">annotation_profiler_for_interval.py -i $input1 -c ${input1.metadata.chromCol} -s ${input1.metadata.startCol} -e ${input1.metadata.endCol} -o $out_file1 $keep_empty -p /depot/data2/galaxy/annotation_profiler/$dbkey $summary -l ${GALAXY_DATA_INDEX_DIR}/shared/ucsc/chrom/${dbkey}.len -b 3 -t $table_names</command> <inputs> <param format="interval" name="input1" type="data" label="Choose Intervals"> <validator type="dataset_metadata_in_file" filename="annotation_profiler_valid_builds.txt" metadata_name="dbkey" metadata_column="0" message="Profiling is not currently available for this species."/>

1 0

[hg] galaxy 1726: More fixes to eggs.require()
by Nate Coraor 27 Jan '09

27 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/9a06f9910524 changeset: 1726:9a06f9910524 user: Nate Coraor <nate(a)bx.psu.edu> date: Tue Jan 27 17:12:04 2009 -0500 description: More fixes to eggs.require() 1 file(s) affected in this change: lib/galaxy/eggs/__init__.py diffs (35 lines): diff -r 7411b119a3c9 -r 9a06f9910524 lib/galaxy/eggs/__init__.py --- a/lib/galaxy/eggs/__init__.py Tue Jan 27 16:17:22 2009 -0500 +++ b/lib/galaxy/eggs/__init__.py Tue Jan 27 17:12:04 2009 -0500 @@ -527,7 +527,6 @@ pkg_resources.working_set.require( pkg ) else: pkg_resources.working_set.require( "%s==%s" % ( name, egg.get_vertag() ) ) - return except pkg_resources.VersionConflict, e: # there's a conflicting egg on the pythonpath, remove it dist = e.args[0] @@ -539,10 +538,13 @@ working_set.entries.remove( entry ) break else: - raise # some path weirdness has prevented us from finding the offender + location = None del working_set.by_key[dist.key] working_set.entry_keys[entry] = [] sys.path.remove(entry) + require( pkg ) + if location is not None and not location.endswith( '.egg' ): + working_set.entries.append( location ) # re-add to the set if it's a dir. except pkg_resources.DistributionNotFound, e: # the initial require itself is the first dep, but it can have # multiple deps, which will be fetched by the require below. @@ -555,7 +557,7 @@ if not egg.have: if not egg.fetch(): raise EggNotFetchable( egg.name ) - require( pkg ) + require( pkg ) # convenience stuff def get_ucs():

1 0

[hg] galaxy 1725: Adding modified hyphy functional test output f...
by gua110＠scofield.bx.psu.edu 27 Jan '09

27 Jan '09

details: http://www.bx.psu.edu/hg/galaxy/rev/7411b119a3c9 changeset: 1725:7411b119a3c9 user: guru date: Tue Jan 27 16:17:22 2009 -0500 description: Adding modified hyphy functional test output files, which have changed as the new version of HYPHY is installed on Galaxy. Also changing version numbers for all the affected tools. 8 file(s) affected in this change: test-data/branchlength_out.dat test-data/dnds_out.tabular test-data/nj_tree_newick_out.dat tools/hyphy/hyphy_branch_lengths_wrapper.xml tools/hyphy/hyphy_dnds_wrapper.xml tools/hyphy/hyphy_nj_tree_wrapper.xml tools/regVariation/microsats_mutability.xml tools/stats/grouping.xml diffs (140 lines): diff -r 546e2d835e16 -r 7411b119a3c9 test-data/branchlength_out.dat --- a/test-data/branchlength_out.dat Tue Jan 27 14:48:51 2009 -0500 +++ b/test-data/branchlength_out.dat Tue Jan 27 16:17:22 2009 -0500 @@ -9,25 +9,25 @@ 1 Total Tree 3.45679e-30 2.60643e-34 2.54701e-20 2 hg17.chr7(+):26907310-26907326|hg17_1 1.48148e-31 0 0.0915385 2 panTro1.chr6(+):28037328-28037344|panTro1_1 1.48148e-31 0 0.0915385 -2 Node1 1.48148e-31 0 0.0915385 -2 mm5.chr6(+):52104031-52104047|mm5_1 1.48148e-31 0 0.0915385 -2 rn3.chr4(+):80734404-80734420|rn3_1 0.0435496 0.00241411 0.219651 +2 Node1 6.69055e-19 0 0.0915385 +2 mm5.chr6(+):52104031-52104047|mm5_1 5.00368e-19 0 0.0915385 +2 rn3.chr4(+):80734404-80734420|rn3_1 0.0435526 0.00241411 0.219651 2 Node4 1.48148e-31 0 0.0915385 -2 canFam1.chr14(+):42826418-42826434|canFam1_1 1.48148e-31 0 0.0915385 -2 Total Tree 0.0435496 0.00241411 0.219655 +2 canFam1.chr14(+):42826418-42826434|canFam1_1 1.76058e-19 0 0.0915385 +2 Total Tree 0.0435526 0.00241411 0.219651 3 hg17.chr7(+):26907326-26907338|hg17_2 1.70011e-31 0 0.143452 3 panTro1.chr6(+):28037344-28037356|panTro1_2 1.70011e-31 0 0.143452 -3 Node1 1.70011e-31 0 0.143452 +3 Node1 1.27931e-17 0 0.143452 3 mm5.chr6(+):52104047-52104059|mm5_2 1.70011e-31 0 0.143452 3 rn3.chr4(+):80734420-80734432|rn3_2 0 0 0.143452 -3 Node4 1.70011e-31 0 0.143452 +3 Node4 0 0 0.143452 3 canFam1.chr14(+):42826434-42826446|canFam1_2 1.70011e-31 0 0.143452 -3 Total Tree 1.02006e-30 7.69131e-35 7.51597e-21 -4 hg17.chr7(+):26907338-26907654|hg17_3 1.98983e-31 0 0.0066257 -4 panTro1.chr6(+):28037356-28037672|panTro1_3 1.98983e-31 0 0.0066257 -4 Node1 0.0148072 0.00246575 0.0381554 -4 mm5.chr6(+):52104059-52104375|mm5_3 0.0275943 0.0109457 0.0548099 -4 rn3.chr4(+):80734432-80734748|rn3_3 0.0217662 0.00736615 0.0466518 -4 Node4 0.045162 0.0224707 0.0792981 -4 canFam1.chr14(+):42826446-42826762|canFam1_3 0.0320463 0.0139406 0.0605161 -4 Total Tree 0.141376 0.100446 0.192464 +3 Total Tree 1.27931e-17 9.64599e-22 9.4261e-08 +4 hg17.chr7(+):26907338-26907654|hg17_3 1.98974e-31 0 0.00662576 +4 panTro1.chr6(+):28037356-28037672|panTro1_3 1.98974e-31 0 0.00662576 +4 Node1 0.014806 0.00246541 0.0381554 +4 mm5.chr6(+):52104059-52104375|mm5_3 0.0275935 0.0109456 0.0548103 +4 rn3.chr4(+):80734432-80734748|rn3_3 0.0217688 0.00736645 0.0466551 +4 Node4 0.0451597 0.0224663 0.0792997 +4 canFam1.chr14(+):42826446-42826762|canFam1_3 0.0320496 0.013941 0.0605175 +4 Total Tree 0.141378 0.100447 0.192467 diff -r 546e2d835e16 -r 7411b119a3c9 test-data/dnds_out.tabular --- a/test-data/dnds_out.tabular Tue Jan 27 14:48:51 2009 -0500 +++ b/test-data/dnds_out.tabular Tue Jan 27 16:17:22 2009 -0500 @@ -1,12 +1,12 @@ BLOCK BP S_sites NS_sites Stop_codons LogL omega omega_range AC AT CG CT GT Tree -1 1554 355.669 1129.12 1 -2747.41 0.101159 0.0765061-0.130636 1 1 1 1 1 (human:0,chimp:0.00240762,mouse:0.146649) -2 3795 924.935 2749.35 1 -5781.01 0.0179485 0.0119925-0.02562 1 1 1 1 1 (human:0.0004305,chimp:0.00147714,mouse:0.144943) -3 1194 272.323 881.914 1 -2043.72 0.0578339 0.0372207-0.0849736 1 1 1 1 1 (human:0.00285309,chimp:0.00410455,mouse:0.11988) -4 1146 248.261 850.364 1 -1871.67 0.012771 0.00548036-0.0247371 1 1 1 1 1 (human:0,chimp:0.00093192,mouse:0.147209) -5 1500 366.374 1086.09 2 -2522.97 0.0331894 0.0203743-0.0505255 1 1 1 1 1 (human:0,chimp:0.00225976,mouse:0.156089) -6 3444 824.107 2475.54 1 -5459.18 0.0439755 0.0326426-0.0576899 1 1 1 1 1 (human:0.000512784,chimp:0.00203485,mouse:0.126653) -7 2175 505.186 1587.7 0 -3307.52 0.0712367 0.0564019-0.088443 1 1 1 1 1 (human:0.00229486,chimp:0.00236745,mouse:0.226418) -8 3519 792.702 2601.36 0 -6219.39 0.0669992 0.0568977-0.078214 1 1 1 1 1 (human:0.00103092,chimp:0.00431199,mouse:0.251293) -9 501 100.22 376.039 1 -834.329 0.13041 0.0835498-0.192248 1 1 1 1 1 (human:0,chimp:0,mouse:0.150462) -10 633 160.252 453.228 1 -618.997 0.0150887 0.00250412-0.0467112 1 1 1 1 1 (human:0,chimp:0.00611143,mouse:0.104315) -11 915 229.705 658.82 0 -1664.02 0.299737 0.227826-0.385548 1 1 1 1 1 (human:0.0045721,chimp:0,mouse:0.145563) +1 1554 355.669 1129.12 1 -2747.41 0.101087 0.0764772-0.130598 1 1 1 1 1 (human:0,chimp:0.0024086,mouse:0.146678) +2 3795 924.935 2749.35 1 -5781.01 0.0179487 0.0119926-0.0256204 1 1 1 1 1 (human:0.000420365,chimp:0.00148429,mouse:0.144946) +3 1194 272.323 881.914 1 -2043.72 0.057806 0.0372158-0.0849625 1 1 1 1 1 (human:0.00284622,chimp:0.0041096,mouse:0.119889) +4 1146 248.261 850.364 1 -1871.67 0.0127683 0.00548023-0.0247365 1 1 1 1 1 (human:1.64845e-31,chimp:0.000931909,mouse:0.147211) +5 1500 366.374 1086.09 2 -2522.97 0.0331759 0.0203729-0.0505178 1 1 1 1 1 (human:0,chimp:0.00225973,mouse:0.156094) +6 3444 824.107 2475.54 1 -5459.18 0.0439729 0.032642-0.0576888 1 1 1 1 1 (human:0.000513623,chimp:0.00203456,mouse:0.126654) +7 2175 505.186 1587.7 0 -3307.52 0.0711481 0.0563562-0.0883841 1 1 1 1 1 (human:0.0022939,chimp:0.00236854,mouse:0.226516) +8 3519 792.702 2601.36 0 -6219.39 0.0669805 0.0568888-0.0782017 1 1 1 1 1 (human:0.00102641,chimp:0.00431646,mouse:0.25132) +9 501 100.22 376.039 1 -834.329 0.130019 0.0833975-0.191896 1 1 1 1 1 (human:0,chimp:2.19778e-31,mouse:0.150592) +10 633 160.252 453.228 1 -618.997 0.0150806 0.00250396-0.0467057 1 1 1 1 1 (human:0,chimp:0.00611203,mouse:0.104319) +11 915 229.705 658.82 0 -1664.02 0.299406 0.227634-0.385273 1 1 1 1 1 (human:0.00457505,chimp:3.44909e-31,mouse:0.145589) diff -r 546e2d835e16 -r 7411b119a3c9 test-data/nj_tree_newick_out.dat --- a/test-data/nj_tree_newick_out.dat Tue Jan 27 14:48:51 2009 -0500 +++ b/test-data/nj_tree_newick_out.dat Tue Jan 27 16:17:22 2009 -0500 @@ -1,10 +1,10 @@ -0 ((hg17,panTro1),rheMac2,(rn3,mm7)) ((hg17.chr7(+):127471195-127471526|hg17_0:0.00346971,panTro1.chr6(+):129885076-129885407|panTro1_0:0.00260712):0.0126114,rheMac2.chr3(+):165787989-165788319|rheMac2_0:0.028188,(rn3.chr4(+):56178191-56178473|rn3_0:0.0617996,mm7.chr6(+):28984529-28984886|mm7_0:0.0662353):0.305784) +0 (hg17,panTro1,(rheMac2,(rn3,mm7))) (hg17.chr7(+):127471195-127471526|hg17_0:0.00283776,panTro1.chr6(+):129885076-129885407|panTro1_0:0.00323907,(rheMac2.chr3(+):165787989-165788319|rheMac2_0:0.0324438,(rn3.chr4(+):56178191-56178473|rn3_0:0.0264183,mm7.chr6(+):28984529-28984886|mm7_0:0.148227):0.457076):0.00849339) 1 ((hg17,panTro1),mm7,rheMac2) ((hg17.chr7(+):127471526-127471584|hg17_1:0.00446699,panTro1.chr6(+):129885407-129885465|panTro1_1:0.013401):0.00483543,mm7.chr6(+):28984886-28984940|mm7_1:999.986,rheMac2.chr3(+):165788319-165788377|rheMac2_1:0.0137694) -2 (((hg17,panTro1),rheMac2),(canFam2,dasNov1),(bosTau2,rn3)) (((hg17.chr7(+):127471584-127471688|hg17_2:0.00234609,panTro1.chr6(+):129885465-129885569|panTro1_2:0.0173725):0.0101799,rheMac2.chr3(+):165788377-165788482|rheMac2_2:0.063065):0.106752,(canFam2.chr14(-):11090703-11090811|canFam2_2:0.187046,dasNov1.scaffold_256527(+):298-392|dasNov1_2:0.0930379):0.0263477,(bosTau2.chr4(-):50243931-50244034|bosTau2_2:0.154553,rn3.chr4(+):56182200-56182295|rn3_2:0.21476):0.0231563) -3 ((hg17,panTro1),rheMac2,(rn3,((bosTau2,dasNov1),canFam2))) ((hg17.chr7(+):127471688-127471871|hg17_3:0.0186952,panTro1.chr6(+):129885569-129885752|panTro1_3:0.00389261):0.024721,rheMac2.chr3(+):165788482-165788684|rheMac2_3:0.0326957,(rn3.chr4(+):56182295-56182489|rn3_3:0.151674,((bosTau2.chr4(-):50243792-50243930|bosTau2_3:0.0510929,dasNov1.scaffold_256527(+):393-625|dasNov1_3:0.119004):0.0284421,canFam2.chr14(-):11090345-11090505|canFam2_3:0.0489193):0.0426697):0.0457765) -4 ((hg17,rheMac2),panTro1,((bosTau2,canFam2),dasNov1)) ((hg17.chr7(+):127471871-127471910|hg17_4:0,rheMac2.chr3(+):165788684-165788723|rheMac2_4:0.0586327):0.0163653,panTro1.chr6(+):129885752-129885791|panTro1_4:0,((bosTau2.chr4(-):50243734-50243773|bosTau2_4:0.462227,canFam2.chr14(-):11090081-11090120|canFam2_4:0.199596):0.112898,dasNov1.scaffold_256527(+):625-665|dasNov1_4:0.160286):0.163376) -5 (((hg17,panTro1),rheMac2),((mm7,rn3),dasNov1),(bosTau2,canFam2)) (((hg17.chr7(+):127471910-127472074|hg17_5:0,panTro1.chr6(+):129885791-129885955|panTro1_5:0.0185501):0.0510514,rheMac2.chr3(+):165788723-165788885|rheMac2_5:0.0284199):0.105274,((mm7.chr6(+):28990714-28990875|mm7_5:0.110927,rn3.chr4(+):56183448-56183705|rn3_5:0):0.18866,dasNov1.scaffold_256527(+):665-786|dasNov1_5:0.1964):0.0403849,(bosTau2.chr4(-):50243566-50243734|bosTau2_5:0.169804,canFam2.chr14(-):11089913-11090081|canFam2_5:0.135334):0.0416476) -6 (hg17,panTro1,(rheMac2,((mm7,rn3),(canFam2,dasNov1)))) (hg17.chr7(+):127472074-127472258|hg17_6:0.00678011,panTro1.chr6(+):129885955-129886139|panTro1_6:0,(rheMac2.chr3(+):165788885-165789069|rheMac2_6:0.0438345,((mm7.chr6(+):28990875-28991025|mm7_6:0.0290692,rn3.chr4(+):56183705-56183879|rn3_6:0.13263):0.168495,(canFam2.chr14(-):11089743-11089913|canFam2_6:0.138698,dasNov1.scaffold_256527(+):786-964|dasNov1_6:0.1994):0.0351372):0.128281):0.0237393) -7 (((hg17,(canFam2,dasNov1)),panTro1),rheMac2,(mm7,rn3)) (((hg17.chr7(+):127472258-127472280|hg17_7:0,(canFam2.chr14(-):11089526-11089548|canFam2_7:0.513656,dasNov1.scaffold_256527(+):964-987|dasNov1_7:0.205512):0.34336):0,panTro1.chr6(+):129886139-129886161|panTro1_7:0):0,rheMac2.chr3(+):165789069-165789091|rheMac2_7:0,(mm7.chr6(+):28991025-28991048|mm7_7:0.0744383,rn3.chr4(+):56183879-56183902|rn3_7:0.0195258):125.2) -8 (((hg17,panTro1),rheMac2),(rn3,canFam2),dasNov1) (((hg17.chr7(+):127472280-127472681|hg17_8:0.00751116,panTro1.chr6(+):129886161-129886562|panTro1_8:2.84007e-05):0.0348269,rheMac2.chr3(+):165789091-165789492|rheMac2_8:0.0280716):0.115196,(rn3.chr4(+):56183902-56184219|rn3_8:0.16879,canFam2.chr14(-):11089143-11089523|canFam2_8:0.158484):0.0249651,dasNov1.scaffold_256527(+):987-1401|dasNov1_8:0.21816) -9 (((hg17,panTro1),canFam2),rheMac2,dasNov1) (((hg17.chr7(+):127472681-127472715|hg17_9:0.0115551,panTro1.chr6(+):129886562-129886596|panTro1_9:0.0221032):0.044958,canFam2.chr14(-):11089108-11089143|canFam2_9:0.218642):0.030703,rheMac2.chr3(+):165789492-165789526|rheMac2_9:0.00773783,dasNov1.scaffold_256527(+):1401-1433|dasNov1_9:0.113241) +2 ((((hg17,panTro1),rheMac2),(canFam2,dasNov1)),bosTau2,rn3) ((((hg17.chr7(+):127471584-127471688|hg17_2:0.00103752,panTro1.chr6(+):129885465-129885569|panTro1_2:0.018681):0.00493709,rheMac2.chr3(+):165788377-165788482|rheMac2_2:0.0691856):0.106183,(canFam2.chr14(-):11090703-11090811|canFam2_2:0.213381,dasNov1.scaffold_256527(+):298-392|dasNov1_2:0.123834):0.0305914):0.028679,bosTau2.chr4(-):50243931-50244034|bosTau2_2:0.170682,rn3.chr4(+):56182200-56182295|rn3_2:0.285956) +3 ((hg17,panTro1),rheMac2,(rn3,(bosTau2,(canFam2,dasNov1)))) ((hg17.chr7(+):127471688-127471871|hg17_3:0.0225878,panTro1.chr6(+):129885569-129885752|panTro1_3:0):0.0221216,rheMac2.chr3(+):165788482-165788684|rheMac2_3:0.0429855,(rn3.chr4(+):56182295-56182489|rn3_3:0.286513,(bosTau2.chr4(-):50243792-50243930|bosTau2_3:0.135108,(canFam2.chr14(-):11090345-11090505|canFam2_3:0.0418005,dasNov1.scaffold_256527(+):393-625|dasNov1_3:0.198071):0.0128256):0.0852898):0.0472209) +4 ((hg17,rheMac2),panTro1,((bosTau2,canFam2),dasNov1)) ((hg17.chr7(+):127471871-127471910|hg17_4:0,rheMac2.chr3(+):165788684-165788723|rheMac2_4:0.0586327):0.0166268,panTro1.chr6(+):129885752-129885791|panTro1_4:0,((bosTau2.chr4(-):50243734-50243773|bosTau2_4:0.462377,canFam2.chr14(-):11090081-11090120|canFam2_4:0.199446):0.119837,dasNov1.scaffold_256527(+):625-665|dasNov1_4:0.17977):0.172715) +5 (((hg17,panTro1),rheMac2),(mm7,rn3),((bosTau2,dasNov1),canFam2)) (((hg17.chr7(+):127471910-127472074|hg17_5:0.000139728,panTro1.chr6(+):129885791-129885955|panTro1_5:0.0184104):0.05206,rheMac2.chr3(+):165788723-165788885|rheMac2_5:0.0285966):0.0250925,(mm7.chr6(+):28990714-28990875|mm7_5:0.123289,rn3.chr4(+):56183448-56183705|rn3_5:0.0718529):0.38361,((bosTau2.chr4(-):50243566-50243734|bosTau2_5:0.181878,dasNov1.scaffold_256527(+):665-786|dasNov1_5:0.429124):0.0618692,canFam2.chr14(-):11089913-11090081|canFam2_5:0.133548):0.11305) +6 ((hg17,panTro1),rheMac2,((mm7,rn3),(canFam2,dasNov1))) ((hg17.chr7(+):127472074-127472258|hg17_6:0.00549431,panTro1.chr6(+):129885955-129886139|panTro1_6:0):0.0241181,rheMac2.chr3(+):165788885-165789069|rheMac2_6:0.0447415,((mm7.chr6(+):28990875-28991025|mm7_6:0.0851965,rn3.chr4(+):56183705-56183879|rn3_6:0.109546):0.319094,(canFam2.chr14(-):11089743-11089913|canFam2_6:0.186816,dasNov1.scaffold_256527(+):786-964|dasNov1_6:0.219003):0.0699235):0.0966238) +7 (((hg17,(canFam2,dasNov1)),panTro1),rheMac2,(mm7,rn3)) (((hg17.chr7(+):127472258-127472280|hg17_7:0,(canFam2.chr14(-):11089526-11089548|canFam2_7:0.572157,dasNov1.scaffold_256527(+):964-987|dasNov1_7:0.212507):0.327632):0,panTro1.chr6(+):129886139-129886161|panTro1_7:0):0,rheMac2.chr3(+):165789069-165789091|rheMac2_7:0,(mm7.chr6(+):28991025-28991048|mm7_7:0.0708909,rn3.chr4(+):56183879-56183902|rn3_7:0.0230732):125.221) +8 (((hg17,panTro1),rheMac2),(rn3,canFam2),dasNov1) (((hg17.chr7(+):127472280-127472681|hg17_8:0.00753956,panTro1.chr6(+):129886161-129886562|panTro1_8:0):0.0353922,rheMac2.chr3(+):165789091-165789492|rheMac2_8:0.0278572):0.0967916,(rn3.chr4(+):56183902-56184219|rn3_8:0.379264,canFam2.chr14(-):11089143-11089523|canFam2_8:0.207268):0.0346594,dasNov1.scaffold_256527(+):987-1401|dasNov1_8:0.268281) +9 (hg17,((rheMac2,dasNov1),canFam2),panTro1) (hg17.chr7(+):127472681-127472715|hg17_9:0.0183718,((rheMac2.chr3(+):165789492-165789526|rheMac2_9:0,dasNov1.scaffold_256527(+):1401-1433|dasNov1_9:0.160078):0.0595979,canFam2.chr14(-):11089108-11089143|canFam2_9:0.237132):0.0383088,panTro1.chr6(+):129886562-129886596|panTro1_9:0.0152865) diff -r 546e2d835e16 -r 7411b119a3c9 tools/hyphy/hyphy_branch_lengths_wrapper.xml --- a/tools/hyphy/hyphy_branch_lengths_wrapper.xml Tue Jan 27 14:48:51 2009 -0500 +++ b/tools/hyphy/hyphy_branch_lengths_wrapper.xml Tue Jan 27 16:17:22 2009 -0500 @@ -1,4 +1,4 @@ -<?xml version="1.0"?> +<?xml version="1.1.0"?> <tool name="Branch Lengths" id="hyphy_branch_lengths_wrapper1"> <description>Estimation</description> diff -r 546e2d835e16 -r 7411b119a3c9 tools/hyphy/hyphy_dnds_wrapper.xml --- a/tools/hyphy/hyphy_dnds_wrapper.xml Tue Jan 27 14:48:51 2009 -0500 +++ b/tools/hyphy/hyphy_dnds_wrapper.xml Tue Jan 27 16:17:22 2009 -0500 @@ -1,4 +1,4 @@ -<?xml version="1.0"?> +<?xml version="1.1.0"?> <tool name="dN/dS Ratio" id="hyphy_dnds_wrapper1"> <description>Estimation</description> diff -r 546e2d835e16 -r 7411b119a3c9 tools/hyphy/hyphy_nj_tree_wrapper.xml --- a/tools/hyphy/hyphy_nj_tree_wrapper.xml Tue Jan 27 14:48:51 2009 -0500 +++ b/tools/hyphy/hyphy_nj_tree_wrapper.xml Tue Jan 27 16:17:22 2009 -0500 @@ -1,4 +1,4 @@ -<?xml version="1.0"?> +<?xml version="1.1.0"?> <tool name="Neighbor Joining Tree" id="hyphy_nj_tree_wrapper1"> <description>Builder</description> diff -r 546e2d835e16 -r 7411b119a3c9 tools/regVariation/microsats_mutability.xml --- a/tools/regVariation/microsats_mutability.xml Tue Jan 27 14:48:51 2009 -0500 +++ b/tools/regVariation/microsats_mutability.xml Tue Jan 27 16:17:22 2009 -0500 @@ -1,4 +1,4 @@ -<tool id="microsats_mutability1" name="Estimate microsatellite mutability" version="1.0.0"> +<tool id="microsats_mutability1" name="Estimate microsatellite mutability" version="1.1.0"> <description>by specified attributes</description> <command interpreter="python"> microsats_mutability.py diff -r 546e2d835e16 -r 7411b119a3c9 tools/stats/grouping.xml --- a/tools/stats/grouping.xml Tue Jan 27 14:48:51 2009 -0500 +++ b/tools/stats/grouping.xml Tue Jan 27 16:17:22 2009 -0500 @@ -1,4 +1,4 @@ -<tool id="Grouping1" name="Group" version="1.4.0"> +<tool id="Grouping1" name="Group" version="1.5.0"> <description>data by a column and perform aggregate operation on other columns.</description> <command interpreter="python"> grouping.py

1 0