- galaxy-dev - lists.galaxyproject.org

trouble running galaxy server
by Andrew Stewart 04 Jun '09

04 Jun '09

sh run.sh seems to work okay until hitting this block of report... Any ideas what this socket business is about? It looks to me like possibly something python-specific. Starting server in PID 11963. Traceback (most recent call last): File "./scripts/paster.py", line 27, in ? command.run() File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ PasteScript-1.3.6-py2.4.egg/paste/script/command.py", line 78, in run File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ PasteScript-1.3.6-py2.4.egg/paste/script/command.py", line 117, in invoke File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ PasteScript-1.3.6-py2.4.egg/paste/script/command.py", line 212, in run File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ PasteScript-1.3.6-py2.4.egg/paste/script/serve.py", line 232, in command File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ PasteDeploy-1.3.1-py2.4.egg/paste/deploy/loadwsgi.py", line 139, in server_wrapper File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ PasteDeploy-1.3.1-py2.4.egg/paste/deploy/util/fixtypeerror.py", line 57, in fix_call File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ Paste-1.5.1-py2.4.egg/paste/httpserver.py", line 1307, in server_runner File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ Paste-1.5.1-py2.4.egg/paste/httpserver.py", line 1257, in serve File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ Paste-1.5.1-py2.4.egg/paste/httpserver.py", line 1107, in __init__ File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ Paste-1.5.1-py2.4.egg/paste/httpserver.py", line 1087, in __init__ File "/root/src/galaxy-f7336991d0ee/eggs/py2.4-noplatform/ Paste-1.5.1-py2.4.egg/paste/httpserver.py", line 328, in __init__ File "/usr/lib64/python2.4/SocketServer.py", line 330, in __init__ self.server_bind() File "/usr/lib64/python2.4/BaseHTTPServer.py", line 101, in server_bind SocketServer.TCPServer.server_bind(self) File "/usr/lib64/python2.4/SocketServer.py", line 341, in server_bind self.socket.bind(self.server_address) File "<string>", line 1, in bind socket.error: (98, 'Address already in use') galaxy.jobs INFO 2009-06-05 02:15:43,692 sending stop signal to worker thread galaxy.jobs INFO 2009-06-05 02:15:43,692 job queue stopped galaxy.jobs.runners.local INFO 2009-06-05 02:15:43,693 sending stop signal to worker threads galaxy.jobs.runners.local INFO 2009-06-05 02:15:43,694 local job runner stopped galaxy.jobs INFO 2009-06-05 02:15:43,694 sending stop signal to worker thread galaxy.jobs INFO 2009-06-05 02:15:43,695 job stopper stopped

2 7

Mercurial advice
by Stewart Stevens 04 Jun '09

04 Jun '09

Hi, Regarding the install instructions at: * http://g2.trac.bx.psu.edu/wiki/HowToInstall We're wondering about the step: * hg clone http://www.bx.psu.edu/hg/galaxy galaxy_dist Previously we had been using svn to access the sources (some months ago now). At this point I believe we got sources corresponding to the development version of galaxy(http://test.g2.bx.psu.edu/). Since we are doing module development we think it may be more appropriate to use a release version (http://main.g2.bx.psu.edu/) It would be nice to confirm that this is what galaxy_dist from the above step actually corresponds to. If not then it would be good to know this too so we can adjust our approach to updates accordingly. Thanks for your attention. Stewart -- Stewart Stevens Software Developer - Integrated Genomics University of Otago Department of Biochemistry Tel: +64(0)3 479 7863 Fax: +64(0)3 479 7866

2 1

[hg] galaxy 2424: Bug fixes for histogram liftover and grouping ...
by Greg Von Kuster 01 Jun '09

01 Jun '09

details: http://www.bx.psu.edu/hg/galaxy/rev/0fc19c283f78 changeset: 2424:0fc19c283f78 user: guru date: Fri May 29 11:07:18 2009 -0400 description: Bug fixes for histogram liftover and grouping tools. 6 file(s) affected in this change: tools/extract/liftOver_wrapper.py tools/extract/liftOver_wrapper.xml tools/plotting/histogram.py tools/plotting/histogram2.xml tools/stats/grouping.py tools/stats/grouping.xml diffs (102 lines): diff -r 7a6f75bf4be5 -r 0fc19c283f78 tools/extract/liftOver_wrapper.py --- a/tools/extract/liftOver_wrapper.py Thu May 28 15:18:49 2009 -0400 +++ b/tools/extract/liftOver_wrapper.py Fri May 29 11:07:18 2009 -0400 @@ -29,7 +29,7 @@ if in_dbkey == "?": stop_err( "Input dataset genome build unspecified, click the pencil icon in the history item to specify it." ) -cmd_line = "liftOver -minMatch=" + minMatch + " " + infile + " " + mapfilepath + " " + outfile1 + " " + outfile2 + " > /dev/null 2>&1" +cmd_line = "liftOver -minMatch=" + str(minMatch) + " " + infile + " " + mapfilepath + " " + outfile1 + " " + outfile2 + " > /dev/null 2>&1" if not os.path.isfile( mapfilepath ): stop_err( "%s mapping is not currently available." % ( mapfilepath.split('/')[-1].split('.')[0] ) ) diff -r 7a6f75bf4be5 -r 0fc19c283f78 tools/extract/liftOver_wrapper.xml --- a/tools/extract/liftOver_wrapper.xml Thu May 28 15:18:49 2009 -0400 +++ b/tools/extract/liftOver_wrapper.xml Fri May 29 11:07:18 2009 -0400 @@ -1,4 +1,4 @@ -<tool id="liftOver1" name="Convert genome coordinates"> +<tool id="liftOver1" name="Convert genome coordinates" version="1.0.1"> <description> between assemblies and genomes</description> <command interpreter="python">liftOver_wrapper.py $input "$out_file1" "$out_file2" $dbkey $to_dbkey $minMatch</command> <inputs> diff -r 7a6f75bf4be5 -r 0fc19c283f78 tools/plotting/histogram.py --- a/tools/plotting/histogram.py Thu May 28 15:18:49 2009 -0400 +++ b/tools/plotting/histogram.py Fri May 29 11:07:18 2009 -0400 @@ -32,7 +32,7 @@ skipped_lines = 0 first_invalid_line = 0 invalid_value = '' - + i = 0 for i, line in enumerate( file( in_fname ) ): valid = True line = line.rstrip('\r\n') @@ -79,7 +79,10 @@ except Exception, exc: stop_err( "%s" %str( exc ) ) else: - stop_err( "All values in column %s are non-numeric." %sys.argv[3] ) + if i == 0: + stop_err("Input dataset is empty.") + else: + stop_err( "All values in column %s are non-numeric." %sys.argv[3] ) print "Histogram of column %s. " %sys.argv[3] if skipped_lines > 0: diff -r 7a6f75bf4be5 -r 0fc19c283f78 tools/plotting/histogram2.xml --- a/tools/plotting/histogram2.xml Thu May 28 15:18:49 2009 -0400 +++ b/tools/plotting/histogram2.xml Fri May 29 11:07:18 2009 -0400 @@ -1,4 +1,4 @@ -<tool id="histogram_rpy" name="Histogram"> +<tool id="histogram_rpy" name="Histogram" version="1.0.1"> <description>of a numeric column</description> <command interpreter="python">histogram.py $input $out_file1 $numerical_column "$title" "$xlab" $breaks $density</command> <inputs> diff -r 7a6f75bf4be5 -r 0fc19c283f78 tools/stats/grouping.py --- a/tools/stats/grouping.py Thu May 28 15:18:49 2009 -0400 +++ b/tools/stats/grouping.py Fri May 29 11:07:18 2009 -0400 @@ -149,9 +149,15 @@ rout = eval( rfunc )( rout ) if op in ['c', 'cuniq']: if op == 'c': - out_str += "\t" + ','.join(rout) + if type(rout) == type([]): + out_str += "\t" + ','.join(rout) + else: + out_str += "\t" + str(rout) else: - out_str += "\t" + ','.join(list(set(rout))) + if type(rout) == type([]): + out_str += "\t" + ','.join(list(set(rout))) + else: + out_str += "\t" + str(rout) else: out_str += "\t" + str(rout) @@ -211,9 +217,15 @@ rout = eval( rfunc )( rout ) if op in ['c','cuniq']: if op == 'c': - out_str += "\t" + ','.join(rout) + if type(rout) == type([]): + out_str += "\t" + ','.join(rout) + else: + out_str += "\t" + str(rout) else: - out_str += "\t" + ','.join(list(set(rout))) + if type(rout) == type([]): + out_str += "\t" + ','.join(list(set(rout))) + else: + out_str += "\t" + str(rout) else: out_str += "\t" + str( rout ) except: diff -r 7a6f75bf4be5 -r 0fc19c283f78 tools/stats/grouping.xml --- a/tools/stats/grouping.xml Thu May 28 15:18:49 2009 -0400 +++ b/tools/stats/grouping.xml Fri May 29 11:07:18 2009 -0400 @@ -1,4 +1,4 @@ -<tool id="Grouping1" name="Group" version="1.6.0"> +<tool id="Grouping1" name="Group" version="1.7.0"> <description>data by a column and perform aggregate operation on other columns.</description> <command interpreter="python"> grouping.py

1 0

[hg] galaxy 2425: Allow the configuration of a job walltime (cur...
by Greg Von Kuster 01 Jun '09

01 Jun '09

details: http://www.bx.psu.edu/hg/galaxy/rev/c8a4a4ea0f2c changeset: 2425:c8a4a4ea0f2c user: Nate Coraor <nate(a)bx.psu.edu> date: Fri May 29 13:19:42 2009 -0400 description: Allow the configuration of a job walltime (currently only supported in PBS) 4 file(s) affected in this change: lib/galaxy/config.py lib/galaxy/jobs/runners/pbs.py lib/galaxy/util/bunch.py universe_wsgi.ini.sample diffs (223 lines): diff -r 0fc19c283f78 -r c8a4a4ea0f2c lib/galaxy/config.py --- a/lib/galaxy/config.py Fri May 29 11:07:18 2009 -0400 +++ b/lib/galaxy/config.py Fri May 29 13:19:42 2009 -0400 @@ -55,6 +55,7 @@ self.job_working_directory = resolve_path( kwargs.get( "job_working_directory", "database/job_working_directory" ), self.root ) self.outputs_to_working_directory = string_as_bool( kwargs.get( 'outputs_to_working_directory', False ) ) self.output_size_limit = int( kwargs.get( 'output_size_limit', 0 ) ) + self.job_walltime = kwargs.get( 'job_walltime', None ) self.admin_users = kwargs.get( "admin_users", "" ) self.sendmail_path = kwargs.get('sendmail_path',"/usr/sbin/sendmail") self.mailing_join_addr = kwargs.get('mailing_join_addr',"galaxy-user-join(a)bx.psu.edu") @@ -123,7 +124,7 @@ """ admin_users = self.get( "admin_users", "" ).split( "," ) return ( user is not None and user.email in admin_users ) - + def get_database_engine_options( kwargs ): """ Allow options for the SQLAlchemy database engine to be passed by using diff -r 0fc19c283f78 -r c8a4a4ea0f2c lib/galaxy/jobs/runners/pbs.py --- a/lib/galaxy/jobs/runners/pbs.py Fri May 29 11:07:18 2009 -0400 +++ b/lib/galaxy/jobs/runners/pbs.py Fri May 29 13:19:42 2009 -0400 @@ -1,8 +1,10 @@ import os, logging, threading, time +from datetime import timedelta from Queue import Queue, Empty from galaxy import model from galaxy.datatypes.data import nice_size +from galaxy.util.bunch import Bunch from paste.deploy.converters import asbool @@ -88,6 +90,10 @@ # set the default server during startup self.default_pbs_server = None self.determine_pbs_server( 'pbs:///' ) + self.job_walltime = None + if self.app.config.job_walltime is not None: + h, m, s = [ int( v ) for v in self.app.config.job_walltime.split( ':' ) ] + self.job_walltime = timedelta( 0, s, 0, 0, m, h ) self.monitor_thread = threading.Thread( target=self.monitor ) self.monitor_thread.start() self.work_queue = Queue() @@ -138,8 +144,8 @@ self.queue_job( obj ) elif op == 'finish': self.finish_job( obj ) - elif op == 'fail_oversize_job': - self.fail_oversize_job( obj ) + elif op == 'fail': + self.fail_job( obj ) except: log.exception( "Uncaught exception %sing job" % op ) @@ -297,7 +303,7 @@ """ new_watched = [] # reduce pbs load by batching status queries - ( failures, states ) = self.check_all_jobs() + ( failures, statuses ) = self.check_all_jobs() for pbs_job_state in self.watched: job_id = pbs_job_state.job_id galaxy_job_id = pbs_job_state.job_wrapper.job_id @@ -307,28 +313,43 @@ log.debug( "(%s/%s) Skipping state check because PBS server connection failed" % ( galaxy_job_id, job_id ) ) new_watched.append( pbs_job_state ) continue - if states.has_key( job_id ): - state = states[job_id] - if state != old_state: - log.debug("(%s/%s) job state changed from %s to %s" % ( galaxy_job_id, job_id, old_state, state ) ) - if state == "R" and not pbs_job_state.running: + if statuses.has_key( job_id ): + status = statuses[job_id] + if status.job_state != old_state: + log.debug("(%s/%s) job state changed from %s to %s" % ( galaxy_job_id, job_id, old_state, status.job_state ) ) + if status.job_state == "R" and not pbs_job_state.running: pbs_job_state.running = True pbs_job_state.job_wrapper.change_state( model.Job.states.RUNNING ) - if self.app.config.output_size_limit > 0 and state == "R" and (pbs_job_state.check_count % 10) == 0: - # Every 10th time a job is checked, check the size of its outputs. - fail = False - for outfile, size in pbs_job_state.job_wrapper.check_output_sizes(): - if size > self.app.config.output_size_limit: - pbs_job_state.fail_message = 'Job output grew too large (greater than %s), please try different job parameters or' \ - % nice_size( self.app.config.output_size_limit ) - log.warning( '(%s/%s) Dequeueing job due to output %s growing larger than %s limit' \ - % ( galaxy_job_id, job_id, os.path.basename( outfile ), nice_size( self.app.config.output_size_limit ) ) ) - self.work_queue.put( ( 'fail_oversize_job', pbs_job_state ) ) - fail = True - break - if fail: - continue - pbs_job_state.old_state = state + if status.job_state == "R" and ( pbs_job_state.check_count % 20 ) == 0: + # Every 20th time the job status is checked, do limit checks (if configured) + if self.app.config.output_size_limit > 0: + # Check the size of the job outputs + fail = False + for outfile, size in pbs_job_state.job_wrapper.check_output_sizes(): + if size > self.app.config.output_size_limit: + pbs_job_state.fail_message = 'Job output grew too large (greater than %s), please try different job parameters or' \ + % nice_size( self.app.config.output_size_limit ) + log.warning( '(%s/%s) Dequeueing job due to output %s growing larger than %s limit' \ + % ( galaxy_job_id, job_id, os.path.basename( outfile ), nice_size( self.app.config.output_size_limit ) ) ) + self.work_queue.put( ( 'fail', pbs_job_state ) ) + fail = True + break + if fail: + continue + if self.job_walltime is not None: + # Check the job's execution time + if status.get( 'resources_used', False ): + # resources_used may not be in the status for new jobs + h, m, s = [ int( i ) for i in status.resources_used.walltime.split( ':' ) ] + time_executing = timedelta( 0, s, 0, 0, m, h ) + if time_executing > self.job_walltime: + pbs_job_state.fail_message = 'Job ran longer than maximum allowed execution time (%s), please try different job parameters or' \ + % self.app.config.job_walltime + log.warning( '(%s/%s) Dequeueing job since walltime has been reached' \ + % ( galaxy_job_id, job_id ) ) + self.work_queue.put( ( 'fail', pbs_job_state ) ) + continue + pbs_job_state.old_state = status.job_state new_watched.append( pbs_job_state ) else: try: @@ -350,11 +371,12 @@ def check_all_jobs( self ): """ Returns a list of servers that failed to be contacted and a dict - of "job_id : state" pairs. + of "job_id : status" pairs (where status is a bunchified version + of the API's structure. """ servers = [] failures = [] - states = {} + statuses = {} for pbs_job_state in self.watched: pbs_server_name = self.determine_pbs_server( pbs_job_state.runner_url ) if pbs_server_name not in servers: @@ -366,13 +388,27 @@ log.debug("connection to PBS server %s for state check failed" % pbs_server_name ) failures.append( pbs_server_name ) continue - stat_attrl = pbs.new_attrl(1) - stat_attrl[0].name = 'job_state' + stat_attrl = pbs.new_attrl(2) + stat_attrl[0].name = pbs.ATTR_state + stat_attrl[1].name = pbs.ATTR_used jobs = pbs.pbs_statjob( c, None, stat_attrl, None ) pbs.pbs_disconnect( c ) - for job in jobs: - states[job.name] = job.attribs[0].value - return( ( failures, states ) ) + statuses.update( self.convert_statjob_to_bunches( jobs ) ) + return( ( failures, statuses ) ) + + def convert_statjob_to_bunches( self, statjob_out ): + statuses = {} + for job in statjob_out: + status = {} + for attrib in job.attribs: + if attrib.resource is None: + status[ attrib.name ] = attrib.value + else: + if attrib.name not in status: + status[ attrib.name ] = Bunch() + status[ attrib.name ][ attrib.resource ] = attrib.value + statuses[ job.name ] = Bunch( **status ) + return statuses def check_single_job( self, pbs_server_name, job_id ): """ @@ -384,7 +420,7 @@ log.debug("connection to PBS server %s for state check failed" % pbs_server_name ) return None stat_attrl = pbs.new_attrl(1) - stat_attrl[0].name = 'job_state' + stat_attrl[0].name = pbs.ATTR_state jobs = pbs.pbs_statjob( c, job_id, stat_attrl, None ) pbs.pbs_disconnect( c ) return jobs[0].attribs[0].value @@ -417,7 +453,7 @@ # clean up the pbs files self.cleanup( ( ofile, efile, job_file ) ) - def fail_oversize_job( self, pbs_job_state ): + def fail_job( self, pbs_job_state ): """ Seperated out so we can use the worker threads for it. """ diff -r 0fc19c283f78 -r c8a4a4ea0f2c lib/galaxy/util/bunch.py --- a/lib/galaxy/util/bunch.py Fri May 29 11:07:18 2009 -0400 +++ b/lib/galaxy/util/bunch.py Fri May 29 13:19:42 2009 -0400 @@ -21,4 +21,7 @@ return '%s' % self.__dict__ def __nonzero__(self): - return bool(self.__dict__) \ No newline at end of file + return bool(self.__dict__) + + def __setitem__(self, k, v): + self.__dict__.__setitem__(k, v) diff -r 0fc19c283f78 -r c8a4a4ea0f2c universe_wsgi.ini.sample --- a/universe_wsgi.ini.sample Fri May 29 11:07:18 2009 -0400 +++ b/universe_wsgi.ini.sample Fri May 29 13:19:42 2009 -0400 @@ -171,6 +171,11 @@ # Job queue cleanup interval in minutes. Currently only used by RoundRobin job_queue_cleanup_interval = 30 +# Jobs can be killed after a certain amount of execution time. Format is in +# hh:mm:ss. Currently only implemented for PBS. Leave commented for +# unlimited. +#job_walltime = 10:00:00 + # Clustering Galaxy is not a straightforward process and requires a lot of # pre-configuration. See the ClusteringGalaxy Wiki before attempting to set # any of these options:

1 0

[hg] galaxy 2426: Add missing functional tests for history featu...
by Greg Von Kuster 01 Jun '09

01 Jun '09

details: http://www.bx.psu.edu/hg/galaxy/rev/e50551db6e60 changeset: 2426:e50551db6e60 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Fri May 29 16:36:43 2009 -0400 description: Add missing functional tests for history features. 4 file(s) affected in this change: lib/galaxy/web/controllers/root.py templates/root/history_as_xml.mako test/base/twilltestcase.py test/functional/test_history_functions.py diffs (264 lines): diff -r c8a4a4ea0f2c -r e50551db6e60 lib/galaxy/web/controllers/root.py --- a/lib/galaxy/web/controllers/root.py Fri May 29 13:19:42 2009 -0400 +++ b/lib/galaxy/web/controllers/root.py Fri May 29 16:36:43 2009 -0400 @@ -59,7 +59,7 @@ return trans.fill_template( '/no_access.mako', message = 'Please log in to access Galaxy histories.' ) if as_xml: trans.response.set_content_type('text/xml') - return trans.fill_template_mako( "root/history_as_xml.mako", history=history ) + return trans.fill_template_mako( "root/history_as_xml.mako", history=history, show_deleted=util.string_as_bool( show_deleted ) ) else: template = "root/history.mako" return trans.fill_template( "root/history.mako", history = history, show_deleted = util.string_as_bool( show_deleted ) ) diff -r c8a4a4ea0f2c -r e50551db6e60 templates/root/history_as_xml.mako --- a/templates/root/history_as_xml.mako Fri May 29 13:19:42 2009 -0400 +++ b/templates/root/history_as_xml.mako Fri May 29 16:36:43 2009 -0400 @@ -1,8 +1,16 @@ <?xml version="1.0"?> <history> - %for data in history.active_datasets: - <data id="${data.id}" hid="${data.hid}" name="${data.name}" state="${data.state}" dbkey="${data.dbkey}"> - ${_(data.blurb)} - </data> - %endfor + %if show_deleted: + %for data in history.activatable_datasets: + <data id="${data.id}" hid="${data.hid}" name="${data.name}" state="${data.state}" dbkey="${data.dbkey}"> + ${_(data.blurb)} + </data> + %endfor + %else: + %for data in history.active_datasets: + <data id="${data.id}" hid="${data.hid}" name="${data.name}" state="${data.state}" dbkey="${data.dbkey}"> + ${_(data.blurb)} + </data> + %endfor + %endif </history> diff -r c8a4a4ea0f2c -r e50551db6e60 test/base/twilltestcase.py --- a/test/base/twilltestcase.py Fri May 29 13:19:42 2009 -0400 +++ b/test/base/twilltestcase.py Fri May 29 16:36:43 2009 -0400 @@ -98,6 +98,7 @@ # Functions associated with histories def check_history_for_errors( self ): """Raises an exception if there are errors in a history""" + self.home() self.visit_page( "history" ) page = self.last_page() if page.find( 'error' ) > -1: @@ -105,6 +106,7 @@ def check_history_for_string( self, patt ): """Looks for 'string' in history page""" + self.home() self.visit_page( "history" ) for subpatt in patt.split(): tc.find(subpatt) @@ -130,16 +132,16 @@ data_list = [ elem for elem in tree.findall("data") ] return data_list - def get_history( self ): + def get_history( self, show_deleted=False ): """Returns a history""" - tree = self.history_as_xml_tree() + tree = self.history_as_xml_tree( show_deleted=show_deleted ) data_list = [ elem for elem in tree.findall("data") ] return data_list - def history_as_xml_tree( self ): + def history_as_xml_tree( self, show_deleted=False ): """Returns a parsed xml object of a history""" self.home() - self.visit_page( 'history?as_xml=True' ) + self.visit_page( 'history?as_xml=True&show_deleted=%s' % show_deleted ) xml = self.last_page() tree = ElementTree.fromstring(xml) return tree @@ -201,6 +203,7 @@ name = elem.get( 'name' ) self.assertTrue( name ) self.visit_url( "%s/history/share?id=%s&email=%s&history_share_btn=Submit" % ( self.url, id, email ) ) + self.check_page_for_string( 'History (%s) has been shared with: %s' % ( name, email ) ) return id, name, email def share_history_containing_private_datasets( self, history_id, email='test(a)bx.psu.edu' ): """Attempt to share a history containing private datasets with a different user""" @@ -237,8 +240,10 @@ self.assertEqual(len(elems), 1) self.visit_page( "history/list?operation=switch&id=%s" % elems[0].get('id') ) - def view_stored_histories( self ): + def view_stored_histories( self, check_str='' ): self.visit_page( "history/list" ) + if check_str: + self.check_page_for_string( check_str ) # Functions associated with datasets (history items) and meta data def get_job_stderr( self, id ): @@ -264,16 +269,36 @@ self.visit_page( "edit?hid=%d" % hid ) for subpatt in patt.split(): tc.find(subpatt) - - def delete_history_item( self, hid ): + def delete_history_item( self, hid, check_str='' ): """Deletes an item from a history""" + try: + hid = int( hid ) + except: + raise AssertionError, "Invalid hid '%s' - must be int" % hid hid = str(hid) data_list = self.get_history() self.assertTrue( data_list ) - elems = [ elem for elem in data_list if elem.get('hid') == hid ] - self.assertEqual(len(elems), 1) - self.visit_page( "delete?id=%s" % elems[0].get('id') ) - + elems = [ elem for elem in data_list if elem.get( 'hid' ) == hid ] + self.assertEqual( len( elems ), 1 ) + self.home() + self.visit_page( "delete?id=%s" % elems[0].get( 'id' ) ) + if check_str: + self.check_page_for_string( check_str ) + def undelete_history_item( self, hid, show_deleted=False, check_str='' ): + """Un-deletes a deleted item in a history""" + try: + hid = int( hid ) + except: + raise AssertionError, "Invalid hid '%s' - must be int" % hid + hid = str( hid ) + data_list = self.get_history( show_deleted=show_deleted ) + self.assertTrue( data_list ) + elems = [ elem for elem in data_list if elem.get( 'hid' ) == hid ] + self.assertEqual( len( elems ), 1 ) + self.home() + self.visit_url( "%s/dataset/undelete?id=%s" % ( self.url, elems[0].get( 'id' ) ) ) + if check_str: + self.check_page_for_string( check_str ) def edit_metadata( self, hid=None, form_no=0, **kwd ): """ Edits the metadata associated with a history item.""" diff -r c8a4a4ea0f2c -r e50551db6e60 test/functional/test_history_functions.py --- a/test/functional/test_history_functions.py Fri May 29 13:19:42 2009 -0400 +++ b/test/functional/test_history_functions.py Fri May 29 16:36:43 2009 -0400 @@ -1,4 +1,6 @@ -from base.twilltestcase import TwillTestCase +import galaxy.model +from galaxy.model.orm import * +from base.twilltestcase import * class TestHistory( TwillTestCase ): @@ -20,15 +22,28 @@ def test_10_history_options_when_logged_in( self ): """Testing history options when logged in""" self.history_options() - self.check_page_for_string( 'Rename</a> current history') - self.check_page_for_string( 'List</a> previously stored histories') - self.check_page_for_string( 'Share</a> current history') - self.check_page_for_string( 'Delete</a> current history') - def test_10_history_rename( self ): + self.check_page_for_string( 'Rename</a> current history' ) + self.check_page_for_string( 'List</a> previously stored histories' ) + self.check_page_for_string( 'Construct workflow</a> from the current history' ) + self.check_page_for_string( 'Share</a> current history' ) + # Tests for changing default history permissions are done in test_security_and_libraries.py + self.check_page_for_string( 'Change default permissions</a> for the current history' ) + self.check_page_for_string( 'Show deleted</a> datasets in history' ) + self.check_page_for_string( 'Delete</a> current history' ) + # Need to add a history item in order to create a new empty history + try: + self.check_page_for_string( 'Create</a> a new empty history' ) + raise AssertionError, "Incorrectly able to create a new empty history when the current history is empty." + except: + pass + self.upload_file( '1.bed', dbkey='hg18' ) + self.history_options() + self.check_page_for_string( 'Create</a> a new empty history' ) + def test_15_history_rename( self ): """Testing renaming a history""" id, old_name, new_name = self.rename_history() self.check_page_for_string( 'History: %s renamed to: %s' %(old_name, new_name) ) - def test_15_history_view( self ): + def test_20_history_list( self ): """Testing viewing previously stored histories""" self.view_stored_histories() self.check_page_for_string( 'Stored histories' ) @@ -36,24 +51,15 @@ self.check_page_for_string( 'operation=Rename&id' ) self.check_page_for_string( 'operation=Switch&id' ) self.check_page_for_string( 'operation=Delete&id' ) - def test_20_delete_history_item( self ): - """Testing deleting history item""" - self.upload_file('1.bed', dbkey='hg15') - self.check_history_for_string('hg15 bed') - self.assertEqual ( len(self.get_history()), 1) - self.delete_history_item(1) - self.check_history_for_string("Your history is empty") - self.assertEqual ( len(self.get_history()), 0) - def test_25_share_history( self ): + def test_25_history_share( self ): """Testing sharing a history with another user""" self.upload_file('1.bed', dbkey='hg18') id, name, email = self.share_history() - self.last_page() - self.check_page_for_string( 'History (%s) has been shared with: %s' %(name, email) ) self.logout() - self.login( email='test2(a)bx.psu.edu' ) - self.view_stored_histories() - self.check_page_for_string( 'Unnamed history from test(a)bx.psu.edu' ) + self.login( email=email ) + self.home() + check_str = 'Unnamed history from test(a)bx.psu.edu' + self.view_stored_histories( check_str=check_str ) histories = self.get_histories() for history in histories: if history.get( 'name' ) == 'Unnamed history from test(a)bx.psu.edu': @@ -63,6 +69,47 @@ self.delete_history( id ) self.logout() self.login( email='test(a)bx.psu.edu' ) + def test_30_history_show_and_hide_deleted_datasets( self ): + """Testing displaying deleted history items""" + self.new_history() + self.upload_file('1.bed', dbkey='hg18') + latest_hda = galaxy.model.HistoryDatasetAssociation.query() \ + .order_by( desc( galaxy.model.HistoryDatasetAssociation.table.c.create_time ) ).first() + self.home() + self.visit_url( "%s/root/delete?show_deleted_on_refresh=False&id=%s" % ( self.url, str( latest_hda.id ) ) ) + self.check_history_for_string( 'Your history is empty' ) + self.home() + self.visit_url( "%s/history/?show_deleted=True" % self.url ) + self.check_page_for_string( 'This dataset has been deleted.' ) + self.check_page_for_string( '1.bed' ) + self.home() + self.visit_url( "%s/history/?show_deleted=False" % self.url ) + self.check_page_for_string( 'Your history is empty' ) + def test_35_deleting_and_undeleting_history_items( self ): + """Testing deleting and un-deleting history items""" + self.new_history() + # Add a new history item + self.upload_file( '1.bed', dbkey='hg15' ) + self.home() + self.visit_url( "%s/history/?show_deleted=False" % self.url ) + self.check_page_for_string( '1.bed' ) + self.check_page_for_string( 'hg15' ) + self.assertEqual ( len( self.get_history() ), 1 ) + # Delete the history item + self.delete_history_item( 1, check_str="Your history is empty" ) + self.assertEqual ( len( self.get_history() ), 0 ) + # Try deleting an invalid hid + try: + self.delete_history_item( 'XXX' ) + raise AssertionError, "Inproperly able to delete hid 'XXX' which is not an integer" + except: + pass + # Undelete the history item + self.undelete_history_item( 1, show_deleted=True ) + self.home() + self.visit_url( "%s/history/?show_deleted=False" % self.url ) + self.check_page_for_string( '1.bed' ) + self.check_page_for_string( 'hg15' ) def test_9999_clean_up( self ): self.delete_history() self.logout()

1 0

[hg] galaxy 2421: Add a defaulttimeout setting of 10 minutes to ...
by Greg Von Kuster 29 May '09

29 May '09

details: http://www.bx.psu.edu/hg/galaxy/rev/186fcb977e75 changeset: 2421:186fcb977e75 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu May 28 11:10:44 2009 -0400 description: Add a defaulttimeout setting of 10 minutes to the urlopen() call for requests to remote data sources. 1 file(s) affected in this change: tools/data_source/data_source.py diffs (43 lines): diff -r 5c2d30d3fb04 -r 186fcb977e75 tools/data_source/data_source.py --- a/tools/data_source/data_source.py Wed May 27 14:19:59 2009 -0400 +++ b/tools/data_source/data_source.py Thu May 28 11:10:44 2009 -0400 @@ -1,8 +1,8 @@ #!/usr/bin/env python -#Retreives data from UCSC and stores in a file. UCSC parameters are provided in the input/output file. -import urllib, sys, os, gzip, tempfile, shutil +# Retrieves data from external data source applications and stores in a dataset file. +# Data source application parameters are temporarily stored in the dataset file. +import socket, urllib, sys, os, gzip, tempfile, shutil from galaxy import eggs -#from galaxy.datatypes import data from galaxy.util import gzip_magic assert sys.version_info[:2] >= ( 2, 4 ) @@ -34,15 +34,23 @@ open( filename, 'w' ).write( "" ) stop_err( 'The remote data source application has not sent back a URL parameter in the request.' ) URL_method = params.get( 'URL_method', None ) - out = open( filename, 'w' ) - CHUNK_SIZE = 2**20 # 1Mb + CHUNK_SIZE = 2**20 # 1Mb + # The Python support for fetching resources from the web is layered. urllib uses the httplib + # library, which in turn uses the socket library. As of Python 2.3 you can specify how long + # a socket should wait for a response before timing out. By default the socket module has no + # timeout and can hang. Currently, the socket timeout is not exposed at the httplib or urllib2 + # levels. However, you can set the default timeout ( in seconds ) globally for all sockets by + # doing the following. + socket.setdefaulttimeout( 600 ) + # The following calls to urllib2.urlopen() will use the above default timeout try: if not URL_method or URL_method == 'get': page = urllib.urlopen( URL ) elif URL_method == 'post': page = urllib.urlopen( URL, urllib.urlencode( params ) ) - except: - stop_err( 'It appears that the remote data source application is currently off line. Please try again later.' ) + except Exception, e: + stop_err( 'The remote data source application may be off line, please try again later. Error: %s' % str( e ) ) + out = open( filename, 'w' ) while 1: chunk = page.read( CHUNK_SIZE ) if not chunk:

1 0

[hg] galaxy 2423: Fix the cleanup_datasets script
by Greg Von Kuster 29 May '09

29 May '09

details: http://www.bx.psu.edu/hg/galaxy/rev/7a6f75bf4be5 changeset: 2423:7a6f75bf4be5 user: Nate Coraor <nate(a)bx.psu.edu> date: Thu May 28 15:18:49 2009 -0400 description: Fix the cleanup_datasets script 1 file(s) affected in this change: scripts/cleanup_datasets/cleanup_datasets.py diffs (17 lines): diff -r a07e84e73b45 -r 7a6f75bf4be5 scripts/cleanup_datasets/cleanup_datasets.py --- a/scripts/cleanup_datasets/cleanup_datasets.py Thu May 28 13:58:56 2009 -0400 +++ b/scripts/cleanup_datasets/cleanup_datasets.py Thu May 28 15:18:49 2009 -0400 @@ -264,8 +264,13 @@ if dataset.extra_files_path and os.path.exists( dataset.extra_files_path ): shutil.rmtree( dataset.extra_files_path ) #we need to delete the directory and its contents; os.unlink would always fail on a directory dataset.purged = True + dataset.flush() else: print "# This dataset (%i) is not purgable, the file (%s) will not be removed.\n" % ( dataset.id, dataset.file_name ) + except OSError, exc: + print "# Error, file has already been removed: %s" % str( exc ) + dataset.purged = True + dataset.flush() except Exception, exc: print "# Error, exception: %s caught attempting to purge %s\n" %( str( exc ), dataset.file_name ) else:

1 0

[hg] galaxy 2422: Fix for DrillDownParameter when no option is s...
by Greg Von Kuster 29 May '09

29 May '09

details: http://www.bx.psu.edu/hg/galaxy/rev/a07e84e73b45 changeset: 2422:a07e84e73b45 user: Dan Blankenberg <dan(a)bx.psu.edu> date: Thu May 28 13:58:56 2009 -0400 description: Fix for DrillDownParameter when no option is selected and saved in worflow editing. 1 file(s) affected in this change: lib/galaxy/tools/parameters/basic.py diffs (15 lines): diff -r 186fcb977e75 -r a07e84e73b45 lib/galaxy/tools/parameters/basic.py --- a/lib/galaxy/tools/parameters/basic.py Thu May 28 11:10:44 2009 -0400 +++ b/lib/galaxy/tools/parameters/basic.py Thu May 28 13:58:56 2009 -0400 @@ -947,7 +947,10 @@ def from_html( self, value, trans=None, other_values={} ): if self.need_late_validation( trans, other_values ): if self.multiple: - value = value.split( "\n" ) + if value == '': #No option selected + value = None + else: + value = value.split( "\n" ) return UnvalidatedValue( value ) if not value: return None if not isinstance( value, list ):

1 0

[hg] galaxy 2419: Add a hack to force UCSC links to be rewritten...
by Greg Von Kuster 27 May '09

27 May '09

details: http://www.bx.psu.edu/hg/galaxy/rev/1acbcbcebeb3 changeset: 2419:1acbcbcebeb3 user: Nate Coraor <nate(a)bx.psu.edu> date: Wed May 27 11:15:35 2009 -0400 description: Add a hack to force UCSC links to be rewritten as http if your upstream uses https 1 file(s) affected in this change: lib/galaxy/datatypes/interval.py diffs (16 lines): diff -r 7414f90d7eae -r 1acbcbcebeb3 lib/galaxy/datatypes/interval.py --- a/lib/galaxy/datatypes/interval.py Tue May 26 16:35:23 2009 -0400 +++ b/lib/galaxy/datatypes/interval.py Wed May 27 11:15:35 2009 -0400 @@ -223,6 +223,12 @@ stop = viewport_tuple[2] for site_name, site_url in util.get_ucsc_by_build(dataset.dbkey): if site_name in app.config.ucsc_display_sites: + # HACK: UCSC doesn't support https, so force http even + # if our URL scheme is https. Making this work + # requires additional hackery in your upstream proxy. + # If UCSC ever supports https, remove this hack. + if base_url.startswith( 'https://' ): + base_url = base_url.replace( 'https', 'http', 1 ) display_url = urllib.quote_plus( "%s%s/display_as?id=%i&display_app=%s" % (base_url, url_for( controller='root' ), dataset.id, type) ) link = "%sdb=%s&position=%s:%s-%s&hgt.customText=%s" % (site_url, dataset.dbkey, chrom, start, stop, display_url ) ret_val.append( (site_name, link) )

1 0

[hg] galaxy 2420: Add tool config for GrameneMart data source.
by Greg Von Kuster 27 May '09

27 May '09

details: http://www.bx.psu.edu/hg/galaxy/rev/5c2d30d3fb04 changeset: 2420:5c2d30d3fb04 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed May 27 14:19:59 2009 -0400 description: Add tool config for GrameneMart data source. 3 file(s) affected in this change: tool_conf.xml.main tool_conf.xml.sample tools/data_source/gramene_mart.xml diffs (67 lines): diff -r 1acbcbcebeb3 -r 5c2d30d3fb04 tool_conf.xml.main --- a/tool_conf.xml.main Wed May 27 11:15:35 2009 -0400 +++ b/tool_conf.xml.main Wed May 27 14:19:59 2009 -0400 @@ -6,6 +6,7 @@ <tool file="data_source/ucsc_tablebrowser_archaea.xml" /> <tool file="data_source/microbial_import.xml" /> <tool file="data_source/biomart.xml" /> + <tool file="data_source/gramene_mart.xml" /> <tool file="data_source/flymine.xml" /> <tool file="data_source/encode_db.xml" /> <tool file="data_source/epigraph_import.xml" /> diff -r 1acbcbcebeb3 -r 5c2d30d3fb04 tool_conf.xml.sample --- a/tool_conf.xml.sample Wed May 27 11:15:35 2009 -0400 +++ b/tool_conf.xml.sample Wed May 27 14:19:59 2009 -0400 @@ -8,6 +8,7 @@ <tool file="data_source/microbial_import.xml" /> <tool file="data_source/biomart.xml" /> <tool file="data_source/biomart_test.xml" /> + <tool file="data_source/gramene_mart.xml" /> <tool file="data_source/wormbase.xml" /> <tool file="data_source/wormbase_test.xml" /> <tool file="data_source/flymine.xml" /> diff -r 1acbcbcebeb3 -r 5c2d30d3fb04 tools/data_source/gramene_mart.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/data_source/gramene_mart.xml Wed May 27 14:19:59 2009 -0400 @@ -0,0 +1,41 @@ +<?xml version="1.0"?> + +<tool name="GrameneMart" id="gramenemart" tool_type="data_source" URL_method="get" version="1.0.1"> + <description> Central server</description> + <command interpreter="python">data_source.py $output</command> + <inputs action="http://www.gramene.org/biomart/martview" check_values="false" method="get" target="_top"> + <display>go to GrameneMart Central $GALAXY_URL</display> + <param name="GALAXY_URL" type="baseurl" value="/tool_runner/biomart" /> + </inputs> + <request_param_translation> + <request_param galaxy_name="URL" remote_name="URL" missing=""> + <add_to_url> + <param_from_source name="_export" missing="1" /> + <param_from_source name="GALAXY_URL" missing="0" /> + </add_to_url> + </request_param> + <request_param galaxy_name="data_type" remote_name="exportView_outputformat" missing="tabular"> + <data_type_translation> + <format galaxy_format="tabular" remote_format="TSV" /> + </data_type_translation> + </request_param> + <request_param galaxy_name="dbkey" remote_name="dbkey" missing="?" /> + <request_param galaxy_name="organism" remote_name="organism" missing="" /> + <request_param galaxy_name="table" remote_name="table" missing="" /> + <request_param galaxy_name="description" remote_name="description" missing="" /> + <request_param galaxy_name="name" remote_name="name" missing="Biomart query" /> + <request_param galaxy_name="info" remote_name="info" missing="" /> + </request_param_translation> + <uihints minwidth="800"/> + <outputs> + <data name="output" format="tabular" /> + </outputs> + <options sanitize="False" refresh="True"/> +</tool>

1 0