- galaxy-dev - lists.galaxyproject.org

Not-So-Running Jobs
by Assaf Gordon 24 Apr '09

24 Apr '09

Hello, I've encountered a strange situation (at at least strange for me): Users are submitting jobs, and the jobs start almost immediately. However, the jobs look like they are running for very long time. Checking the report web page, a job looks like the attached image. The status is "running", but the command line is empty, and no program was executed for this job (I checked with "ps ax -H" and looked for python's child-processes). Some technical information: Running on Fedora with Python 2.4.3, PostgreSQL 8.0. The server is loaded but not too loaded ( 14.4 load average for 16 cores ). Relevant settings from universe_wsgi.ini: use_threadpool = true threadpool_workers = 10 local_job_queue_workers = 5 job_scheduler_policy = galaxy.jobs.schedulingpolicy.roundrobin:UserRoundRobin job_queue_cleanup_interval = 30 Is this normal ? Thanks, Gordon.

2 3

Re: [galaxy-dev] Cluster: job to User identification
by Nate Coraor 24 Apr '09

24 Apr '09

Hi Alastair, I've moved this discussion over to galaxy-dev since it's more relevant to that list. Alastair Kerr wrote: > I am looking to add galaxy to our local cluster. One of the conditions > of use is that I have to ensure that any job submitted to the cluster > can be traced back to an individual user. What is the easiest means to > achieve this? Since Galaxy runs under a single non-root user, we can't submit jobs as another user. So, all of the jobs in the scheduler itself will be run as that single user. > I assume that I have to force user logins (this can be done via the > universe_wsgi.ini I think). > However my local install running on a single machine does not appear to > list user information in the paster.log file, just the ip address. I'll look in to adding the username to the log statement, but in the meantime, a job's user can be selected from the database using a statement like: SELECT galaxy_user.email FROM galaxy_user, history, job WHERE job.history_id = history.id AND galaxy_user.id = history.user_id AND job.id = <job id>; --nate

1 0

New tool for Galaxy: column normalizer
by Assaf Gordon 23 Apr '09

23 Apr '09

Hello, I'd like to share a new tool for Galaxy: A Column Normalizer - Takes a tab-delimiter file with numeric values, and normalizes them. The tool is useful if you need to compare ratios between files of different values. Binaries + source + xml are available here: http://hannonlab.cshl.edu/column_normalizer/ Although you can normalize data manually in Galaxy (by using group + compute tools), it's hard to normalize data inside a workflow - which is what this tool is for. This tool also provide finer control over the output format (scientific or decimal-point notations). The tool is free for use. Comments are welcomed, Gordon.

1 0

[hg] galaxy 2364: Fix for uploading binary files into a library.
by Greg Von Kuster 23 Apr '09

23 Apr '09

details: http://www.bx.psu.edu/hg/galaxy/rev/f7336991d0ee changeset: 2364:f7336991d0ee user: Greg Von Kuster <greg(a)bx.psu.edu> date: Thu Apr 23 09:20:11 2009 -0400 description: Fix for uploading binary files into a library. 2 file(s) affected in this change: lib/galaxy/util/__init__.py lib/galaxy/web/controllers/library_dataset.py diffs (29 lines): diff -r feb4f5dab9cf -r f7336991d0ee lib/galaxy/util/__init__.py --- a/lib/galaxy/util/__init__.py Wed Apr 22 14:43:54 2009 -0400 +++ b/lib/galaxy/util/__init__.py Thu Apr 23 09:20:11 2009 -0400 @@ -37,9 +37,8 @@ for char in chars: try: char = unicode( char ) - except Exception, e: + except UnicodeDecodeError, e: # Probably binary - log.exception( e ) return False if wchartype.is_asian( char ) or \ wchartype.is_full_width( char ) or \ diff -r feb4f5dab9cf -r f7336991d0ee lib/galaxy/web/controllers/library_dataset.py --- a/lib/galaxy/web/controllers/library_dataset.py Wed Apr 22 14:43:54 2009 -0400 +++ b/lib/galaxy/web/controllers/library_dataset.py Thu Apr 23 09:20:11 2009 -0400 @@ -69,7 +69,10 @@ ext = file_format if not data_type: if self.check_binary( temp_name ): - ext = file_name.split( "." )[1].strip().lower() + try: + ext = name.split( "." )[1].strip().lower() + except: + ext = '' if not( ext == 'ab1' or ext == 'scf' ): raise BadFileException( "you attempted to upload an inappropriate file." ) if ext == 'ab1' and file_format != 'ab1':

1 0

[hg] galaxy 2363: Fixes for native Leopard Python eggs
by Nate Coraor 22 Apr '09

22 Apr '09

details: http://www.bx.psu.edu/hg/galaxy/rev/feb4f5dab9cf changeset: 2363:feb4f5dab9cf user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Apr 22 14:43:54 2009 -0400 description: Fixes for native Leopard Python eggs 3 file(s) affected in this change: dist-eggs.ini eggs.ini scripts/scramble/scripts/MySQL_python.py diffs (96 lines): diff -r 8c93a7c02c27 -r feb4f5dab9cf dist-eggs.ini --- a/dist-eggs.ini Wed Apr 22 12:36:17 2009 -0400 +++ b/dist-eggs.ini Wed Apr 22 14:43:54 2009 -0400 @@ -17,6 +17,7 @@ py2.5-linux-x86_64-ucs4 = herbie.bx.psu.edu /depot/projects/pythons/linux-x86_64-ucs4/bin/python2.5 py2.4-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.4 py2.5-macosx-10.3-fat-ucs2 = medeski.bx.psu.edu /usr/local/bin/python2.5 +py2.5-macosx-10.5-i386-ucs2 = lion.bx.psu.edu /usr/bin/python2.5 py2.4-solaris-2.11-i86pc-ucs2 = victory.bx.psu.edu /depot/projects/pythons/solaris-2.11-i86pc-ucs2/bin/python2.4 py2.5-solaris-2.11-i86pc-ucs2 = victory.bx.psu.edu /depot/projects/pythons/solaris-2.11-i86pc-ucs2/bin/python2.5 py2.4-solaris-2.10-sun4u-ucs2 = v880.bx.psu.edu /depot/projects/pythons/solaris-2.10-sun4u-ucs2/bin/python2.4 @@ -27,7 +28,7 @@ py2.5-linux = py2.5-linux-i686-ucs2 py2.5-linux-i686-ucs4 py2.5-linux-x86_64-ucs2 py2.5-linux-x86_64-ucs4 linux = py2.4-linux py2.5-linux py2.4-macosx = py2.4-macosx-10.3-fat-ucs2 -py2.5-macosx = py2.5-macosx-10.3-fat-ucs2 +py2.5-macosx = py2.5-macosx-10.3-fat-ucs2 py2.5-macosx-10.5-i386-ucs2 macosx = py2.4-macosx py2.5-macosx py2.4-solaris = py2.4-solaris-2.11-i86pc-ucs2 py2.4-solaris-2.10-sun4u-ucs2 py2.5-solaris = py2.5-solaris-2.11-i86pc-ucs2 py2.5-solaris-2.10-sun4u-ucs2 diff -r 8c93a7c02c27 -r feb4f5dab9cf eggs.ini --- a/eggs.ini Wed Apr 22 12:36:17 2009 -0400 +++ b/eggs.ini Wed Apr 22 14:43:54 2009 -0400 @@ -70,7 +70,7 @@ DRMAA_python = http://gridengine.sunsource.net/files/documents/7/36/DRMAA-python-0.2.tar.gz MySQL_python = http://superb-west.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python… http://mysql.mirrors.pair.com/Downloads/MySQL-5.0/mysql-5.0.67.tar.gz pbs_python = http://ftp.sara.nl/pub/outgoing/pbs_python-2.9.4.tar.gz -psycopg2 = http://initd.org/pub/software/psycopg/PSYCOPG-2-0/psycopg2-2.0.6.tar.gz http://ftp8.us.postgresql.org/postgresql/source/v8.2.6/postgresql-8.2.6.tar… +psycopg2 = http://initd.org/pub/software/psycopg/PSYCOPG-2-0/psycopg2-2.0.6.tar.gz ftp://ftp-archives.postgresql.org/pub/source/v8.2.6/postgresql-8.2.6.tar.bz2 pycrypto = http://www.amk.ca/files/python/crypto/pycrypto-2.0.1.tar.gz pysqlite = http://initd.org/pub/software/pysqlite/releases/2.3/2.3.5/pysqlite-2.3.5.ta… http://www.sqlite.org/sqlite-source-3_5_4.zip python_lzo = http://www.oberhumer.com/opensource/lzo/download/LZO-v1/python-lzo-1.08.tar… http://www.oberhumer.com/opensource/lzo/download/LZO-v1/lzo-1.08.tar.gz diff -r 8c93a7c02c27 -r feb4f5dab9cf scripts/scramble/scripts/MySQL_python.py --- a/scripts/scramble/scripts/MySQL_python.py Wed Apr 22 12:36:17 2009 -0400 +++ b/scripts/scramble/scripts/MySQL_python.py Wed Apr 22 14:43:54 2009 -0400 @@ -43,7 +43,7 @@ print "build_mysql(): Installing mysql (make install) failed" sys.exit( 1 ) # pack - print "build_mysql(): Creating binary mysql archive for future builds of psycopg2" + print "build_mysql(): Creating binary mysql archive for future builds of MySQL_python" t = tarfile.open( MYSQL_BINARY_ARCHIVE, "w:bz2" ) t.add( "mysql/bin/mysql_config" ) t.add( "mysql/include" ) @@ -57,14 +57,10 @@ # find setuptools scramble_lib = os.path.join( "..", "..", "..", "lib" ) sys.path.append( scramble_lib ) -try: - from setuptools import * - import pkg_resources -except: - from ez_setup import use_setuptools - use_setuptools( download_delay=8, to_dir=scramble_lib ) - from setuptools import * - import pkg_resources +from ez_setup import use_setuptools +use_setuptools( download_delay=8, to_dir=scramble_lib ) +from setuptools import * +import pkg_resources # get the tag if os.access( ".galaxy_tag", os.F_OK ): @@ -100,7 +96,7 @@ # patch file = "site.cfg" -print "scramble_it(): Patching", file +print "scramble(): Patching", file if not os.access( "%s.orig" %file, os.F_OK ): shutil.copyfile( file, "%s.orig" %file ) i = open( "%s.orig" %file, "r" ) @@ -112,6 +108,23 @@ i.close() o.close() +if pkg_resources.get_platform() == 'macosx-10.5-i386': + file = "_mysql.c" + print "scramble(): Patching", file + if not os.access( "%s.orig" %file, os.F_OK ): + shutil.copyfile( file, "%s.orig" %file ) + i = open( "%s.orig" %file, "r" ) + o = open( file, "w" ) + for line in i: + if line == '#ifndef uint\n': + # skip 3 lines + i.next() + i.next() + line = i.next() + print >>o, line, + i.close() + o.close() + # tag me = sys.argv[0] sys.argv = [ me ]

1 0

[hg] galaxy 2362: Clean up (unify) workflow sharing UI a bit
by Nate Coraor 22 Apr '09

22 Apr '09

details: http://www.bx.psu.edu/hg/galaxy/rev/8c93a7c02c27 changeset: 2362:8c93a7c02c27 user: James Taylor <james(a)jamestaylor.org> date: Wed Apr 22 12:36:17 2009 -0400 description: Clean up (unify) workflow sharing UI a bit 3 file(s) affected in this change: lib/galaxy/web/controllers/workflow.py templates/workflow/list.mako templates/workflow/sharing.mako diffs (62 lines): diff -r ed6bfb311fb0 -r 8c93a7c02c27 lib/galaxy/web/controllers/workflow.py --- a/lib/galaxy/web/controllers/workflow.py Wed Apr 22 12:13:29 2009 -0400 +++ b/lib/galaxy/web/controllers/workflow.py Wed Apr 22 12:36:17 2009 -0400 @@ -91,7 +91,7 @@ session.save_or_update( share ) session.flush() trans.set_message( "Workflow '%s' shared with user '%s'" % ( stored.name, other.email ) ) - return self.list( trans ) + return trans.response.send_redirect( url_for( controller='workflow', action='sharing', id=id ) ) return trans.fill_template( "workflow/share.mako", message = msg, messagetype = mtype, diff -r ed6bfb311fb0 -r 8c93a7c02c27 templates/workflow/list.mako --- a/templates/workflow/list.mako Wed Apr 22 12:13:29 2009 -0400 +++ b/templates/workflow/list.mako Wed Apr 22 12:36:17 2009 -0400 @@ -47,8 +47,7 @@ <a class="action-button" href="${h.url_for( action='editor', id=trans.security.encode_id(workflow.id) )}" target="_parent">Edit</a> <a class="action-button" href="${h.url_for( controller='root', action='index', workflow_id=trans.security.encode_id(workflow.id) )}" target="_parent">Run</a> <a class="action-button" href="${h.url_for( action='rename', id=trans.security.encode_id(workflow.id) )}">Rename</a> - <a class="action-button" href="${h.url_for( action='share', id=trans.security.encode_id(workflow.id) )}">Share</a> - <a class="action-button" href="${h.url_for( action='sharing', id=trans.security.encode_id(workflow.id) )}">Manage sharing</a> + <a class="action-button" href="${h.url_for( action='sharing', id=trans.security.encode_id(workflow.id) )}">Sharing</a> <a class="action-button" confirm="Are you sure you want to delete workflow '${workflow.name}'?" href="${h.url_for( action='delete', id=trans.security.encode_id(workflow.id) )}">Delete</a> </div> </td> diff -r ed6bfb311fb0 -r 8c93a7c02c27 templates/workflow/sharing.mako --- a/templates/workflow/sharing.mako Wed Apr 22 12:13:29 2009 -0400 +++ b/templates/workflow/sharing.mako Wed Apr 22 12:36:17 2009 -0400 @@ -39,7 +39,15 @@ The following users will see this workflow in thier workflow list, and be able to run it or create their own copy of it: </p> - + + <ul class="manage-table-actions"> + <li> + <a class="action-button" href="${h.url_for( action='share', id=trans.security.encode_id(stored.id) )}"> + <span>Share with another user</span> + </a> + </li> + </ul> + <table class="colored" border="0" cellspacing="0" cellpadding="0" width="100%"> <tr class="header"> <th>Email</th> @@ -60,8 +68,13 @@ </tr> %endfor </table> + %else: - You have not shared this workflow with any users. - + <p>You have not shared this workflow with any users.</p> + + <a class="action-button" href="${h.url_for( action='share', id=trans.security.encode_id(stored.id) )}"> + <span>Share with another user</span> + </a> + %endif \ No newline at end of file

1 0

[hg] galaxy 2361: Fixes for copying library permissions, do not ...
by Greg Von Kuster 22 Apr '09

22 Apr '09

details: http://www.bx.psu.edu/hg/galaxy/rev/ed6bfb311fb0 changeset: 2361:ed6bfb311fb0 user: Greg Von Kuster <greg(a)bx.psu.edu> date: Wed Apr 22 12:13:29 2009 -0400 description: Fixes for copying library permissions, do not set permissions for the current user when changing permissions on library items. 3 file(s) affected in this change: lib/galaxy/security/__init__.py lib/galaxy/web/controllers/admin.py lib/galaxy/web/controllers/library.py diffs (62 lines): diff -r 2751a34dbd4b -r ed6bfb311fb0 lib/galaxy/security/__init__.py --- a/lib/galaxy/security/__init__.py Wed Apr 22 10:57:07 2009 -0400 +++ b/lib/galaxy/security/__init__.py Wed Apr 22 12:13:29 2009 -0400 @@ -382,7 +382,7 @@ permissions[role_assoc.action] = [ role_assoc.role ] self.set_all_library_permissions( target_library_item, permissions ) if user: - # Make sure user's private role is included + # The user passed will be the current Galaxy user. Make sure user's private role is included item_class = None for item_class, permission_class, info_association_class in self.library_item_assocs: if isinstance( target_library_item, item_class ): @@ -396,7 +396,6 @@ else: raise 'Invalid class (%s) specified for target_library_item (%s)' % ( target_library_item.__class__, target_library_item.__class__.__name__ ) def show_library_item( self, user, library_item ): - # TODO: possibly needs to support other library item types if self.allow_action( user, self.permitted_actions.LIBRARY_MODIFY, library_item=library_item ) or \ self.allow_action( user, self.permitted_actions.LIBRARY_MANAGE, library_item=library_item ) or \ self.allow_action( user, self.permitted_actions.LIBRARY_ADD, library_item=library_item ): diff -r 2751a34dbd4b -r ed6bfb311fb0 lib/galaxy/web/controllers/admin.py --- a/lib/galaxy/web/controllers/admin.py Wed Apr 22 10:57:07 2009 -0400 +++ b/lib/galaxy/web/controllers/admin.py Wed Apr 22 12:13:29 2009 -0400 @@ -935,7 +935,7 @@ trans.app.security_agent.set_all_library_permissions( library, permissions ) library.refresh() # Copy the permissions to the root folder - trans.app.security_agent.copy_library_permissions( library, library.root_folder, user=trans.get_user() ) + trans.app.security_agent.copy_library_permissions( library, library.root_folder ) msg = "Permissions updated for library '%s'" % library.name return trans.response.send_redirect( web.url_for( controller='admin', action='library', @@ -1058,7 +1058,7 @@ folder.add_folder( new_folder ) new_folder.flush() # New folders default to having the same permissions as their parent folder - trans.app.security_agent.copy_library_permissions( folder, new_folder, user=trans.get_user() ) + trans.app.security_agent.copy_library_permissions( folder, new_folder ) msg = "New folder named '%s' has been added to the library" % new_folder.name return trans.response.send_redirect( web.url_for( controller='admin', action='browse_library', diff -r 2751a34dbd4b -r ed6bfb311fb0 lib/galaxy/web/controllers/library.py --- a/lib/galaxy/web/controllers/library.py Wed Apr 22 10:57:07 2009 -0400 +++ b/lib/galaxy/web/controllers/library.py Wed Apr 22 12:13:29 2009 -0400 @@ -137,7 +137,7 @@ trans.app.security_agent.set_all_library_permissions( library, permissions ) library.refresh() # Copy the permissions to the root folder - trans.app.security_agent.copy_library_permissions( library, library.root_folder, user=trans.get_user() ) + trans.app.security_agent.copy_library_permissions( library, library.root_folder ) msg = "Permissions updated for library '%s'" % library.name return trans.response.send_redirect( web.url_for( controller='library', action='library', @@ -934,7 +934,7 @@ folder.add_folder( new_folder ) new_folder.flush() # New folders default to having the same permissions as their parent folder - trans.app.security_agent.copy_library_permissions( folder, new_folder, user=trans.get_user() ) + trans.app.security_agent.copy_library_permissions( folder, new_folder ) msg = "New folder named '%s' has been added to the library" % new_folder.name return trans.response.send_redirect( web.url_for( controller='library', action='browse_library',

1 0

[hg] galaxy 2360: Set users to external when admin created and u...
by Nate Coraor 22 Apr '09

22 Apr '09

details: http://www.bx.psu.edu/hg/galaxy/rev/2751a34dbd4b changeset: 2360:2751a34dbd4b user: Nate Coraor <nate(a)bx.psu.edu> date: Wed Apr 22 10:57:07 2009 -0400 description: Set users to external when admin created and use_remote_user is set 1 file(s) affected in this change: lib/galaxy/web/controllers/admin.py diffs (12 lines): diff -r 541ae0e5e599 -r 2751a34dbd4b lib/galaxy/web/controllers/admin.py --- a/lib/galaxy/web/controllers/admin.py Tue Apr 21 10:53:34 2009 -0400 +++ b/lib/galaxy/web/controllers/admin.py Wed Apr 22 10:57:07 2009 -0400 @@ -494,6 +494,8 @@ else: user = trans.app.model.User( email=email ) user.set_password_cleartext( password ) + if trans.app.config.use_remote_user: + user.external = True user.flush() trans.app.security_agent.create_private_user_role( user ) trans.app.security_agent.user_set_default_permissions( user, history=False, dataset=False )

1 0

Libraries
by SHAUN WEBB 22 Apr '09

22 Apr '09

I have a few questions regarding the new library feature: Firstly, when logged in I get the message "You are not authorized to access any libraries". How do I request access to available libraries? Secondly, we have our on local Galaxy install. How can we add a library of files to be used with our on tools? Can we control access to each folder in the library itself or only the entire library. Thanks Shaun Webb -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

[hg] galaxy 2359: Bug fixed in linear regression tool, and new o...
by gua110＠scofield.bx.psu.edu 21 Apr '09

21 Apr '09

details: http://www.bx.psu.edu/hg/galaxy/rev/541ae0e5e599 changeset: 2359:541ae0e5e599 user: guru date: Tue Apr 21 10:53:34 2009 -0400 description: Bug fixed in linear regression tool, and new option added to grouping tool. 4 file(s) affected in this change: tools/regVariation/linear_regression.py tools/regVariation/linear_regression.xml tools/stats/grouping.py tools/stats/grouping.xml diffs (186 lines): diff -r e9971a098d14 -r 541ae0e5e599 tools/regVariation/linear_regression.py --- a/tools/regVariation/linear_regression.py Tue Apr 21 10:45:13 2009 -0400 +++ b/tools/regVariation/linear_regression.py Tue Apr 21 10:53:34 2009 -0400 @@ -108,10 +108,15 @@ r.pdf( outfile2, 8, 8 ) if len(x_vals) == 1: #Simple linear regression case with 1 predictor variable sub_title = "Slope = %s; Y-int = %s" %(slope,yintercept) - r.plot(x=x_vals[0], y=y_vals, xlab="X", ylab="Y", sub=sub_title, main="Scatterplot with regression") - r.abline(a=yintercept, b=slope, col="red") + try: + r.plot(x=x_vals[0], y=y_vals, xlab="X", ylab="Y", sub=sub_title, main="Scatterplot with regression") + r.abline(a=yintercept, b=slope, col="red") + except: + pass else: r.pairs(dat, main="Scatterplot Matrix", col="blue") - -r.plot(linear_model) +try: + r.plot(linear_model) +except: + pass r.dev_off() diff -r e9971a098d14 -r 541ae0e5e599 tools/regVariation/linear_regression.xml --- a/tools/regVariation/linear_regression.xml Tue Apr 21 10:45:13 2009 -0400 +++ b/tools/regVariation/linear_regression.xml Tue Apr 21 10:53:34 2009 -0400 @@ -1,4 +1,4 @@ -<tool id="LinearRegression1" name="Perform Linear Regression"> +<tool id="LinearRegression1" name="Perform Linear Regression" version="1.0.1"> <description> </description> <command interpreter="python"> linear_regression.py @@ -48,7 +48,7 @@ **Note** -- This tool currently treats all predictor and response variables as continuous variables. +- This tool currently treats all predictor and response variables as continuous variables. Running the tool on categorical variables might result in incorrect results. - Rows containing non-numeric (or missing) data in any of the chosen columns will be skipped from the analysis. diff -r e9971a098d14 -r 541ae0e5e599 tools/stats/grouping.py --- a/tools/stats/grouping.py Tue Apr 21 10:45:13 2009 -0400 +++ b/tools/stats/grouping.py Tue Apr 21 10:53:34 2009 -0400 @@ -3,7 +3,7 @@ """ This tool provides the SQL "group by" functionality. """ -import sys, string, re, commands, tempfile, random +import sys, string, re, commands, tempfile, random, sets from rpy import * def stop_err(msg): @@ -48,7 +48,7 @@ for k,col in enumerate(cols): col = int(col)-1 - if ops[k] not in ['c', 'length', 'unique', 'random']: + if ops[k] not in ['c', 'length', 'unique', 'random', 'cuniq']: # We'll get here only if the user didn't choose 'Concatenate' or 'Count' or 'Count Distinct' or 'pick randmly', which are the # only aggregation functions that can be used on columns containing strings. try: @@ -104,7 +104,7 @@ valid = True # Before appending the current value, make sure it is numeric if the # operation for the column requires it. - if ops[i] not in ['c','length', 'unique','random']: + if ops[i] not in ['c','length', 'unique','random','cuniq']: try: float( fields[col].strip()) except: @@ -125,13 +125,18 @@ out_str = prev_item for i, op in enumerate( ops ): - rfunc = "r." + op - if op not in ['c','length','unique','random']: + if op == 'cuniq': + rfunc = "r.c" + else: + rfunc = "r." + op + if op not in ['c','length','unique','random','cuniq']: for j, elem in enumerate( prev_vals[i] ): prev_vals[i][j] = float( elem ) - rout = "%g" %( eval( rfunc )( prev_vals[i] )) + rout = eval( rfunc )( prev_vals[i] ) if rounds[i] == 'yes': rout = int(round(float(rout))) + else: + rout = '%g' %(float(rout)) else: if op != 'random': rout = eval( rfunc )( prev_vals[i] ) @@ -142,8 +147,14 @@ if op == 'unique': rfunc = "r.length" rout = eval( rfunc )( rout ) - out_str += "\t" + str(rout) - + if op in ['c', 'cuniq']: + if op == 'c': + out_str += "\t" + ','.join(rout) + else: + out_str += "\t" + ','.join(list(set(rout))) + else: + out_str += "\t" + str(rout) + print >>fout, out_str prev_item = item @@ -175,14 +186,19 @@ out_str = prev_item for i, op in enumerate(ops): - rfunc = "r." + op + if op == 'cuniq': + rfunc = "r.c" + else: + rfunc = "r." + op try: - if op not in ['c','length','unique','random']: + if op not in ['c','length','unique','random','cuniq']: for j, elem in enumerate( prev_vals[i] ): prev_vals[i][j] = float( elem ) - rout = '%g' %( eval( rfunc )( prev_vals[i] )) + rout = eval( rfunc )( prev_vals[i] ) if rounds[i] == 'yes': rout = int(round(float(rout))) + else: + rout = '%g' %(float(rout)) else: if op != 'random': rout = eval( rfunc )( prev_vals[i] ) @@ -192,8 +208,14 @@ if op == 'unique': rfunc = "r.length" - rout = eval( rfunc )( rout ) - out_str += "\t" + str( rout ) + rout = eval( rfunc )( rout ) + if op in ['c','cuniq']: + if op == 'c': + out_str += "\t" + ','.join(rout) + else: + out_str += "\t" + ','.join(list(set(rout))) + else: + out_str += "\t" + str( rout ) except: skipped_lines += 1 if not first_invalid_line: @@ -212,6 +234,8 @@ op = 'count_distinct' elif op == 'random': op = 'randomly_pick' + elif op == 'cuniq': + op = 'concat_distinct' msg += op + "[c" + cols[i] + "] " if skipped_lines > 0: msg+= "--skipped %d invalid lines starting with line %d. Value '%s' in column %d is not numeric." % ( skipped_lines, first_invalid_line, invalid_value, invalid_column ) diff -r e9971a098d14 -r 541ae0e5e599 tools/stats/grouping.xml --- a/tools/stats/grouping.xml Tue Apr 21 10:45:13 2009 -0400 +++ b/tools/stats/grouping.xml Tue Apr 21 10:53:34 2009 -0400 @@ -1,4 +1,4 @@ -<tool id="Grouping1" name="Group" version="1.5.0"> +<tool id="Grouping1" name="Group" version="1.6.0"> <description>data by a column and perform aggregate operation on other columns.</description> <command interpreter="python"> grouping.py @@ -23,6 +23,7 @@ <option value="length">Count</option> <option value="unique">Count Distinct</option> <option value="c">Concatenate</option> + <option value="cuniq">Concatenate Distinct</option> <option value="random">Randomly pick</option> </param> <param name="opcol" label="On column" type="data_column" data_ref="input1" /> @@ -87,7 +88,7 @@ - running this tool with **Group by column 1**, Operations **Mean on column 2** and **Concatenate on column 3** will return:: - chr10 1700.00 ['NM_11', 'NM_10'] - chr22 1533.33 ['NM_17', 'NM_19', 'NM_18'] + chr10 1700.00 NM_11,NM_10 + chr22 1533.33 NM_17,NM_19,NM_18 </help> </tool>

1 0