June 2011 - galaxy-dev - lists.galaxyproject.org

Early registration for BOSC ends tomorrow, Friday June 3
by Brad Chapman 02 Jun '11

02 Jun '11

If you haven't already registered for BOSC, now is your chance--after June 3, prices will go up! Registration for BOSC is through the ISMB main conference website: http://www.iscb.org/ismbeccb2011-registration#sigs . Since BOSC is a two-day SIG, the price is 2x the one-day SIG price listed on the ISMB website. You can register for BOSC without registering for the main ISMB conference, if you want. The preliminary BOSC schedule (subject to change) is now up at http://www.open-bio.org/wiki/BOSC_2011_Schedule (more details will be added soon). There is also a two day Codefest proceeding BOSC; please add yourself to the list of attendees if you are interested: http://www.open-bio.org/wiki/Codefest_2011 The BOSC talks have already been chosen, but we have spaces for last-minute posters. If you'd like your poster abstract to appear in the BOSC program, you should submit it now--see http://www.open-bio.org/wiki/BOSC_2011#Abstract_Submission_Information We hope to see you at BOSC! Nomi Harris Co-Chair, BOSC 2011

1 0

Appropriate place to ask questions about your NGLIMS for Galaxy
by scott.coutts＠monash.edu 02 Jun '11

02 Jun '11

Hi, I'm a newbie when it comes to this kind of thing, so I was wondering whether you can tell me the most appropriate place to ask questions about making some modifications to your nglims extension for galaxy? Should they be posted to the normal galaxy development list? Should I ask you directly? Cheers, Scott.

2 1

library_upload_from_import_dir
by George, David 02 Jun '11

02 Jun '11

Hi, I'm a Galaxy and Python newbie and I'm working on project that desires to upload files from a directory to the Galaxy server. We'd like to physically copy the files rather than maintain references to them. I'm starting by following the examples in scripts/api/README. All the display and library_create_library examples work. However, the library_upload_from_import_dir fails. In my universe...ini, library_import_dir is defined to be /home/dgeorge/galaxy/import and there is a bed directory under that with a short file, bed1.bed in the correct format. I'm not sure what to provide for the last parameter, the dbkey - maybe that's my problem? I ran: ./library_upload_from_import_dir.py 59d2fd4e020e178f8c48e61150e513c2 http://localhost:8080/api/libraries/f597429621d6eb2b/contents c6ca0ddb55be603a10e891fff2e902c3 bed bed hg19 And I get the error: Exception happened during processing of request from ('127.0.0.1', 38857) Traceback (most recent call last): File "/home/dgeorge/galaxy/galaxy-central/eggs/Paste-1.6-py2.6.egg/paste/http server.py", line 1053, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.6/SocketServer.py", line 618, in __init__ self.finish() File "/usr/lib/python2.6/SocketServer.py", line 661, in finish self.wfile.flush() File "/usr/lib/python2.6/socket.py", line 297, in flush self._sock.sendall(buffer(data, write_offset, buffer_size)) error: [Errno 32] Broken pipe Any ideas or suggestions? Thanks! David George Staff Software Engineer Illumina, Inc. 25861 Industrial Blvd. Hayward, CA 94545 Tel: 510-670-9326 Fax: 510-670-9302 Email: dgeorge(a)illumina.com <mailto:dgeorge@illumina.com>

2 1

Uploading Zip Files
by SHAUN WEBB 02 Jun '11

02 Jun '11

Hi, I have a tool that takes a zipped archive as input, finds all the sequence files (~90 of them), separates them by barcode, does a little extra processing and outputs a single fasta file. I would like this to run on Galaxy in a workflow with a few other tools. I know there have been discussions about uploading zip files to Galaxy before and that currently Galaxy automatically unzips a file and uploads the first file in the archive. Are there any plans to change this behaviour and be able to upload a zipped file without decompressing. Thanks Shaun -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

Re: [galaxy-dev] suggestion for multithreading
by Assaf Gordon 02 Jun '11

02 Jun '11

(moved to galaxy-dev) Nate Coraor wrote, On 06/02/2011 01:31 PM: > Peter Cock wrote: >> On Thu, Jun 2, 2011 at 6:23 PM, Nate Coraor <nate(a)bx.psu.edu> wrote: >>> >>> pbs.py then knows to translate '<resource type="cores">8</resource>' to >>> '-l nodes=1:ppn=8'. >>> >>> Your tool can access that value a bunch, like $__resources__.cores. >>> >>> The same should be possible for other consumables. >>> Just a thought here: The actual parameters that are passed to the scheduler are not necessarily hard-coded. Meaning, at least with SGE, specifying the number of cores can be: qsub -pe threads=8 or qsub -pe cores=8 or qsub -pe jiffies=8 and same thing for memory limitation (e.g. "-l virtual_free=800M"). The reason is that those resources (e.g. "threads", "cores", "virtual_free") are just identifiers, and they are created and configured by whomever installed SGE - they are not built-in or hard-coded). So just be careful in your design/implementation when automatically translating XML resources to hard-coded parameters. If you do hard-code them, just make sure the specifically document it (i.e. Galaxy expect the SGE threads parameter to be "-pe threads=8" and nothing else). -gordon

2 1

Problem with Torque/Maui
by Marco Moretto 02 Jun '11

02 Jun '11

Hi all, first of all, thanks to the Galaxy Team for this really useful software. Actually I don't really know if my problem is related with Galaxy or with Torque/Maui but I didn't find any solution looking in both Torque and Maui user lists, so I hope that some of you with more experience could give me some good advices. I'm trying to set up Galaxy in a small local virtual environment in order to test it. I started with 2 virtual ubuntu servers called galaxy1 and galaxy2. On galaxy1 I succesfully installed Galaxy, Apache, Torque and Maui. I'm using Postgres ad DBMS. It is installed on another "real" DB server. The virtual server galaxy2 is used as node. Galaxy is working like a charm locally but when I try to use Torque problems arise. Torque alone works correctly. That means that I can submit a job with qsub and everything works. The 2 virtual server (galaxy1 and galaxy2) share a directory (through NFS) in which I installed Galaxy following the "unified method" from the documentation. Now, as I said, Galaxy alone works, Torque/Maui alone works. When I put the two together nothing works. As a test I upload (using local runner) a gff file. Then I try to make a filter to the gff using "Filter and sort -> Extract features". When I run this tool the corresponding job on the Torque queue runs forever in Hold state. I report some output from diagnose programs: The diagnose -j reports the following: Name State Par Proc QOS WCLimit R Min User Group Account QueuedTime Network Opsys Arch Mem Disk Procs Class Features 29 Hold DEF 1 DEF 1:00:00 0 1 galaxy galaxy - 00:02:36 [NONE] [NONE] [NONE] >=0 >=0 NC0 [batch:1] [NONE] While the showq command reports ACTIVE JOBS-------------------- JOBNAME USERNAME STATE PROC REMAINING STARTTIME 0 Active Jobs 0 of 1 Processors Active (0.00%) IDLE JOBS---------------------- JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME 0 Idle Jobs BLOCKED JOBS---------------- JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME 29 galaxy Hold 1 1:00:00 Wed May 4 03:56:40 The checkjob reports: checking job 29 State: Hold Creds: user:galaxy group:galaxy class:batch qos:DEFAULT WallTime: 00:00:00 of 1:00:00 SubmitTime: Wed May 4 03:56:40 (Time Queued Total: 00:03:07 Eligible: 00:00:01) The qstat -f reports Job Id: 33.galaxy1.research.intra.ismaa.it Job_Name = 27_Extract_features1_marco.moretto(a)iasma.it Job_Owner = galaxy(a)galaxy1.research.intra.ismaa.it job_state = W queue = batch server = galaxy1.research.intra.ismaa.it ctime = Wed May 4 04:56:36 2011 Error_Path = galaxy1:/mnt/equallogic1/galaxy/galaxy-dist/database/pbs/27.e exec_host = galaxy2/0 exec_port = 15003 Execution_Time = Wed May 4 05:26:41 2011 mtime = Wed May 4 04:56:37 2011 Output_Path = galaxy1:/mnt/equallogic1/galaxy/galaxy-dist/database/pbs/27. o qtime = Wed May 4 04:56:36 2011 Resource_List.neednodes = 1 Resource_List.nodect = 1 Resource_List.nodes = 1 Resource_List.walltime = 01:00:00 stagein = /mnt/equallogic1/galaxy/tmp/dataset_18.dat@galaxy1 :/mnt/equallog ic1/galaxy/galaxy-dist/database/files/000/dataset_18.dat, /mnt/equallogic1/galaxy/tmp/dataset_30.dat@galaxy1 :/mnt/equallogic1/g alaxy/galaxy-dist/database/files/000/dataset_30.dat stageout = /mnt/equallogic1/galaxy/galaxy-dist/database/files/000/dataset_ 30.dat@galaxy1 :/mnt/equallogic1/galaxy/galaxy-dist/database/files/000/ dataset_30.dat substate = 37 Variable_List = PBS_O_QUEUE=batch, PBS_O_HOST=galaxy1.research.intra.ismaa.it euser = galaxy egroup = galaxy hashname = 33.galaxy1.research.intra.ismaa.it queue_rank = 33 queue_type = E StartDate: -00:03:06 Wed May 4 03:56:41 Total Tasks: 1 Req[0] TaskCount: 1 Partition: DEFAULT Network: [NONE] Memory >= 0 Disk >= 0 Swap >= 0 Opsys: [NONE] Arch: [NONE] Features: [NONE] IWD: [NONE] Executable: [NONE] Bypass: 0 StartCount: 1 PartitionMask: [ALL] PE: 1.00 StartPriority: 1 cannot select job 29 for partition DEFAULT (non-idle state 'Hold') and finally the tracejob reports /var/spool/torque/server_priv/accounting/20110504: Permission denied /var/spool/torque/mom_logs/20110504: No such file or directory /var/spool/torque/sched_logs/20110504: No such file or directory Job: 33.galaxy1.research.intra.ismaa.it 05/04/2011 04:56:36 S enqueuing into batch, state 1 hop 1 05/04/2011 04:56:36 S Job Queued at request of galaxy(a)galaxy1.research.intra.ismaa.it, owner = galaxy(a)galaxy1.research.intra.ismaa.it, job name = 27_Extract_features1_marco.moretto(a)iasma.it, queue = batch 05/04/2011 04:56:37 S Job Run at request of galaxy(a)galaxy1.research.intra.ismaa.it 05/04/2011 04:56:41 S Email 's' to galaxy(a)galaxy1.research.intra.ismaa.it failed: Child process 'sendmail -f adm galaxy(a)galaxy1.research.intra.ismaa.it' returned 127 (errno 10:No child processes) The only clear thing to me is that after the submission the scheduler puts it in a Hold state. But I cannot understand why. I also try to run the Galaxy-generated sh script with qsub. Following the Galaxy log: galaxy.jobs.runners.pbs DEBUG 2011-05-04 04:56:36,748 (27) submitting file /mnt/equallogic1/galaxy/galaxy-dist/database/pbs/27.sh I copied and run the 27.sh script with the command: qsub 27.sh And the job runs correctly. So what it is not clear to me is if the problem is related to Torque/Maui or is related to the way in which the job is submitted from Galaxy. Sorry for the very long e-mail and thank you very much for any help. --- Marco

2 1

TSS for list of genes
by shamsher jagat 02 Jun '11

02 Jun '11

I am not sure if this is the right place to post this message, I want to download TSS fro a list of around 1000 human genes. Is this something I can do in Galaxy? Thanks.

1 0

Error: unable to read Galaxy config
by Lewis, Brian Andrew 02 Jun '11

02 Jun '11

After following the instructions in the wiki for setting up scaling/load balancing, I was testing out submitting a job and got this error when trying to pull data from USCS Main: Traceback (most recent call last): File "/usr/local/galaxy-dist/tools/data_source/data_source.py", line 6, in from galaxy.util import gzip_magic File "/usr/local/galaxy-dist/lib/galaxy/util/__init__.py", line 20, in pkg_resources.require( 'docutils' ) File "/usr/local/galaxy-dist/lib/galaxy/eggs/__init__.py", line 400, in require c = Crate() File "/usr/local/galaxy-dist/lib/galaxy/eggs/__init__.py", line 259, in __init__ self.galaxy_config = GalaxyConfig() File "/usr/local/galaxy-dist/lib/galaxy/eggs/__init__.py", line 358, in __init__ raise Exception( "error: unable to read Galaxy config from %s" % GalaxyConfig.config_file ) Exception: error: unable to read Galaxy config from /usr/local/galaxy-dist/0 I'm running Galaxy on a RHEL6 server and I have one web server and one runner set up using Apache as a proxy server. Thanks, Brian

2 4

Migration error: fields in MySQL
by John Eppley 02 Jun '11

02 Jun '11

I had an error upgrading my galaxy instance. I got the following exception while migrating the db (during step 64->65): sqlalchemy.exc.ProgrammingError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'fields FROM form_definition' at line 1") u'SELECT id, fields FROM form_definition' [] It seems my version (4.1.22-log) of MySQL did not like 'fields' as a column name. If I alias the formdefinition as f and us f.fields, the error goes away. I also had to modify migration 76 for the same reason. Here is my diff of the migrations dir: diff -r 50e249442c5a lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py --- a/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py Thu Apr 07 08:39:07 2011 -0400 +++ b/lib/galaxy/model/migrate/versions/0065_add_name_to_form_fields_and_values.py Fri Apr 15 11:09:26 2011 -0400 @@ -39,7 +39,7 @@ return '' # Go through the entire table and add a 'name' attribute for each field # in the list of fields for each form definition - cmd = "SELECT id, fields FROM form_definition" + cmd = "SELECT f.id, f.fields FROM form_definition f" result = db_session.execute( cmd ) for row in result: form_definition_id = row[0] @@ -53,7 +53,7 @@ field[ 'helptext' ] = field[ 'helptext' ].replace("'", "''").replace('"', "") field[ 'label' ] = field[ 'label' ].replace("'", "''") fields_json = to_json_string( fields_list ) - cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" %( fields_json, form_definition_id ) + cmd = "UPDATE form_definition f SET f.fields='%s' WHERE f.id=%i" %( fields_json, form_definition_id ) db_session.execute( cmd ) # replace the values list in the content field of the form_values table with a name:value dict cmd = "SELECT form_values.id, form_values.content, form_definition.fields" \ @@ -112,7 +112,7 @@ cmd = "UPDATE form_values SET content='%s' WHERE id=%i" %( to_json_string( values_list ), form_values_id ) db_session.execute( cmd ) # remove name attribute from the field column of the form_definition table - cmd = "SELECT id, fields FROM form_definition" + cmd = "SELECT f.id, f.fields FROM form_definition f" result = db_session.execute( cmd ) for row in result: form_definition_id = row[0] @@ -124,5 +124,5 @@ for index, field in enumerate( fields_list ): if field.has_key( 'name' ): del field[ 'name' ] - cmd = "UPDATE form_definition SET fields='%s' WHERE id=%i" %( to_json_string( fields_list ), form_definition_id ) + cmd = "UPDATE form_definition f SET f.fields='%s' WHERE id=%i" %( to_json_string( fields_list ), form_definition_id ) db_session.execute( cmd ) diff -r 50e249442c5a lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py --- a/lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py Thu Apr 07 08:39:07 2011 -0400 +++ b/lib/galaxy/model/migrate/versions/0076_fix_form_values_data_corruption.py Fri Apr 15 11:09:26 2011 -0400 @@ -32,7 +32,7 @@ def upgrade(): print __doc__ metadata.reflect() - cmd = "SELECT form_values.id as id, form_values.content as field_values, form_definition.fields as fields " \ + cmd = "SELECT form_values.id as id, form_values.content as field_values, form_definition.fields as fdfields " \ + " FROM form_definition, form_values " \ + " WHERE form_values.form_definition_id=form_definition.id " \ + " ORDER BY form_values.id" @@ -46,7 +46,7 @@ except Exception, e: corrupted_rows = corrupted_rows + 1 # content field is corrupted - fields_list = from_json_string( _sniffnfix_pg9_hex( str( row['fields'] ) ) ) + fields_list = from_json_string( _sniffnfix_pg9_hex( str( row['fdfields'] ) ) ) field_values_str = _sniffnfix_pg9_hex( str( row['field_values'] ) ) try: #Encoding errors? Just to be safe. -j

2 1

Support for subdirs in dataset extra_files_path
by Jim Johnson 02 Jun '11

02 Jun '11

Request is issue#494 https://bitbucket.org/galaxy/galaxy-central/issue/494/support-sub-dirs-in-e… I'm finding that some qiime metagenomics applications build HTML results with an inherent directory structure. For some other applications, e.g. FastQC, I've been able to flatten the hierarchy and edit the html, but that appears problematic for qiime. Galaxy hasn't supported a dataset extra_files_path hierarchy, though the developers don't seem opposed to the idea: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-October/003605.html I added a route in lib/galaxy/web/buildapp.py and modified the dataset download code to traverse a hierarchy in lib/galaxy/web/controllers/dataset.py I don't think these add any security vulnerabilities, (I tried the obvious ../../ ). $ hg diff lib/galaxy/web/buildapp.py diff -r 6ae06d89fec7 lib/galaxy/web/buildapp.py --- a/lib/galaxy/web/buildapp.py Wed Mar 16 09:01:57 2011 -0400 +++ b/lib/galaxy/web/buildapp.py Wed Mar 16 10:24:13 2011 -0500 @@ -94,6 +94,8 @@ webapp.add_route( '/async/:tool_id/:data_id/:data_secret', controller='async', action='index', tool_id=None, data_id=None, data_secret=None ) webapp.add_route( '/:controller/:action', action='index' ) webapp.add_route( '/:action', controller='root', action='index' ) + # allow for subdirectories in extra_files_path + webapp.add_route( '/datasets/:dataset_id/display/{filename:.+?}', controller='dataset', action='display', dataset_id=None, filename=None) webapp.add_route( '/datasets/:dataset_id/:action/:filename', controller='dataset', action='index', dataset_id=None, filename=None) webapp.add_route( '/display_application/:dataset_id/:app_name/:link_name/:user_id/:app_action/:action_param', controller='dataset', action='display_application', dataset_id=None, user_id=None, app_name = None, link_name = None, app_action = None, action_param = None ) webapp.add_route( '/u/:username/d/:slug', controller='dataset', action='display_by_username_and_slug' ) $ $ hg diff lib/galaxy/web/controllers/dataset.py diff -r 6ae06d89fec7 lib/galaxy/web/controllers/dataset.py --- a/lib/galaxy/web/controllers/dataset.py Wed Mar 16 09:01:57 2011 -0400 +++ b/lib/galaxy/web/controllers/dataset.py Wed Mar 16 10:24:29 2011 -0500 @@ -266,17 +266,18 @@ log.exception( "Unable to add composite parent %s to temporary library download archive" % data.file_name) msg = "Unable to create archive for download, please report this error" messagetype = 'error' - flist = glob.glob(os.path.join(efp,'*.*')) # glob returns full paths - for fpath in flist: - efp,fname = os.path.split(fpath) - try: - archive.add( fpath,fname ) - except IOError: - error = True - log.exception( "Unable to add %s to temporary library download archive" % fname) - msg = "Unable to create archive for download, please report this error" - messagetype = 'error' - continue + for root, dirs, files in os.walk(efp): + for fname in files: + fpath = os.path.join(root,fname) + rpath = os.path.relpath(fpath,efp) + try: + archive.add( fpath,rpath ) + except IOError: + error = True + log.exception( "Unable to add %s to temporary library download archive" % rpath) + msg = "Unable to create archive for download, please report this error" + messagetype = 'error' + continue if not error: if params.do_action == 'zip': archive.close()

2 1