July 2010 - galaxy-dev - lists.galaxyproject.org

stupid logging question
by Michael Pheasant 07 Sep '10

07 Sep '10

Using the run.sh script, and the default 'log_level = DEBUG' universe_wsgi.ini file setting gives a huge amount of output. Changing the ini to either log_level = INFO, WARNING or ERROR and i get the same output, including the INFO messages every second. I assumed that WARNING or ERROR would not give the INFO messages; is there some way to not see these? Cheers m -- Michael Pheasant Software Engineer Queensland Facility for Advanced Bioinformatics Level 6, QBP University of Queensland, QLD 4072 T: +61 (0)7 3346 2070 F: +61 (0)7 3346 2101 www.qfab.org

4 8

Server error from history view
by simon andrews (BI) 11 Aug '10

11 Aug '10

I'm making pretty good progress setting up our galaxy server, but have hit a snag. Since moving from an SQLite to a MySQL backend I'm getting an internal server error from the history panel on our galaxy instance. Running the galaxy server from a console gives me the following python traceback: ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 60795) Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 320, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 615, in __init__ self.handle() File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 287, in wsgi_execute self.wsgi_start_response) File "/usr/lib/python2.6/site-packages/paste/urlmap.py", line 203, in __call__ return app(environ, start_response) File "/data/private/galaxy/galaxy_dist/lib/galaxy/web/framework/middleware/xforwardedhost.py", line 21, in __call__ return self.app( environ, start_response ) File "/usr/lib/python2.6/site-packages/paste/deploy/config.py", line 164, in __call__ app_iter = self.application(environ, start_response) File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__ return self.application(environ, replacement_start_response) File "/usr/lib/python2.6/site-packages/paste/exceptions/errormiddleware.py", line 154, in __call__ exc_info) File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 67, in replacement_start_response return start_response(status, headers) File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 166, in wsgi_start_response assert 0, "Attempt to set headers a second time w/o an exc_info" AssertionError: Attempt to set headers a second time w/o an exc_info ---------------------------------------- Any ideas? Thanks Simon.

2 6

Sort tab error?
by Minh Bui 06 Aug '10

06 Aug '10

Hi, I always get this error when I use Sort feature *"An error occurred running this job: sort: multi-character tab `$\t'"* Even I already use Text Manipulation->Convert to convert my table to Tabs format. Here is the screenshot of what happen (I want to sort C2 on job #12) : http://img408.imageshack.us/img408/7906/screenshotbu.png Could someone please show me how to fix this? Thank you.

5 14

adding another UCSC mirror to display
by Davide Cittaro 04 Aug '10

04 Aug '10

Hi again, this may be a FAQ, sorry for that... How can I add an alternate UCSC mirror site (our local mirror) to galaxy so that everything that could be displayed at UCSC-main will have the UCSC-localmirror option? Besides... how can I remove the bx main mirror? I've tried to add a ${GALAXYROOT}/tool-data/shared/campus/campus_build_sites.txt but that's not enough... I've modified the ucsc/bam.xml file adding another dynamic_link section but that's not enough... any hint? Thanks again d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

2 1

Upload issue: 'Upload a directory of files' and compressed files
by padioleau 02 Aug '10

02 Aug '10

Dear Galaxy developer, Using the data-library upload, 'Upload a directory of files', I encounter an issue. When uploading compressed files, without copying them in galaxy folder, they are uncompressed in my folder but the file name extension is not changed. This mean that I have files with 'gz' extension that are in fact uncompressed. Is there a possibility to avoid this issue? I'm using tools which can use compressed files, so for me the best would be to just add the compressed file in my data-library. I believe galaxy can accept compressed files, but I didn't found any explanation in your wiki (index: compress, gzip or zip) to configure galaxy this way. Thanks, Ismael -- Padioleau Ismael EMBL Heidelberg ( V205)

2 1

Re: [galaxy-dev] Fwd: Re: Help about zip upload (esd files)
by Adhemar 30 Jul '10

30 Jul '10

Dear Nate, We're trying to implement different tools to handle sequences generated by Megabace/AB1 sequencers. The very first step in this process is to load the chromatograms provided by the machines. The thing is these machines produce an individual file per sequence and the phred software (base caller) receive as parameters a compressed file or a directory path. Since galaxy 'get data' feature loads only individual files and we're not able to implement phred to do such things, what options are left? Thank you so much, Adhemar On 07/29/2010 01:57 PM, Eric Aguiar wrote: > > > -------- Original Message -------- > Subject: Re: [galaxy-dev] Help about zip upload (esd files) > Date: Wed, 28 Jul 2010 12:09:06 -0400 > From: > To: Eric Aguiar <eric(a)cebio.org> > CC: List > > > > Eric Aguiar wrote: > > Dear, > > > > I followed all steps in the galaxy tutorial, but I didn't have success. > > I'm trying to create a datatype for megabase chromatograms (.esd) very > > similar to the Ab1 ones. > > > > Here is my configurations. > > > > > > *1 - Creating datatypes in datatypes_conf.xml* > > > > <datatype extension="zip" type="galaxy.datatypes.binary:Esd" > > mimetype="application/zip" display_in_upload="true"/> > > <datatype extension="esd" type="galaxy.datatypes.binary:Esd" > > mimetype="application/octet-stream" display_in_upload="true"/> > > > > *2 - Defining types in lib/galaxy/datatypes/binary.py* > > > > class Esd( Binary ): > > """Class describing an ab1 binary sequence file""" > > file_ext = "esd" > > > > def set_peek( self, dataset, is_multi_byte=False ): > > if not dataset.dataset.purged: > > dataset.peek = "Binary chromatograms sequence file" > > dataset.blurb = data.nice_size( dataset.get_size() ) > > else: > > dataset.peek = 'file does not exist' > > dataset.blurb = 'file purged from disk' > > def display_peek( self, dataset ): > > try: > > return dataset.peek > > except: > > return "Binary esd sequence file (%s)" % ( data.nice_size( > > dataset.get_size() ) ) > > > > > > class Zip( Binary ): > > """Class describing a zip archive of binary sequence files""" > > file_ext = "zip" > > > > def set_peek( self, dataset, is_multi_byte=False ): > > if not dataset.dataset.purged: > > zip_file = zipfile.ZipFile( dataset.file_name, "r" ) > > num_files = len( zip_file.namelist() ) > > dataset.peek = "Archive of %s binary sequence files" % ( > > str( num_files ) ) > > dataset.blurb = data.nice_size( dataset.get_size() ) > > else: > > dataset.peek = 'file does not exist' > > dataset.blurb = 'file purged from disk' > > def display_peek( self, dataset ): > > try: > > return dataset.peek > > except: > > return "Binary sequence file archive (%s)" % ( > > data.nice_size( dataset.get_size() ) ) > > def get_mime( self ): > > """Returns the mime type of the datatype""" > > return 'application/zip' > > > > > > When I'm going to send the file in zip format (.esd files compressed),it > > shows me the following error: > > *"An error occurred running this job: Invalid 'File Format' for archive > > consisting of binary files - use 'Binseq.zip'"* > > Hi Eric, > > Have you updated to a recent revision of Galaxy? Binseq.zip and the > code that prevents uploading zip files has been removed. Currently, > though, a zip file can only contain one member. > > --nate > > > > > I tried somethings, but I don't have success. > > > > Thank you, > > -- > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > galaxy-dev mailing list > > galaxy-dev(a)lists.bx.psu.edu > > http://lists.bx.psu.edu/listinfo/galaxy-dev > -- Adhemar Zerlotini Neto, PhD FIOCRUZ - Fundação Oswaldo Cruz CPqRR - Centro de Pesquisas René Rachou http://www.cpqrr.fiocruz.br http://www.cebio.org +55 31 3337 3863 +55 31 3337 3649

2 1

DRMMA and SGE problem
by George Chalkidis 29 Jul '10

29 Jul '10

Hi Galaxy-Dev-Team! First of all, I would like to thank you for providing the community with such a great framework! I've downloaded and set up locally the latest packed release. When I want to use Galaxy with the Sun Grid Engine, I run into problems. It looks like Galaxy isn't able to fully control the Grid Engine and I think it might be a version issue with DRMMA and our current SGE. Jobs get submitted, appear in the queue and are executed, but their status is not returned to Galaxy and therefore the job's status on the website remains always "Job is waiting to run". Here's a more detailed error-message: ------------------------------------------------------ [environment] CentOS release 5.4 (Final) SGE 6.2u5 Python 2.5.2 Jun 30th source tar.gz file http://bitbucket.org/galaxy/galaxy-dist/get/tip.tar.gz --------------------------------------------------------- [config] $ cat universe_wsgi.ini |grep sge # currently available are 'pbs' and 'sge'. start_job_runners = sge #default_cluster_job_runner = sge://default/mjobs.q/ cant lunch service default_cluster_job_runner = sge:///mjobs.q/ $ grep -5 SGE_ROOT eggs.ini [general] repository = http://eggs.g2.bx.psu.edu/new ; these eggs must be scrambled for your local environment no_auto = pbs_python DRMAA_python SGE_ROOT = /home/geadmin/N1GE python scripts/scramble.py DRMAA_python [eggs:platform] bx_python = 0.5.0 Cheetah = 2.2.2 DRMAA_python = 0.2 --------------------------------------------------------- [Error Messages] galaxy.jobs DEBUG 2010-07-16 17:30:24,794 job 10 dispatched galaxy.jobs.runners.sge DEBUG 2010-07-16 17:30:27,480 (10) submitting file /share1/home/icgc/galaxy-dist/database/pbs/galaxy_10.sh galaxy.jobs.runners.sge DEBUG 2010-07-16 17:30:27,480 (10) command is: perl /share1/home/icgc/galaxy-dist/tools/myTools/toolExample.pl /share1/home/icgc/galaxy-dist/database/files/000/dataset_3.dat /share1/home/icgc/galaxy-dist/database/files/000/dataset_10.dat galaxy.jobs.runners.sge DEBUG 2010-07-16 17:30:27,522 (10) queued in mjobs.q queue as 9440764 202.175.149.92 - - [16/Jul/2010:17:30:26 +0900] "POST /root/history_item_updates HTTP/1.1" 200 - "http://gw01.hgc.jp:8090/history" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6" galaxy.jobs.runners.sge ERROR 2010-07-16 17:30:29,524 (10/9440764) Unable to check job status Traceback (most recent call last): File "/share1/home/icgc/galaxy-dist/lib/galaxy/jobs/runners/sge.py", line 277, in check_watched_items state = self.ds.getJobProgramStatus( job_id ) File "/usr/local/lib/python2.5/site-packages/DRMAA.py", line 395, in getJobProgramStatus return ps TypeError: writelines() argument must be a sequence of strings galaxy.jobs.runners.sge WARNING 2010-07-16 17:30:29,525 (10/9440764) job will now be errored galaxy.jobs.runners.sge ERROR 2010-07-16 17:30:29,526 Uncaught exception failing job Traceback (most recent call last): File "/share1/home/icgc/galaxy-dist/lib/galaxy/jobs/runners/sge.py", line 138, in run_next self.fail_job( obj ) File "/share1/home/icgc/galaxy-dist/lib/galaxy/jobs/runners/sge.py", line 338, in fail_job self.stop_job( self.sa_session.query( self.app.model.Job ).get( sge_job_state.job_wrapper.job_id ) ) AttributeError: 'SGEJobRunner' object has no attribute 'sa_session' [when I use sample code] #!/usr/bin/env python import DRMAA import time import os def main(): """Submit a job, and check its progress. Note, need file called sleeper.sh in home directory. An example: echo 'Hello World $1' sleep 30s """ s=DRMAA.Session() s.init() print 'Creating job template' jt = s.createJobTemplate() jt.remoteCommand = os.getcwd() + '/sleeper.sh' jt.args = ['42','Simon says:'] jt.joinFiles=True jt.outputPath=":"+DRMAA.JobTemplate.HOME_DIRECTORY+'/tmp/JOB_OUT' jobid = s.runJob(jt) print 'Your job has been submitted with id ' + jobid # Who needs a case statement when you have dictionaries? decodestatus = { DRMAA.Session.UNDETERMINED: 'process status cannot be determined', DRMAA.Session.QUEUED_ACTIVE: 'job is queued and active', DRMAA.Session.SYSTEM_ON_HOLD: 'job is queued and in system hold', DRMAA.Session.USER_ON_HOLD: 'job is queued and in user hold', DRMAA.Session.USER_SYSTEM_ON_HOLD: 'job is queued and in user and system hold', DRMAA.Session.RUNNING: 'job is running', DRMAA.Session.SYSTEM_SUSPENDED: 'job is system suspended', DRMAA.Session.USER_SUSPENDED: 'job is user suspended', DRMAA.Session.DONE: 'job finished normally', DRMAA.Session.FAILED: 'job finished, but failed', } for ix in range(10): print ix print 'Checking ' + str(ix) + ' of 10 times' status = s.getJobProgramStatus(jobid) print decodestatus.get(status) time.sleep(5) print 'Cleaning up' s.deleteJobTemplate(jt) s.exit() if __name__=='__main__': main() $ ./test Creating job template Your job has been submitted with id 9440771 0 Checking 0 of 10 times Traceback (most recent call last): File "./test", line 52, in <module> main() File "./test", line 43, in main status = s.getJobProgramStatus(jobid) File "/usr/local/lib/python2.5/site-packages/DRMAA.py", line 393, in getJobProgramStatus eno, ps, estr = cDRMAA.drmaa_job_ps(jobName) ValueError: need more than 2 values to unpack [debug] print cDRMAA.drmaa_job_ps(jobName) eno, ps, estr = cDRMAA.drmaa_job_ps(jobName) => ([0, 16], '') Thank you for taking time! Hopefully, you can give me some hints on how to resolve this issue. Cheers, George Chalkidis

2 2

Textbox Size
by Dennis Gascoigne 29 Jul '10

29 Jul '10

I have tools that require a wide textbox area. It seems however that there is a limit coded into Galaxy soewhere of about 60 characters. Can you advise where it is so I can bump it up please? All i am trying to do is have a textbox with area 20x170. Cheers Dennis

2 1

Setup failed due to simplejson version
by simon andrews (BI) 29 Jul '10

29 Jul '10

I've just tried an initial installation of Galaxy (galaxy_dist) on a Fedora 13 machine. When I try to run the setup.sh script I get the error below: Creating tool-data/shared/jars Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 31, in <module> c.resolve() # Only fetch eggs required by the config File "/data/private/galaxy/galaxy_dist/lib/galaxy/eggs/__init__.py", line 284, in resolve egg.resolve() File "/data/private/galaxy/galaxy_dist/lib/galaxy/eggs/__init__.py", line 172, in resolve r = pkg_resources.working_set.resolve( ( self.distribution.as_requirement(), ), env, self.fetch ) File "/data/private/galaxy/galaxy_dist/lib/pkg_resources.py", line 565, in res olve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: simplejson==1.5 Fedora 13 comes with simplejson v2.1.1. Is there really a strict requirement on v1.5 or should the test be >=1.5? Is there a way to parallel install different versions of this package if 1.5 is strictly required? Thanks Simon.

2 3

LSF/DRMAA Support
by Nate Coraor 29 Jul '10

29 Jul '10

Hello, Support for the LSF distributed resource manager (DRM) has recently been committed to our development repository in changeset 4071:fd9514c5349f. This support has actually been implemented through the use of the python-drmaa module and none of the code is DRM-specific. This means that any DRM which implements a DRMAA library should be able to be supported now (including PBS Pro). For more information on how to configure a DRMAA library to work with your DRM, please see: http://www.drmaa.org/ The code is heavily based on the former SGE job runner, which will soon be deprecated in favor of the new DRMAA job runner. Thanks to John Brunelle for adapting and testing the original implementation. For information on how to set up Galaxy to use the DRMAA job runner, please see: http://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster This changeset is not yet available in our stable distribution repository, although it should be soon. In the meantime, you can grab it from our development repository if you'd like: http://bitbucket.org/galaxy/galaxy-central/ This code has had minimal testing and should not be used for a production server. But please do test it out and report any bugs (either here or in our issue tracker in Bitbucket). Thanks! --nate

1 0