- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Fwd: Re: Help about zip upload (esd files)
by Adhemar 30 Jul '10

30 Jul '10

Dear Nate, We're trying to implement different tools to handle sequences generated by Megabace/AB1 sequencers. The very first step in this process is to load the chromatograms provided by the machines. The thing is these machines produce an individual file per sequence and the phred software (base caller) receive as parameters a compressed file or a directory path. Since galaxy 'get data' feature loads only individual files and we're not able to implement phred to do such things, what options are left? Thank you so much, Adhemar On 07/29/2010 01:57 PM, Eric Aguiar wrote: > > > -------- Original Message -------- > Subject: Re: [galaxy-dev] Help about zip upload (esd files) > Date: Wed, 28 Jul 2010 12:09:06 -0400 > From: > To: Eric Aguiar <eric(a)cebio.org> > CC: List > > > > Eric Aguiar wrote: > > Dear, > > > > I followed all steps in the galaxy tutorial, but I didn't have success. > > I'm trying to create a datatype for megabase chromatograms (.esd) very > > similar to the Ab1 ones. > > > > Here is my configurations. > > > > > > *1 - Creating datatypes in datatypes_conf.xml* > > > > <datatype extension="zip" type="galaxy.datatypes.binary:Esd" > > mimetype="application/zip" display_in_upload="true"/> > > <datatype extension="esd" type="galaxy.datatypes.binary:Esd" > > mimetype="application/octet-stream" display_in_upload="true"/> > > > > *2 - Defining types in lib/galaxy/datatypes/binary.py* > > > > class Esd( Binary ): > > """Class describing an ab1 binary sequence file""" > > file_ext = "esd" > > > > def set_peek( self, dataset, is_multi_byte=False ): > > if not dataset.dataset.purged: > > dataset.peek = "Binary chromatograms sequence file" > > dataset.blurb = data.nice_size( dataset.get_size() ) > > else: > > dataset.peek = 'file does not exist' > > dataset.blurb = 'file purged from disk' > > def display_peek( self, dataset ): > > try: > > return dataset.peek > > except: > > return "Binary esd sequence file (%s)" % ( data.nice_size( > > dataset.get_size() ) ) > > > > > > class Zip( Binary ): > > """Class describing a zip archive of binary sequence files""" > > file_ext = "zip" > > > > def set_peek( self, dataset, is_multi_byte=False ): > > if not dataset.dataset.purged: > > zip_file = zipfile.ZipFile( dataset.file_name, "r" ) > > num_files = len( zip_file.namelist() ) > > dataset.peek = "Archive of %s binary sequence files" % ( > > str( num_files ) ) > > dataset.blurb = data.nice_size( dataset.get_size() ) > > else: > > dataset.peek = 'file does not exist' > > dataset.blurb = 'file purged from disk' > > def display_peek( self, dataset ): > > try: > > return dataset.peek > > except: > > return "Binary sequence file archive (%s)" % ( > > data.nice_size( dataset.get_size() ) ) > > def get_mime( self ): > > """Returns the mime type of the datatype""" > > return 'application/zip' > > > > > > When I'm going to send the file in zip format (.esd files compressed),it > > shows me the following error: > > *"An error occurred running this job: Invalid 'File Format' for archive > > consisting of binary files - use 'Binseq.zip'"* > > Hi Eric, > > Have you updated to a recent revision of Galaxy? Binseq.zip and the > code that prevents uploading zip files has been removed. Currently, > though, a zip file can only contain one member. > > --nate > > > > > I tried somethings, but I don't have success. > > > > Thank you, > > -- > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > galaxy-dev mailing list > > galaxy-dev(a)lists.bx.psu.edu > > http://lists.bx.psu.edu/listinfo/galaxy-dev > -- Adhemar Zerlotini Neto, PhD FIOCRUZ - Fundação Oswaldo Cruz CPqRR - Centro de Pesquisas René Rachou http://www.cpqrr.fiocruz.br http://www.cebio.org +55 31 3337 3863 +55 31 3337 3649

2 1

DRMMA and SGE problem
by George Chalkidis 29 Jul '10

29 Jul '10

Hi Galaxy-Dev-Team! First of all, I would like to thank you for providing the community with such a great framework! I've downloaded and set up locally the latest packed release. When I want to use Galaxy with the Sun Grid Engine, I run into problems. It looks like Galaxy isn't able to fully control the Grid Engine and I think it might be a version issue with DRMMA and our current SGE. Jobs get submitted, appear in the queue and are executed, but their status is not returned to Galaxy and therefore the job's status on the website remains always "Job is waiting to run". Here's a more detailed error-message: ------------------------------------------------------ [environment] CentOS release 5.4 (Final) SGE 6.2u5 Python 2.5.2 Jun 30th source tar.gz file http://bitbucket.org/galaxy/galaxy-dist/get/tip.tar.gz --------------------------------------------------------- [config] $ cat universe_wsgi.ini |grep sge # currently available are 'pbs' and 'sge'. start_job_runners = sge #default_cluster_job_runner = sge://default/mjobs.q/ cant lunch service default_cluster_job_runner = sge:///mjobs.q/ $ grep -5 SGE_ROOT eggs.ini [general] repository = http://eggs.g2.bx.psu.edu/new ; these eggs must be scrambled for your local environment no_auto = pbs_python DRMAA_python SGE_ROOT = /home/geadmin/N1GE python scripts/scramble.py DRMAA_python [eggs:platform] bx_python = 0.5.0 Cheetah = 2.2.2 DRMAA_python = 0.2 --------------------------------------------------------- [Error Messages] galaxy.jobs DEBUG 2010-07-16 17:30:24,794 job 10 dispatched galaxy.jobs.runners.sge DEBUG 2010-07-16 17:30:27,480 (10) submitting file /share1/home/icgc/galaxy-dist/database/pbs/galaxy_10.sh galaxy.jobs.runners.sge DEBUG 2010-07-16 17:30:27,480 (10) command is: perl /share1/home/icgc/galaxy-dist/tools/myTools/toolExample.pl /share1/home/icgc/galaxy-dist/database/files/000/dataset_3.dat /share1/home/icgc/galaxy-dist/database/files/000/dataset_10.dat galaxy.jobs.runners.sge DEBUG 2010-07-16 17:30:27,522 (10) queued in mjobs.q queue as 9440764 202.175.149.92 - - [16/Jul/2010:17:30:26 +0900] "POST /root/history_item_updates HTTP/1.1" 200 - "http://gw01.hgc.jp:8090/history" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6" galaxy.jobs.runners.sge ERROR 2010-07-16 17:30:29,524 (10/9440764) Unable to check job status Traceback (most recent call last): File "/share1/home/icgc/galaxy-dist/lib/galaxy/jobs/runners/sge.py", line 277, in check_watched_items state = self.ds.getJobProgramStatus( job_id ) File "/usr/local/lib/python2.5/site-packages/DRMAA.py", line 395, in getJobProgramStatus return ps TypeError: writelines() argument must be a sequence of strings galaxy.jobs.runners.sge WARNING 2010-07-16 17:30:29,525 (10/9440764) job will now be errored galaxy.jobs.runners.sge ERROR 2010-07-16 17:30:29,526 Uncaught exception failing job Traceback (most recent call last): File "/share1/home/icgc/galaxy-dist/lib/galaxy/jobs/runners/sge.py", line 138, in run_next self.fail_job( obj ) File "/share1/home/icgc/galaxy-dist/lib/galaxy/jobs/runners/sge.py", line 338, in fail_job self.stop_job( self.sa_session.query( self.app.model.Job ).get( sge_job_state.job_wrapper.job_id ) ) AttributeError: 'SGEJobRunner' object has no attribute 'sa_session' [when I use sample code] #!/usr/bin/env python import DRMAA import time import os def main(): """Submit a job, and check its progress. Note, need file called sleeper.sh in home directory. An example: echo 'Hello World $1' sleep 30s """ s=DRMAA.Session() s.init() print 'Creating job template' jt = s.createJobTemplate() jt.remoteCommand = os.getcwd() + '/sleeper.sh' jt.args = ['42','Simon says:'] jt.joinFiles=True jt.outputPath=":"+DRMAA.JobTemplate.HOME_DIRECTORY+'/tmp/JOB_OUT' jobid = s.runJob(jt) print 'Your job has been submitted with id ' + jobid # Who needs a case statement when you have dictionaries? decodestatus = { DRMAA.Session.UNDETERMINED: 'process status cannot be determined', DRMAA.Session.QUEUED_ACTIVE: 'job is queued and active', DRMAA.Session.SYSTEM_ON_HOLD: 'job is queued and in system hold', DRMAA.Session.USER_ON_HOLD: 'job is queued and in user hold', DRMAA.Session.USER_SYSTEM_ON_HOLD: 'job is queued and in user and system hold', DRMAA.Session.RUNNING: 'job is running', DRMAA.Session.SYSTEM_SUSPENDED: 'job is system suspended', DRMAA.Session.USER_SUSPENDED: 'job is user suspended', DRMAA.Session.DONE: 'job finished normally', DRMAA.Session.FAILED: 'job finished, but failed', } for ix in range(10): print ix print 'Checking ' + str(ix) + ' of 10 times' status = s.getJobProgramStatus(jobid) print decodestatus.get(status) time.sleep(5) print 'Cleaning up' s.deleteJobTemplate(jt) s.exit() if __name__=='__main__': main() $ ./test Creating job template Your job has been submitted with id 9440771 0 Checking 0 of 10 times Traceback (most recent call last): File "./test", line 52, in <module> main() File "./test", line 43, in main status = s.getJobProgramStatus(jobid) File "/usr/local/lib/python2.5/site-packages/DRMAA.py", line 393, in getJobProgramStatus eno, ps, estr = cDRMAA.drmaa_job_ps(jobName) ValueError: need more than 2 values to unpack [debug] print cDRMAA.drmaa_job_ps(jobName) eno, ps, estr = cDRMAA.drmaa_job_ps(jobName) => ([0, 16], '') Thank you for taking time! Hopefully, you can give me some hints on how to resolve this issue. Cheers, George Chalkidis

2 2

Textbox Size
by Dennis Gascoigne 29 Jul '10

29 Jul '10

I have tools that require a wide textbox area. It seems however that there is a limit coded into Galaxy soewhere of about 60 characters. Can you advise where it is so I can bump it up please? All i am trying to do is have a textbox with area 20x170. Cheers Dennis

2 1

Setup failed due to simplejson version
by simon andrews (BI) 29 Jul '10

29 Jul '10

I've just tried an initial installation of Galaxy (galaxy_dist) on a Fedora 13 machine. When I try to run the setup.sh script I get the error below: Creating tool-data/shared/jars Traceback (most recent call last): File "./scripts/fetch_eggs.py", line 31, in <module> c.resolve() # Only fetch eggs required by the config File "/data/private/galaxy/galaxy_dist/lib/galaxy/eggs/__init__.py", line 284, in resolve egg.resolve() File "/data/private/galaxy/galaxy_dist/lib/galaxy/eggs/__init__.py", line 172, in resolve r = pkg_resources.working_set.resolve( ( self.distribution.as_requirement(), ), env, self.fetch ) File "/data/private/galaxy/galaxy_dist/lib/pkg_resources.py", line 565, in res olve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: simplejson==1.5 Fedora 13 comes with simplejson v2.1.1. Is there really a strict requirement on v1.5 or should the test be >=1.5? Is there a way to parallel install different versions of this package if 1.5 is strictly required? Thanks Simon.

2 3

LSF/DRMAA Support
by Nate Coraor 29 Jul '10

29 Jul '10

Hello, Support for the LSF distributed resource manager (DRM) has recently been committed to our development repository in changeset 4071:fd9514c5349f. This support has actually been implemented through the use of the python-drmaa module and none of the code is DRM-specific. This means that any DRM which implements a DRMAA library should be able to be supported now (including PBS Pro). For more information on how to configure a DRMAA library to work with your DRM, please see: http://www.drmaa.org/ The code is heavily based on the former SGE job runner, which will soon be deprecated in favor of the new DRMAA job runner. Thanks to John Brunelle for adapting and testing the original implementation. For information on how to set up Galaxy to use the DRMAA job runner, please see: http://bitbucket.org/galaxy/galaxy-central/wiki/Config/Cluster This changeset is not yet available in our stable distribution repository, although it should be soon. In the meantime, you can grab it from our development repository if you'd like: http://bitbucket.org/galaxy/galaxy-central/ This code has had minimal testing and should not be used for a production server. But please do test it out and report any bugs (either here or in our issue tracker in Bitbucket). Thanks! --nate

1 0

workflow empty input dataset
by Jorge.DUARTE＠biogemma.com 29 Jul '10

29 Jul '10

Hi, I managed to install galaxy from the source available on the bitbucket website and i was able use it and to create a simple workflow, but then even if i have loaded data from the menu Get data > Upload file when i run the workflow the drop down list for input data set stays empty !? Can you help ? Thanks Jorge. --- Jorge Duarte Bioinformatics Research Engineer BIOGEMMA - Upstream Genomics Group Z.I. Du Brézet 8, Rue des Frères Lumière 63028 CLERMONT FERRAND Cedex 2 FRANCE Tel : +33 (0)4 73 39 60 73 Fax : +33 (0)4 73 42 79 81 E-mail : jorge.duarte(a)biogemma.com ***************************************************************** Pour toute demande de support merci d'inclure BIOGEMMA_BioInfo_Service ou bioinfo(a)biogemma.com dans les destinataires lors du premier contact ***************************************************************** BIOGEMMA S.A.S. au capital social de 48.335.652,00 ?. 1, Rue Edouard Colonne - 75001 PARIS. RCS PARIS 412 514 366 This message and any attachments are confidential and intended solely for the use of the addressee(s) named above. The information contained in this email may also be legally privileged. If you have received this email in error, please notify us immediately by reply email or by fax and then delete it. Any use, distribution or reproduction of this message is strictly prohibited. The integrity or authenticity of this message cannot be guaranteed. We therefore shall not be liable for the message if altered, changed or falsified. Thank you. Cet email et ses pièces jointes sont strictement confidentiels et destinés uniquement à l'usage du (des) destinataire(s) sus-indiqué(s). Les informations contenues dans cet email sont légalement protégées. Si vous avez reçu cet email par erreur, merci de nous le retourner immédiatement par courrier électronique ou télécopie avant de le supprimer. Toute utilisation ou reproduction de cet email est strictement interdite. La véracité et l'authenticité de cet email et de son contenu ne peuvent être garanties et nous ne pouvons être tenus responsables de leur altération, modification ou falsification. Merci.

2 3

Re: [galaxy-dev] Eucalyptus genome in Galaxy
by Reddy.Thumma＠csiro.au 29 Jul '10

29 Jul '10

Hi, Could you please include Eucalyptus genome (http://eucalyptusdb.bi.up.ac.za/) in the public version of Galaxy. It is being sequenced at JGI and recently 8x assembly was completed. Thank you, Regards, Bala Thumma

2 1

does anyone have Galaxy running with PBS Pro (not torque)?
by Michael Pheasant 28 Jul '10

28 Jul '10

If so, i'm interested to hear how that went. Cheers Mike. -- Michael Pheasant Software Engineer Queensland Facility for Advanced Bioinformatics Level 5, QBP University of Queensland, QLD 4072 T: +61 (0)7 3346 2070 F: +61 (0)7 3346 2101 www.qfab.org

4 6

check_galaxy.py with external user authentification
by Michael Siebauer 28 Jul '10

28 Jul '10

Hi, is there a way to run the check_galaxy script when external user authentification through apache is enabled? I get the following error: |root@bioapp07:/home/galaxy/galaxy_dist/scripts# python check_galaxy.py 127.0.0.1:8080 ==> at http://127.0.0.1:8080/user Traceback (most recent call last): File "check_galaxy.py", line 358, in <module> if b.check_if_logged_in(): File "check_galaxy.py", line 278, in check_if_logged_in self.get("/user") File "check_galaxy.py", line 128, in get tc.code(200) File "/usr/lib/pymodules/python2.6/twill/commands.py", line 133, in code should_be)) twill.errors.TwillAssertionError: code is 403 != 200 | Bytheway, there is no hint in the documentation that a package "python-twill" is required to run the script ;-) Thx, Michael

2 1

Help about zip upload (esd files)
by Eric Aguiar 28 Jul '10

28 Jul '10

Dear, I followed all steps in the galaxy tutorial, but I didn't have success. I'm trying to create a datatype for megabase chromatograms (.esd) very similar to the Ab1 ones. Here is my configurations. *1 - Creating datatypes in datatypes_conf.xml* <datatype extension="zip" type="galaxy.datatypes.binary:Esd" mimetype="application/zip" display_in_upload="true"/> <datatype extension="esd" type="galaxy.datatypes.binary:Esd" mimetype="application/octet-stream" display_in_upload="true"/> *2 - Defining types in lib/galaxy/datatypes/binary.py* class Esd( Binary ): """Class describing an ab1 binary sequence file""" file_ext = "esd" def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: dataset.peek = "Binary chromatograms sequence file" dataset.blurb = data.nice_size( dataset.get_size() ) else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "Binary esd sequence file (%s)" % ( data.nice_size( dataset.get_size() ) ) class Zip( Binary ): """Class describing a zip archive of binary sequence files""" file_ext = "zip" def set_peek( self, dataset, is_multi_byte=False ): if not dataset.dataset.purged: zip_file = zipfile.ZipFile( dataset.file_name, "r" ) num_files = len( zip_file.namelist() ) dataset.peek = "Archive of %s binary sequence files" % ( str( num_files ) ) dataset.blurb = data.nice_size( dataset.get_size() ) else: dataset.peek = 'file does not exist' dataset.blurb = 'file purged from disk' def display_peek( self, dataset ): try: return dataset.peek except: return "Binary sequence file archive (%s)" % ( data.nice_size( dataset.get_size() ) ) def get_mime( self ): """Returns the mime type of the datatype""" return 'application/zip' When I'm going to send the file in zip format (.esd files compressed),it shows me the following error: *"An error occurred running this job: Invalid 'File Format' for archive consisting of binary files - use 'Binseq.zip'"* I tried somethings, but I don't have success. Thank you, --

2 1