February 2012 - galaxy-dev - lists.galaxyproject.org

FastX tools
by Borrone, James 22 Feb '12

22 Feb '12

hello, I am trying to use the FastX tools on the Main server to trim and manipulate FASTQ files extracted directly from an sff file (again using the main galaxy web server). In using Clip and the Reverse compliment tools, I get the following error: An error occurred running this job: fastx_reverse_complement: (or fastx_clip) found invalid nucleotide sequence (gactGCGACTCACGTACAGCAATGCACATACTATATTATATC) on line 2 gzip: stdout: Broken pipe I have downloaded the fastx tool kit and installed it locally on my computer and experience the same problem, so I suspect that it is a format error. My fastq file looks like this: @HA3HTSF01AH1DX gactGCGACTCACGTACAGCAATGCACATACTATATTATATCAACCtcacaacacactacacgacacacaggagagagnnn + IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIFFAAB88..--,,/152///0///14,,,,///00//////!!! @HA3HTSF01B3YMI gactTTTGGGATTTGTTAACTTAGTTTGTAGGTGGTGGTTATAATGTTTGTGTGGTGTTATGTTTGGTGTAGTGttttgggttggtttgtattatttgtggatttggagttcgtgggtaggcaaggcacactagggggattaggtngngtnntntntttnntgntgtgntctgtgtggttgtggtgattttttactttgagtgttttgcgatttgggtgtagtaattagtcggtttttacgtaatttttaggttttgtggggttttgatttgggtttaggttttaatgcttttgatggtgggtgggggttcttacacgggtttaggttagagttggttgggctatgcatatgggtgtttgggggtttcttatttgggatgttttaggtgccatgtaatatttgtttangcgagtcgttttggttcttttaattttattacttgttttaggattgtttttaggtttaatagaaaatttntgactttttttaagggtttgggttaacatggcagttttttggggtttgggaattttttagtacgttagtttaggttaggttggttaggttagttagttagttgttgttggttgttagttgtttggttggttaggtttggttttgggttttgggttttaggttttnnnnnnnnnnn + ??<//00==BA@>>A=BBCFDD:9444GGIIIIEIIIIIIIIIIG9444GGIIEGFDDEIB8443?;:;;@880----0000088001A@EEEECCCGEEEEB@?<<=900033--,/7..000511156675,,,,,,01/,,,!5!11!!0!,!,//!!40!0588!6665511111/,,,,/11,,,,,,,,--/:688<7<77>>>;333---13511///53////////111132.11,,,,,,,,,,,,/,444666772886::43.,,,..,,,,,,066....565...,,,,111....11111.,,,,,,,.2222442,,--------8777:::99900-2----,,,,,,,,-,,,233,,-774-----099564422000-,,,,-,,,,!-02222223433..,,.33,,,,22226//5888,,,,,,446622---..0,,,,0110220000,,,!.222,++++++++---001/-1--..33331001------++++++++++00++++++++++++++---00------+++++++++011111000000------0+++00000+++00--+++00----+++00000--++++0++++++-+++++++--++++!!!!!!!!!!! @HA3HTSF01BHUOB gactCATGCTTCAGATCAAGCAAGTCTTCGCTGACTACGTGCGCCATCGCCGCGAGGcgcagcacggaagtaggagcggcgtcagctcgccaacatgacgcgcgaattccacgaagagacacaaagacaccttgggaacctcttcaagtacgcggaggagaagatacgccgcgtcgcacaggaggagcacaaggcacacaggggataggnn + IIIIIIIIIIIGIIIGG@@@H>>>@IIIIIIIIIIIIIIIIIIIIIIIIEECGGE74426<AA812211/5/111/53385;::;AE>?330113==AACCC<=993333923333<DEGHF:::GIIIIIIIGIIIFCCEBB?:99<9;??=?99//053337<456333;?D@@<;==?943355@@@998895546:4444:4...!! @HA3HTSF01ASNA7 gactATTGATATATTGCAACAGCTATCCAAATACACAAATATAATACTTGTTATAGGCAAATGCGACATTTTTGAGGCGAGTGAGTTGTTGCAATATATCAAtagtcgtgggaggcaaggcacacaggggatagg + IIIIIIIIIIIIIIIIIIIIIIIIIC<<111?1G??111D5IIIIIIIIIIIIIIIIIBBBIIIDIC?00000BBDCFCFFIIIIIIHHHIIIIIHHHIIIIIIIIIHH555GHIIIIIIIIIIIIFHHHIIIII Can someone tell me what the format should be? I have tried converting the fastq file into a different format, to no avail. And I have extracted the FASTA and FASTA.qual files directly from the sff file, recombined them together, and still had the program fail to clip or reverse compliment the sequences. So, i am at a loss as to what the format should look like. All of the extractions were done directly on the sff file uploaded to galaxy. Thanks James Borrone

2 1

Error 500 when trying to execute a workflow with the API
by thondeboer＠me.com 22 Feb '12

22 Feb '12

Hi, I tried to run a workflow with the API, but get an Error 500 when I try to run the WF...The paster.log shows the following error... $ workflow_execute.py 92cc01ed93dc0f0fc91e3ded35497c0a http://srp106:8080/api/workflows ebfb8f50c6abde6d 'TEST the API' '1=ldda=7c5ebce002fc9d5c' Paster.log galaxy.web.framework ERROR 2012-02-21 14:36:33,067 Uncaught exception in exposed API method: Traceback (most recent call last): File "/home/tdeboer/code/galaxy-central/lib/galaxy/web/framework/__init__.py", line 145, in decorator return simplejson.dumps( func( self, trans, *args, **kwargs ), indent=4, sort_keys=True ) File "/home/tdeboer/code/galaxy-central/lib/galaxy/web/api/workflows.py", line 123, in create hda = ldda.to_history_dataset_association(history, add_to_history=add_to_history) AttributeError: 'NoneType' object has no attribute 'to_history_dataset_association' 172.16.108.6 - - [21/Feb/2012:14:36:32 -0700] "POST /api/workflows?key=92cc01ed93dc0f0fc91e3ded35497c0a HTTP/1.1" 500 - "-" "Python-urllib/2.6" Any ideas? Also...I had a hard time finding out what I should use for the dataset source parameter "src" in "step=src=dataset_id" and just tried ldda, but would hope there is a little info on what this hda and ldda is? Thanks, Thon

5 8

Re: [galaxy-dev] Sample Tracking System problem in Galaxy.
by Greg Von Kuster 22 Feb '12

22 Feb '12

Hello Luobin, Galaxy Sample Tracking is not supported by the core Galaxy development team, so you're on your own on this one. I'm told the problem is that the format of the library content ID has been changed in the API, so the script needs to be updated to reflect that. Sorry for the inconvenience. Greg Von Kuster On Feb 21, 2012, at 4:19 PM, Luobin Yang wrote: > Hi, Greg, > > I am a bioinformatics research scientist at Idaho State University and we're planning to use the Sample Tracking System for our ION Torrent PGM sequencer. However, I encountered a couple of problems while setting up the Sample Tracking System on our locally installed Galaxy instance. I sent an email to the developer list last week but it seems nobody's interested in it yet. So I am writing to you directly and hope to get some pointers on how to solve the problem. > > So basically I followed the instructions to set up the Sample Tracking System on our system, which is a computer cluster running Ubuntu 10.04 LTS. The problem I have here is transferring the data from the sequencer to the data libraries. So after I clicked the "Transfer" button, the status of data transfer stays on "In queue" forever and the data_transfer.log shows "PUT method not allowed" error. However the sequence files were copied from the sequencer to the folder pointed by library_import_dir. So instead of using Apache as the proxy server, I let Galaxy to provide web services directly. With this change the status of the data transfer changed to "Complete" after the files are copied to the folder pointed by library_import_dir. However, when I browse the data libraries, the data files are not imported to the data library I specified when I submitted the request. And the data_transfer.log has the following error message: > > "400 Bad Request > The server could not comply with the request since > it is either malformed or otherwise incorrect. > Malformed LibraryFolder id ( 994debf9f6ab02b9912262b0bd04c784 ) specified, unable to decode" > > So I have several questions: > 1. Do we need to configure Apache differently so we can still use Apache as the proxy server when using the Sample Tracking System? > 2. Does the version of RabbitMQ matter for this problem? Because I used the default RabbitMQ version for Utunbu 10.04, which is version 1.7.2 and the latest version of RabbitMQ is 2.7.x. I do not know what versions have you guys tested. > 3. Does the folder pointed by library_import_dir related to this problem? On our system library_import_dir points to a folder that's outside the Galaxy installation folder. > 4. What python code in Galaxy is related to this issue? > > Thanks, > Luobin

1 0

strange error having to do with stderr redirection?
by Nikhil Joshi 22 Feb '12

22 Feb '12

Hi all, I have been writing an R script to run as a tool in galaxy and it seems to work just fine. I needed to add some fine-grained error detection in the script and I wanted it to produce a "red X" if the error occurred. In order to do this I wrote to the stderr handle. Before I had put in the error code, I had redirected stderr to /dev/null in the xml, but I took that out so that galaxy could see the errors I wrote to the stderr handle. For some reason if I do not redirect stderr to /dev/null, I get this error: Traceback (most recent call last): File "/opt/Bio/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 125, in run_job job_wrapper.finish( stdout, stderr ) File "/opt/Bio/galaxy-dist/lib/galaxy/jobs/__init__.py", line 643, in finish self.sa_session.flush() File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/scoping.py", line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py", line 1356, in flush self._flush(objects) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/session.py", line 1434, in _flush flush_context.execute() File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 261, in execute UOWExecutor().execute(self, tasks) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 753, in execute self.execute_save_steps(trans, task) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 768, in execute_save_steps self.save_objects(trans, task) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 759, in save_objects task.mapper._save_obj(task.polymorphic_tosave_objects, trans) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/orm/mapper.py", line 1413, in _save_obj c = connection.execute(statement.values(value_params), params) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement return self.__execute_context(context) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/opt/Bio/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'UPDATE job SET update_time=?, stdout=?, stderr=? WHERE job.id = ?' ['2012-02-22 02:50:49.979136', 'locfit 1.5-6 \t 2010-01-20 \nnull device \n 1 \nnull device \n 1 \nnull device \n 1 \nnull device \n 1 \n\n Inf -Inf NaN \n 924 447 1710 0 \nnull device \n 1 \nnull device \n 1 \nnull device \n 1 \n', 'Loading required package: Biobase\n\nWelcome to Bioconductor\n\n Vignettes contain introductory material. To view, type\n \'browseVignettes()\'. To cite Bioconductor, see\n \'citation("Biobase")\' and for packages \'citation("pkgname")\'.\n\nLoading required package: locfit\nLoading required package: akima\nLoading required package: lattice\nWarning messages:\n1: In xy.coords(x, y, xlabel, ylabel, log) :\n 1710 x values <= 0 omitted from logarithmic plot\n2: In xy.coords(x, y, xlabel, ylabel, log) :\n 2880 y values <= 0 omitted from logarithmic plot\nWarning message:\nIn xy.coords(x, y, xlabel, ylabel, log) :\n 1710 x values <= 0 omitted from logarithmic plot\nLoading required package: R.methodsS3\nR.methodsS3 v1.2.1 (2010-09-18) successfully loaded. See ?R.methodsS3 for help.\nLoading required package: R.oo\nR.oo v1.8.3 (2011-11-01) successfully loaded. See ?R.oo for help.\n\nAttaching package: \xe2\x80\x98R.oo\xe2\x80\x99\n\nThe following object(s) are masked from \xe2\x80\x98package:R.methodsS3\xe2\x80\x99:\n\n throw.default\n\nThe following object(s) are masked from \xe2\x80\x98package:methods\xe2\x80\x99:\n\n getClass, getClasses, getMethods\n\nThe following object(s) are masked from \xe2\x80\x98package:base\xe2\x80\x99:\n\n attach, detach, environment, gc, load, save\n\naroma.light v1.22.0 (2011-10-31) successfully loaded. See ?aroma.light for help.\n', 1345] It seems that R produces some output on stdout that I can't get rid of yet, but it doesn't seem to affect the outcome... it doesn't explain why this error happens... Any ideas? Is R using some sort of weird encoding that the python can't get? - Nik. -- Nikhil Joshi Bioinformatics Programmer UC Davis Genome Center Davis, CA

2 1

advanced parameters: all or nothing?
by Andrew Warren 21 Feb '12

21 Feb '12

Hello, I was recently adjusting advanced parameters when running Tophat in Galaxy and noticed that when advanced parameters are used, every field is converted and submitted as command line parameter to the tool at runtime. Without changing any of the default values I get a different tophat result than if advanced parameters are left off. I'm curious if the Galaxy team has considered a mechanism for disabling an individual parameter? Or perhaps a way of individually enabling parameters from within the Advanced Parameter block? Just trying to think of a way to use one advanced parameter without using all of them. Cheers, Andrew Warren

3 6

Extracting the metrics files from a composite file such as Picard
by Thon Deboer 21 Feb '12

21 Feb '12

Hi, Is there a way to extract the individual files from a composite file, such as the HTML files created by the picard tools? I would like to take the metrics files and use them further down in some workflow, but I only get an HTML file... While these HTML files are nice for quickly looking at the results of one or two files, it becomes a problem if you have 88 samples like I do... I hate to have to re-write all the picard tools to produce actual files, but maybe there is something I am missing about composite datatype files? The wiki only explains how to CREATE them, not how to use those files downstream.... Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeboer(a)me.com LinkedIn Profile

4 6

Re: [galaxy-dev] Functional tests
by Greg Von Kuster 21 Feb '12

21 Feb '12

Frank, I can't see anything that would cause this behavior by looking at your logs, so I cannot answer what is causing this problem in your environment. Basically it seems that nose is not finding any of the Python test scripts in the test subdirectory in your Galaxy installation directory, but I am not able to tell you why (I assume you haven't moved any directories). Sorry for not being of any help here. Greg Von Kuster On Feb 21, 2012, at 2:55 AM, Frank Sørensen wrote: > Hi Greg, > > Den 20-02-2012 15:06, Greg Von Kuster skrev: >> Hello Frank, see my inline comments. >> >> On Feb 20, 2012, at 6:56 AM, Frank Sørensen wrote: >> >>> Hi Guys, >>> >>> Sorry if this is a stupid question, but in order to find out which external reference files (.loc - list files etc.) we need for our newly installed Galaxy server, and to get an overview of which tools are in working order, I have tried for a couple of days now, to run the script "run_functional_tests.sh". >> What is the exact command line that you are using to attempt to execute the functional tests? Are you using the following? >> >> %sh run_functional_test.sh > Yes, exactly - from .../galaxy-dist: ./run_functional_tests.sh >> >> >>> I have copied tool_conf.xml over tool_conf.xml.sample, >> The functional tests use tool_conf.xml.sample, so what are the contents of that file in your environment? > It is the same as tool_conf.xml. The complete list of all the tools that come with the Galaxy installation, plus our own. >>> and I've tried to run all scripts mentioned in http://wiki.g2.bx.psu.edu/Main/Tests/Framework. >> I assume by the above you mean this link: >> >> http://wiki.g2.bx.psu.edu/Main/Tests%20Framework > Yes ofcourse... >> >> Here is another link that provide additional information for testing: >> >> http://wiki.g2.bx.psu.edu/Admin/Running%20Tests > Thanks, I'll check it out. >> >>> I have tried to stop the server and run run_functional_tests.sh and all I get is the message saying "Ran 0 tests in 0.000s", and an empty run_functional_tests.html - file. >>> I haven't been able to find any documentation to clarify this matter. Can anyone give me a pointer please? >> >> We'll need more information about your environment, but we'll get this figured out. > %uname -a > Linux moma 2.6.32-21-server #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC 2010 x86_64 GNU/Linux > > I have attached the result of %lshw and the output from run_functional_tests.sh and our global settings file. > >>> Our Galaxy server is installed using Mercurial on an Ubuntu server. I have written tool-wrappers for some of our own programs, and they all work fine. >>> >> > Kind regards > > - Frank > > -- > Frank Sørensen, B.Sc., Programmer > Molecular Diagnostic Laboratory (MDL) > Molekylær Medicinsk Afdeling (MOMA) > Århus Universitetshospital Skejby, Brendstrupgårdsvej, 8200 Århus N > Tlf. +45 7845 5363 > <lshw.log><tests.log><universe_wsgi.ini><tool_conf.xml>

1 0

Illumina HiSeq 2000 external service type XML
by Leandro Hermida 21 Feb '12

21 Feb '12

Hi everyone, Does anyone have a typical external service type XML file for an Illumina HiSeq 2000? I have created one but not sure if its correct and there is no guide for creating the external service type XMLs thanks, Leandros

1 0

Galaxy Cloudman - How to analyse > 1TB data ?
by Wetzels, Yves [JRDBE Extern] 21 Feb '12

21 Feb '12

Dear I am currently investigating if Galaxy Cloudman can help us in analyzing large NGS datasets. I was first impressed by the simple setup, the autoscaling and useability of Galaxy Cloudman but soon ran into the EBS limit of 1 TB L I thought to be clever and umounted the /mnt/galaxyData EBS volume, created a logical volume of 2 TB and remounted this volume to /mnt/galaxyData. All is green as you can see from the picture below but running a tool is not possible since Galaxy is not configured to work with logical volume I assume. It is truly a waste having this fine setup (autoscaling) but this is not useable if there is not enough storage ? Does anybody has experience with this ? Tips, tricks ... Kind Regards Yves Wetzels Contractor on behalf of Janssen Turnhoutseweg 30 B-2340-Beerse, Belgium Phone 32 (0)14/ 60 7181 ywetzel(a)its.jnj.com <mailto:ywetzel@its.jnj.com>

3 6

Re: [galaxy-dev] anyone using SFTP?
by Dave Clements 21 Feb '12

21 Feb '12

Hi Mattias, Posting to Galaxy-Dev, so others can see the question and response. Is there anyone using SFTP in a local Galaxy installation? I read that > FTP is insecure because it sends the data and user credentials in plaint > text. > The short answer is yes. See. http://galaxy-development-list-archive.2308389.n4.nabble.com/template/NamlS… Dave C. > I found a HOWTO on installing SFTP in ProFTPD: > http://www.directadmin.com/forum/showthread.php?t=30607&page=1 > > But I want to know if anyone is using SFTP. > > Thanks! > > Mattias > > -- http://galaxyproject.org/GCC2012 <http://galaxyproject.org/wiki/GCC2012> http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://galaxyproject.org/wiki/

2 1