November 2010 - galaxy-dev - lists.galaxyproject.org

Puzzling test failure with data_column parameter
by Peter Cock 24 Apr '13

24 Apr '13

Hi all, I'm working on a FASTA filter script, as per this email: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-November/003819.html Current code here: http://bitbucket.org/peterjc/galaxy-central/src/c3ac6d7a02f7/ I've added a basic test, but it won't run (see output below). I think the interesting bit of the error output is this exception: cannot find value/label "blastp_four_human_vs_rhodopsin.tabular" in list control, coming from function set_form_control_value in the twill.utils module. This is something to do with the columns parameter, a multiselect option referencing the columns in this tabular file. Removing the validator makes no difference. However, if I change the columns parameter to a plain text parameter, the test passes: $ hg diff diff -r c3ac6d7a02f7 tools/fasta_tools/fasta_filter_by_id.xml --- a/tools/fasta_tools/fasta_filter_by_id.xml Tue Nov 23 11:37:35 2010 +0000 +++ b/tools/fasta_tools/fasta_filter_by_id.xml Tue Nov 23 14:03:31 2010 +0000 @@ -4,9 +4,7 @@ <inputs> <param name="input_fasta" type="data" format="fasta" label="FASTA file to filter on the identifiers"/> <param name="input_tabular" type="data" format="tabular" label="Tabular file containing FASTA identifiers"/> - <param name="columns" type="data_column" data_ref="input_tabular" multiple="True" numerical="False" label="Column(s) containing FASTA identifiers" help="Multi-select list - hold the appropriate key while clicking to select multiple columns"> - <validator type="no_options" message="Pick at least one column"/> - </param> + <param name="columns" type="text" label="Column(s) containing FASTA identifiers" /> </inputs> <outputs> <data name="output_pos" format="fasta" label="With matched ID" /> I've tried searching the provided wrappers for similar examples (column multi-select from a tabular file). I found filters/uniq.xml (works but uses a bed file rather than a simple tabular file) and stats/cor.xml (needs R and rpy which I don't have installed yet) which have tests, and finally plotting/bar_chart.xml which has no unit tests. Would someone familiar with the internals of the Galaxy tests and how they set tool parameters be able to try reproducing this for me? The branch has all the unit test files required, and there are no new dependencies needed. Thank you, Peter -- Here is the output (on Linux - other tests tried pass): $ ./run_functional_tests.sh -id fasta_filter_by_id ... nose.plugins.manager DEBUG 2010-11-23 13:59:32,958 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2010-11-23 13:59:32,961 DefaultPluginManager load plugin nosetestdiff = nosetestdiff.plugin:NoseTestDiff nose.plugins.manager DEBUG 2010-11-23 13:59:32,962 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML Filter sequences by ID ( fasta_filter_by_id ) > Test-1 ... galaxy.tools.actions.upload_common INFO 2010-11-23 13:59:34,905 tool upload1 created job id 1 galaxy.jobs DEBUG 2010-11-23 13:59:42,709 dispatching job 1 to local runner galaxy.jobs INFO 2010-11-23 13:59:42,899 job 1 dispatched galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:43,213 executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:44,033 execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs DEBUG 2010-11-23 13:59:44,414 job 1 ended galaxy.tools.actions.upload_common INFO 2010-11-23 13:59:49,927 tool upload1 created job id 2 galaxy.jobs DEBUG 2010-11-23 13:59:50,067 dispatching job 2 to local runner galaxy.jobs INFO 2010-11-23 13:59:50,702 job 2 dispatched galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:52,059 executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:52,874 execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs DEBUG 2010-11-23 13:59:53,420 job 2 ended base.twilltestcase DEBUG 2010-11-23 13:59:53,747 In submit_form, continuing, but caught exception: cannot find value/label "blastp_four_human_vs_rhodopsin.tabular" in list control base.twilltestcase DEBUG 2010-11-23 13:59:54,488 ## files diff on /home/pjcock/repositories/galaxy-central/test-data/four_human_proteins_filter_a.fasta and /tmp/tmpijWxYGfour_human_proteins_filter_a.fasta lines_diff=0, found diff = 61 FAIL ====================================================================== FAIL: Filter sequences by ID ( fasta_filter_by_id ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 160, in test_tool self.do_it( td ) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 101, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes ) File "/home/pjcock/repositories/galaxy-central/test/base/twilltestcase.py", line 656, in verify_dataset_correctness raise AssertionError( errmsg ) AssertionError: History item 1 different than expected, difference (using diff): --- local_file +++ history_data @@ -1,2 +1,61 @@ +>sp|Q9BS26|ERP44_HUMAN Endoplasmic reticulum resident protein 44 OS=Homo sapiens GN=ERP44 PE=1 SV=1 +MHPAVFLSLPDLRCSLLLLVTWVFTPVTTEITSLDTENIDEILNNADVALVNFYADWCRF +SQMLHPIFEEASDVIKEEFPNENQVVFARVDCDQHSDIAQRYRISKYPTLKLFRNGMMMK +REYRGQRSVKALADYIRQQKSDPIQEIRDLAEITTLDRSKRNIIGYFEQKDSDNYRVFER +VANILHDDCAFLSAFGDVSKPERYSGDNIIYKPPGHSAPDMVYLGAMTNFDVTYNWIQDK +CVPLVREITFENGEELTEEGLPFLILFHMKEDTESLEIFQNEVARQLISEKGTINFLHAD +CDKFRHPLLHIQKTPADCPVIAIDSFRHMYVFGDFKDVLIPGKLKQFVFDLHSGKLHREF +HHGPDPTDTAPGEQAQDVASSPPESSFQKLAPSEYRYTLLRDRDEL +>sp|Q9NSY1|BMP2K_HUMAN BMP-2-inducible protein kinase OS=Homo sapiens GN=BMP2K PE=1 SV=2 +MKKFSRMPKSEGGSGGGAAGGGAGGAGAGAGCGSGGSSVGVRVFAVGRHQVTLEESLAEG +GFSTVFLVRTHGGIRCALKRMYVNNMPDLNVCKREITIMKELSGHKNIVGYLDCAVNSIS +DNVWEVLILMEYCRAGQVVNQMNKKLQTGFTEPEVLQIFCDTCEAVARLHQCKTPIIHRD +LKVENILLNDGGNYVLCDFGSATNKFLNPQKDGVNVVEEEIKKYTTLSYRAPEMINLYGG +KPITTKADIWALGCLLYKLCFFTLPFGESQVAICDGNFTIPDNSRYSRNIHCLIRFMLEP +DPEHRPDIFQVSYFAFKFAKKDCPVSNINNSSIPSALPEPMTASEAAARKSQIKARITDT +IGPTETSIAPRQRPKANSATTATPSVLTIQSSATPVKVLAPGEFGNHRPKGALRPGNGPE +ILLGQGPPQQPPQQHRVLQQLQQGDWRLQQLHLQHRHPHQQQQQQQQQQQQQQQQQQQQQ +QQQQQQHHHHHHHHLLQDAYMQQYQHATQQQQMLQQQFLMHSVYQPQPSASQYPTMMPQY +QQAFFQQQMLAQHQPSQQQASPEYLTSPQEFSPALVSYTSSLPAQVGTIMDSSYSANRSV +ADKEAIANFTNQKNISNPPDMSGWNPFGEDNFSKLTEEELLDREFDLLRSNRLEERASSD +KNVDSLSAPHNHPPEDPFGSVPFISHSGSPEKKAEHSSINQENGTANPIKNGKTSPASKD +QRTGKKTSVQGQVQKGNDESESDFESDPPSPKSSEEEEQDDEEVLQGEQGDFNDDDTEPE +NLGHRPLLMDSEDEEEEEKHSSDSDYEQAKAKYSDMSSVYRDRSGSGPTQDLNTILLTSA +QLSSDVAVETPKQEFDVFGAVPFFAVRAQQPQQEKNEKNLPQHRFPAAGLEQEEFDVFTK +APFSKKVNVQECHAVGPEAHTIPGYPKSVDVFGSTPFQPFLTSTSKSESNEDLFGLVPFD +EITGSQQQKVKQRSLQKLSSRQRRTKQDMSKSNGKRHHGTPTSTKKTLKPTYRTPERARR +HKKVGRRDSQSSNEFLTISDSKENISVALTDGKDRGNVLQPEESLLDPFGAKPFHSPDLS +WHPPHQGLSDIRADHNTVLPGRPRQNSLHGSFHSADVLKMDDFGAVPFTELVVQSITPHQ +SQQSQPVELDPFGAAPFPSKQ +>sp|P06213|INSR_HUMAN Insulin receptor OS=Homo sapiens GN=INSR PE=1 SV=4 +MATGGRRGAAAAPLLVAVAALLLGAAGHLYPGEVCPGMDIRNNLTRLHELENCSVIEGHL +QILLMFKTRPEDFRDLSFPKLIMITDYLLLFRVYGLESLKDLFPNLTVIRGSRLFFNYAL +VIFEMVHLKELGLYNLMNITRGSVRIEKNNELCYLATIDWSRILDSVEDNYIVLNKDDNE +ECGDICPGTAKGKTNCPATVINGQFVERCWTHSHCQKVCPTICKSHGCTAEGLCCHSECL +GNCSQPDDPTKCVACRNFYLDGRCVETCPPPYYHFQDWRCVNFSFCQDLHHKCKNSRRQG +CHQYVIHNNKCIPECPSGYTMNSSNLLCTPCLGPCPKVCHLLEGEKTIDSVTSAQELRGC +TVINGSLIINIRGGNNLAAELEANLGLIEEISGYLKIRRSYALVSLSFFRKLRLIRGETL -------------------- >> begin captured stdout << --------------------- Uploaded file: four_human_proteins.fasta , ftype: fasta , extra: {'ftype': 'fasta', 'value': 'four_human_proteins.fasta', 'children': []} Uploaded file: blastp_four_human_vs_rhodopsin.tabular , ftype: tabular , extra: {'ftype': 'tabular', 'value': 'blastp_four_human_vs_rhodopsin.tabular', 'children': []} form 'tool_form' contains the following controls ( note the values ) control 0: <HiddenControl(tool_id=fasta_filter_by_id) (readonly)> control 1: <HiddenControl(tool_state=800255c1626362313139396662383639346664316133316565303361646336323163663764636566333538623a376232323566356637303631363736353566356632323361323033303263323032323639366537303735373435663636363137333734363132323361323032323331323232633230323236333666366337353664366537333232336132303232366537353663366332323263323032323639366537303735373435663734363136323735366336313732323233613230323233323232376471002e) (readonly)> control 2: <SelectControl(input_fasta=[*1])> control 3: <SelectControl(input_tabular=[*2])> control 4: <SelectControl(columns=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])> control 5: <SubmitControl(runtool_btn=Execute) (readonly)> page_inputs (0) {'input_fasta': ['four_human_proteins.fasta'], 'columns': ['1'], 'input_tabular': ['blastp_four_human_vs_rhodopsin.tabular']} --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- galaxy.tools.actions.upload_common: INFO: tool upload1 created job id 1 galaxy.jobs: DEBUG: dispatching job 1 to local runner galaxy.jobs: INFO: job 1 dispatched galaxy.jobs.runners.local: DEBUG: executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs.runners.local: DEBUG: execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs: DEBUG: job 1 ended galaxy.tools.actions.upload_common: INFO: tool upload1 created job id 2 galaxy.jobs: DEBUG: dispatching job 2 to local runner galaxy.jobs: INFO: job 2 dispatched galaxy.jobs.runners.local: DEBUG: executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs.runners.local: DEBUG: execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs: DEBUG: job 2 ended base.twilltestcase: DEBUG: In submit_form, continuing, but caught exception: cannot find value/label "blastp_four_human_vs_rhodopsin.tabular" in list control base.twilltestcase: DEBUG: ## files diff on /home/pjcock/repositories/galaxy-central/test-data/four_human_proteins_filter_a.fasta and /tmp/tmpijWxYGfour_human_proteins_filter_a.fasta lines_diff=0, found diff = 61 --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 1 test in 21.504s FAILED (failures=1) functional_tests.py INFO 2010-11-23 13:59:54,494 Shutting down functional_tests.py INFO 2010-11-23 13:59:54,494 Shutting down embedded web server functional_tests.py INFO 2010-11-23 13:59:54,497 Embedded web server stopped functional_tests.py INFO 2010-11-23 13:59:54,497 Shutting down app galaxy.jobs INFO 2010-11-23 13:59:54,497 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:54,498 job queue stopped galaxy.jobs.runners.local INFO 2010-11-23 13:59:54,498 sending stop signal to worker threads galaxy.jobs.runners.local INFO 2010-11-23 13:59:54,498 local job runner stopped galaxy.jobs INFO 2010-11-23 13:59:54,499 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:54,499 job stopper stopped functional_tests.py INFO 2010-11-23 13:59:54,499 Embedded Universe application stopped functional_tests.py INFO 2010-11-23 13:59:54,500 Cleaning up temporary files in /tmp/tmpu4uu_o galaxy.jobs INFO 2010-11-23 13:59:55,410 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:55,411 job queue stopped galaxy.jobs.runners.local INFO 2010-11-23 13:59:55,411 sending stop signal to worker threads galaxy.jobs.runners.local INFO 2010-11-23 13:59:55,411 local job runner stopped galaxy.jobs INFO 2010-11-23 13:59:55,411 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:55,412 job stopper stopped 'run_functional_tests.sh help' for help Note: galaxy test framework uses tool_conf.xml.sample, not tool_conf.xml

2 3

Tabular file metadata - columns names
by Peter 22 Feb '13

22 Feb '13

Hi all, I'd like to know more about Galaxy's column metadata for tabular files. In the workflow editor under "Edit Step Actions" you can pick "Assign Columns", and then give column numbers for five predefined cases: Chrom, Start, End, Strand, Name. Do these "named columns" get shown anywhere in the Galaxy UI? For example, in a column select parameter widget? Is it possible to assign these columns in a tool's wrapper XML file? >From http://bitbucket.org/galaxy/galaxy-central/wiki/ToolConfigSyntax I'm aware of the metadata_source attribute to *copy* the meta data from the input file, but that isn't always relevant. Can I somehow specify that my tool has tabular output where column 1 is "Name"? Is it possible to introduce additional column types? e.g. "evalue" or "Description". Thanks, Peter

3 7

run_functional_tests.sh -sid option
by Peter 11 Feb '13

11 Feb '13

Hi, I'm trying to use run_functional_tests.sh to run all the tests for a section (group of tools). I've read the output from: ./run_functional_tests.sh help For example, from the tools_conf.xml.sample we have <section name="ENCODE Tools" id="EncodeTools"> <tool file="encode/gencode_partition.xml" /> <tool file="encode/random_intervals.xml" /> </section> And looking at these tools' XML files, <tool id="gencode_partition1" name="Gencode Partition"> and: <tool id="random_intervals1" name="Random Intervals"> I'd like to run the functional tests for the ENCODE tools, Using the switch for an individual tool id (-id) works, ./run_functional_tests.sh -id gencode_partition1 ... Ran 1 test in 22.302s ... and so does this (well, it say the tool doesn't have any tests which is true in this example): ./run_functional_tests.sh -id random_intervals1 ... Ran 1 test in 0.027s ... However, I also tried using the section id switch (-sid), ./run_functional_tests.sh -sid EncodeTools ... Ran 0 tests in 0.000s ... I also tried using the section name, /run_functional_tests.sh -sid "ENCODE Tools" Is this (-sid not working) a known issue, or am I using it wrong? Thanks, Peter

4 5

Sharing tool definitions between XML files
by Peter 02 Jun '11

02 Jun '11

Hi all, I'm wondering if there is any established way to share parameter definitions between tool wrapper XML files? For example, I am currently working on NCBI BLAST+ wrappers, and these tools have a lot in common. For example, the output format option will be a select parameter, and it will be the same for blastn, blastp, tblastn, etc. I can just cut and paste the definition but this seems inelegant and will be a long term maintenance burden if it ever needs updating (lots of places would need to be updated the same way). I'd like to have a shared XML snippet defining this parameter which I could then reference/include from each tool wrapper than needs it. I'm thinking of something like XML's <!ENTITY ...> might work. Is this possible? Peter

2 2

Output file naming / dataset's label
by Tanguy Le Carrour 14 Apr '11

14 Apr '11

Dear Galaxy Team, I started using Galaxy few months ago. I'm now setting up my first "real" workflows. I would like to keep the name of my input file through the various steps as dataset's label, maybe also modify it a bit, and finally give a "proper" name to the result file downloaded. A simple example: 1) date-sample_name.gff3 --[step1]-> date-sample_name-step1.tsv 2) date-sample_name-step1.tsv --[step2]-> date-sample_name-step1-step2.txt 3 download date-sample_name-step1-step2.txt I've seen that I can use the 'label' attribute in the tool config "data" tag to do so. But I don't know how to access the ${input.label} (?) and how to modify it. I doesn't stop me using my workflows, but it would definitely help my users not to get lost when they use them. Best regards, Tanguy Le Carrour -- Tanguy LE CARROUR Functional Genomics Center Zurich ETH Zurich / University of Zurich Winterthurerstrasse 190 8057 Zurich Switzerland

4 8

swap issue
by Isabelle Phan 22 Mar '11

22 Mar '11

Hello, I am running galaxy, using a postgres backend. The galaxy server systematically gets killed whenever I try uploading a bam file. The samtools process is still running. This is the end of the log: 2010-08-26 18:08:11,371 INFO sqlalchemy.engine.threadlocal.TLEngine.0x...4610 {'param_1': 8} sqlalchemy.engine.threadlocal.TLEngine.0x...4610 INFO 2010-08-26 18:08:11,371 {'param_1': 8} 2010-08-26 18:08:11,378 INFO sqlalchemy.pool.QueuePool.0x...4ed0 Connection <connection object at 0x567bf30; dsn: 'dbname=galaxy host=/var closed: 0> being returned to pool sqlalchemy.pool.QueuePool.0x...4ed0 INFO 2010-08-26 18:08:11,378 Connection <connection object at 0x567bf30; dsn: 'dbname=galaxy host=/var closed: 0> being returned to pool 10.1.20.44 - - [26/Aug/2010:18:08:06 -0700] "POST /root/history_item_updates HTTP/1.1" 200 - "http://galaxy:8080/history" "Mozilla/5.0 (X1 (x86_64); en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" Killed I have the following options set in universe_wsgi.ini: database_engine_option_server_side_cursors = True database_engine_option_strategy = threadlocal It seems to be running out of swap. I monitored memory usage with 'top' and galaxy died when available swap hit 0k free. I am not superuser on the machine where galaxy is running and I have no other server available. Is there something I could do to prevent galaxy from dying? Thanks, Isabelle -- Isabelle Phan, DPhil Seattle Biomedical Research Institute +1(206)256 7113

3 4

Import files from local file system
by Steve Taylor 22 Feb '11

22 Feb '11

Hi, I noticed the recent update to Galaxy that allowed users to put data on the galaxy FTP server and then import it into their history. Would it be possible to use a similar mechanism to import data from the galaxy host file system but not use an FTP daemon? In our case we we would like to mount a volume to the galaxy file server and allow people to cp or mv data into it and then be able to import the data into their history. For example, In the context of the example at http://bitbucket.org/galaxy/galaxy-central/wiki/Config/UploadViaFTP could you set up a directory called '/home/nate/galaxy_dist/database/localfs/' instead of '/home/nate/galaxy_dist/database/ftp/' which contained email named directories and the files in these in these would be available to import via the upload UI? Or is it all tied into FTP? Thanks for any advice, Steve

5 10

Problems with run_functional_tests.sh and history?
by Peter 21 Feb '11

21 Feb '11

Hi all, I'm having trouble with the Galaxy functional tests. On our Linux server things seem to work, but I'm having trouble on my Mac. This affects multiple tests. For example, using a test for my own tool wrapper, on Linux my test passes. On my Mac, the tool is missing the required binary so I expect the test to fail with error level 127 and a message to stderr. However, I get the following cryptic error: $ ./run_functional_tests.sh -id tmhmm2 ... functional_tests.py INFO 2010-10-28 11:57:01,129 Embedded web server started functional_tests.py INFO 2010-10-28 11:57:01,185 Functional tests will be run against localhost:8980 nose.plugins.manager DEBUG 2010-10-28 11:57:01,210 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2010-10-28 11:57:01,215 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML TMHMM 2.0 ( tmhmm2 ) > Test-1 ... Error - <type 'exceptions.TypeError'>: 'NoneType' object is not callable URL: http://localhost:8980/history File '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.7.2-py2.5.egg/paste/exceptions/errormiddleware.py', line 144 in __call__ app_iter = self.application(environ, sr_checker) File '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.7.2-py2.5.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Paste-1.7.2-py2.5.egg/paste/httpexceptions.py', line 636 in __call__ return self.application(environ, start_response) File '/Users/xxx/repositories/galaxy-central/lib/galaxy/web/framework/base.py', line 154 in __call__ return body( environ, start_response ) File '/Users/xxx/repositories/galaxy-central/lib/galaxy/web/framework/__init__.py', line 668 in render template.render_context( context ) File '/Users/xxx/repositories/galaxy-central/eggs/Mako-0.2.5-py2.5.egg/mako/template.py', line 146 in render_context runtime._render_context(self, self.callable_, context, *args, **kwargs) File '/Users/xxx/repositories/galaxy-central/eggs/Mako-0.2.5-py2.5.egg/mako/runtime.py', line 381 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/Users/xxx/repositories/galaxy-central/eggs/Mako-0.2.5-py2.5.egg/mako/runtime.py', line 414 in _exec_template callable_(context, *args, **kwargs) File '/Users/xxx/repositories/galaxy-central/database/compiled_templates/root/history.mako.py', line 55 in render_body __M_writer(u'\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n\n<html>\n\n<head>\n<title>') File '/Users/xxx/repositories/galaxy-central/lib/galaxy/web/framework/__init__.py', line 665 in write response_write( d.encode( 'utf-8' ) ) TypeError: 'NoneType' object is not callable CGI Variables ------------- CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html; */*' HTTP_ACCEPT_ENCODING: 'identity' HTTP_CONNECTION: 'close' HTTP_COOKIE: '$Version=1; galaxysession=eb142648ac45b770a711a07bbd8189c4a80d9868f9ceff96f0635391728a47dfe5161bca180a2aae; $Path="/"' HTTP_HOST: 'localhost:8980' PATH_INFO: '/history' REMOTE_ADDR: '127.0.0.1' REQUEST_METHOD: 'GET' SERVER_NAME: '127.0.0.1' SERVER_PORT: '8980' SERVER_PROTOCOL: 'HTTP/1.1' Configuration ------------- static_enabled: False use_translogger: False WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0xbfd2e10> paste.cookies: (<SimpleCookie: galaxysession='eb142648ac45b770a711a07bbd8189c4a80d9868f9ceff96f0635391728a47dfe5161bca180a2aae'>, '$Version=1; galaxysession=eb142648ac45b770a711a07bbd8189c4a80d9868f9ceff96f0635391728a47dfe5161bca180a2aae; $Path="/"') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0xbfd3110> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0xccf1f10> paste.recursive.forward: <paste.recursive.Forwarder from /> paste.recursive.include: <paste.recursive.Includer from /> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> paste.recursive.script_name: '' paste.throw_errors: True webob._parsed_query_vars: (MultiDict([]), '') wsgi process: 'Multithreaded' ------------------------------------------------------------ ERROR ====================================================================== ERROR: TMHMM 2.0 ( tmhmm2 ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 160, in test_tool self.do_it( td ) File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 24, in do_it self.new_history() File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 291, in new_history self.check_history_for_string('Your history is empty') File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 215, in check_history_for_string self.visit_page( "history" ) File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 1040, in visit_page tc.code( 200 ) File "/Users/xxx/repositories/galaxy-central/eggs/twill-0.9-py2.5.egg/twill/commands.py", line 133, in code should_be)) TwillAssertionError: code is 500 != 200 ---------------------------------------------------------------------- Ran 1 test in 0.824s FAILED (errors=1) functional_tests.py INFO 2010-10-28 11:57:02,079 Shutting down functional_tests.py INFO 2010-10-28 11:57:02,079 Shutting down embedded web server ... ############################################################################### Here is my suggested fix: diff -r 82743fcbf23c lib/galaxy/web/framework/__init__.py --- a/lib/galaxy/web/framework/__init__.py Thu Oct 28 11:41:18 2010 +0100 +++ b/lib/galaxy/web/framework/__init__.py Thu Oct 28 11:56:16 2010 +0100 @@ -660,6 +660,9 @@ ## return template.render( **data ) def render( environ, start_response ): response_write = start_response( self.response.wsgi_status(), self.response.wsgi_headeritems() ) + if response_write is None: + log.error( "Error: this request returned None from start_response: %s" % self.request.browser_url ) + return [] class StreamBuffer( object ): def write( self, d ): response_write( d.encode( 'utf-8' ) ) ############################################################################### With the patch, I get: $ ./run_functional_tests.sh -id tmhmm2 ... functional_tests.py INFO 2010-10-28 11:52:37,742 Embedded web server started functional_tests.py INFO 2010-10-28 11:52:37,800 Functional tests will be run against localhost:9383 nose.plugins.manager DEBUG 2010-10-28 11:52:37,825 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2010-10-28 11:52:37,829 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML TMHMM 2.0 ( tmhmm2 ) > Test-1 ... galaxy.web.framework ERROR 2010-10-28 11:52:38,685 Error: this request returned None from start_response: http://localhost:9383/history ERROR ====================================================================== ERROR: TMHMM 2.0 ( tmhmm2 ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 160, in test_tool self.do_it( td ) File "/Users/xxx/repositories/galaxy-central/test/functional/test_toolbox.py", line 24, in do_it self.new_history() File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 291, in new_history self.check_history_for_string('Your history is empty') File "/Users/xxx/repositories/galaxy-central/test/base/twilltestcase.py", line 217, in check_history_for_string tc.find(subpatt) File "/Users/xxx/repositories/galaxy-central/eggs/twill-0.9-py2.5.egg/twill/commands.py", line 239, in find raise TwillAssertionError("no match to '%s'" % (what,)) TwillAssertionError: no match to 'Your' -------------------- >> begin captured logging << -------------------- galaxy.web.framework: ERROR: Error: this request returned None from start_response: http://localhost:9383/history --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 1 test in 0.821s FAILED (errors=1) functional_tests.py INFO 2010-10-28 11:52:38,691 Shutting down functional_tests.py INFO 2010-10-28 11:52:38,691 Shutting down embedded web server ... ############################################################################### I'm still puzzled about what is going wrong - this affects other Galaxy tests as well - not just mine. Is there something obvious wrong with the setup on this machine? Actually using Galaxy running on this Mac seems to work. Thanks, Peter

2 6

Re: [galaxy-dev] NCBI BLAST+ wrappers in Galaxy?
by Peter 02 Feb '11

02 Feb '11

On Wed, Oct 13, 2010 at 10:22 AM, Peter <peter(a)maubp.freeserve.co.uk> wrote: > On Wed, Oct 13, 2010 at 12:58 AM, Kanwei Li <kanwei(a)gmail.com> wrote: >> >> Hi Peter, >> >> The BLAST+ branch has been transplanted into galaxy-central. Thank you! >> >> Kanwei > > Wow. Thanks :) There are another few change sets on this follow up branch, http://bitbucket.org/peterjc/galaxy-central/src/blastplus2 These are: Include BLAST-XML to tabular in tool_conf.xml.sample http://bitbucket.org/peterjc/galaxy-central/changeset/629448f1a17b Workaround Issue 397 using 'if' in Cheetah template to avoid hidden parameters http://bitbucket.org/peterjc/galaxy-central/changeset/02ce0b68936a Ignore Unix editor backup files ending with tilde (not BLAST+ specific): http://bitbucket.org/peterjc/galaxy-central/changeset/708816ffd451 Ignore # lines in BLAST tabular output in my filter script http://bitbucket.org/peterjc/galaxy-central/changeset/b2dcd6b8d434 And a capitalisation fix in my XML caption: http://bitbucket.org/peterjc/galaxy-central/changeset/6efadbea0cfc > > I mentioned it needs tests: > I've worked out how to add unit tests to the XML wrapper (that is well documented) and run them via run_functional_tests.sh - so adding some BLAST+ tests should be possible now. Peter

5 29

FASTA filtering by ID
by Peter 06 Jan '11

06 Jan '11

Hi all, Something I want to do in several of my workflows is to filter a FASTA file (or potentially other format sequence files) using a list of desired identifiers (e.g. a column from a tabular file). Right now I can achieve this with three steps in Galaxy. Suppose I have: Dataset #1, FASTA file Dataset #2, Tabular file with identifiers of interest (e.g. BLAST hits, or filtered output from a sequence analysis tool) Then: Create tabular Dataset #3 using FASTA-to-tabular on Dataset #1, subject to the enhancement proposed here: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-November/003717.html Create tabular Dataset #4 using join on Datasets #2 and #3 using the matched identifier columns. This does the filtering. Create FASTA Dataset #5 using tabular-to-FASTA on Dataset #4. This works (at least for reasonably sized datasets), but requires three steps and the creation of at least two temporary files. I'd like to introduce another tool under "FASTA manipulation" to do it on one step (rather than three). Am I going against the apparent Galaxy ideal that complex manipulations should be done with tabular files? Would such a FASTA filter tool be of interest to add directly to Galaxy (e.g. under the "FASTA manipulation" section), or better off on the community tool shed? Here is my current implementation for discussion/consideration: http://bitbucket.org/peterjc/galaxy-central/changeset/730b89c4da26 Thanks, Peter

2 6