January 2011 - galaxy-dev - lists.galaxyproject.org

Puzzling test failure with data_column parameter
by Peter Cock 24 Apr '13

24 Apr '13

Hi all, I'm working on a FASTA filter script, as per this email: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-November/003819.html Current code here: http://bitbucket.org/peterjc/galaxy-central/src/c3ac6d7a02f7/ I've added a basic test, but it won't run (see output below). I think the interesting bit of the error output is this exception: cannot find value/label "blastp_four_human_vs_rhodopsin.tabular" in list control, coming from function set_form_control_value in the twill.utils module. This is something to do with the columns parameter, a multiselect option referencing the columns in this tabular file. Removing the validator makes no difference. However, if I change the columns parameter to a plain text parameter, the test passes: $ hg diff diff -r c3ac6d7a02f7 tools/fasta_tools/fasta_filter_by_id.xml --- a/tools/fasta_tools/fasta_filter_by_id.xml Tue Nov 23 11:37:35 2010 +0000 +++ b/tools/fasta_tools/fasta_filter_by_id.xml Tue Nov 23 14:03:31 2010 +0000 @@ -4,9 +4,7 @@ <inputs> <param name="input_fasta" type="data" format="fasta" label="FASTA file to filter on the identifiers"/> <param name="input_tabular" type="data" format="tabular" label="Tabular file containing FASTA identifiers"/> - <param name="columns" type="data_column" data_ref="input_tabular" multiple="True" numerical="False" label="Column(s) containing FASTA identifiers" help="Multi-select list - hold the appropriate key while clicking to select multiple columns"> - <validator type="no_options" message="Pick at least one column"/> - </param> + <param name="columns" type="text" label="Column(s) containing FASTA identifiers" /> </inputs> <outputs> <data name="output_pos" format="fasta" label="With matched ID" /> I've tried searching the provided wrappers for similar examples (column multi-select from a tabular file). I found filters/uniq.xml (works but uses a bed file rather than a simple tabular file) and stats/cor.xml (needs R and rpy which I don't have installed yet) which have tests, and finally plotting/bar_chart.xml which has no unit tests. Would someone familiar with the internals of the Galaxy tests and how they set tool parameters be able to try reproducing this for me? The branch has all the unit test files required, and there are no new dependencies needed. Thank you, Peter -- Here is the output (on Linux - other tests tried pass): $ ./run_functional_tests.sh -id fasta_filter_by_id ... nose.plugins.manager DEBUG 2010-11-23 13:59:32,958 DefaultPluginManager load plugin sqlalchemy = sqlalchemy.test.noseplugin:NoseSQLAlchemy nose.plugins.manager DEBUG 2010-11-23 13:59:32,961 DefaultPluginManager load plugin nosetestdiff = nosetestdiff.plugin:NoseTestDiff nose.plugins.manager DEBUG 2010-11-23 13:59:32,962 DefaultPluginManager load plugin nosehtml = nosehtml.plugin:NoseHTML Filter sequences by ID ( fasta_filter_by_id ) > Test-1 ... galaxy.tools.actions.upload_common INFO 2010-11-23 13:59:34,905 tool upload1 created job id 1 galaxy.jobs DEBUG 2010-11-23 13:59:42,709 dispatching job 1 to local runner galaxy.jobs INFO 2010-11-23 13:59:42,899 job 1 dispatched galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:43,213 executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:44,033 execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs DEBUG 2010-11-23 13:59:44,414 job 1 ended galaxy.tools.actions.upload_common INFO 2010-11-23 13:59:49,927 tool upload1 created job id 2 galaxy.jobs DEBUG 2010-11-23 13:59:50,067 dispatching job 2 to local runner galaxy.jobs INFO 2010-11-23 13:59:50,702 job 2 dispatched galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:52,059 executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs.runners.local DEBUG 2010-11-23 13:59:52,874 execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs DEBUG 2010-11-23 13:59:53,420 job 2 ended base.twilltestcase DEBUG 2010-11-23 13:59:53,747 In submit_form, continuing, but caught exception: cannot find value/label "blastp_four_human_vs_rhodopsin.tabular" in list control base.twilltestcase DEBUG 2010-11-23 13:59:54,488 ## files diff on /home/pjcock/repositories/galaxy-central/test-data/four_human_proteins_filter_a.fasta and /tmp/tmpijWxYGfour_human_proteins_filter_a.fasta lines_diff=0, found diff = 61 FAIL ====================================================================== FAIL: Filter sequences by ID ( fasta_filter_by_id ) > Test-1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 160, in test_tool self.do_it( td ) File "/home/pjcock/repositories/galaxy-central/test/functional/test_toolbox.py", line 101, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes ) File "/home/pjcock/repositories/galaxy-central/test/base/twilltestcase.py", line 656, in verify_dataset_correctness raise AssertionError( errmsg ) AssertionError: History item 1 different than expected, difference (using diff): --- local_file +++ history_data @@ -1,2 +1,61 @@ +>sp|Q9BS26|ERP44_HUMAN Endoplasmic reticulum resident protein 44 OS=Homo sapiens GN=ERP44 PE=1 SV=1 +MHPAVFLSLPDLRCSLLLLVTWVFTPVTTEITSLDTENIDEILNNADVALVNFYADWCRF +SQMLHPIFEEASDVIKEEFPNENQVVFARVDCDQHSDIAQRYRISKYPTLKLFRNGMMMK +REYRGQRSVKALADYIRQQKSDPIQEIRDLAEITTLDRSKRNIIGYFEQKDSDNYRVFER +VANILHDDCAFLSAFGDVSKPERYSGDNIIYKPPGHSAPDMVYLGAMTNFDVTYNWIQDK +CVPLVREITFENGEELTEEGLPFLILFHMKEDTESLEIFQNEVARQLISEKGTINFLHAD +CDKFRHPLLHIQKTPADCPVIAIDSFRHMYVFGDFKDVLIPGKLKQFVFDLHSGKLHREF +HHGPDPTDTAPGEQAQDVASSPPESSFQKLAPSEYRYTLLRDRDEL +>sp|Q9NSY1|BMP2K_HUMAN BMP-2-inducible protein kinase OS=Homo sapiens GN=BMP2K PE=1 SV=2 +MKKFSRMPKSEGGSGGGAAGGGAGGAGAGAGCGSGGSSVGVRVFAVGRHQVTLEESLAEG +GFSTVFLVRTHGGIRCALKRMYVNNMPDLNVCKREITIMKELSGHKNIVGYLDCAVNSIS +DNVWEVLILMEYCRAGQVVNQMNKKLQTGFTEPEVLQIFCDTCEAVARLHQCKTPIIHRD +LKVENILLNDGGNYVLCDFGSATNKFLNPQKDGVNVVEEEIKKYTTLSYRAPEMINLYGG +KPITTKADIWALGCLLYKLCFFTLPFGESQVAICDGNFTIPDNSRYSRNIHCLIRFMLEP +DPEHRPDIFQVSYFAFKFAKKDCPVSNINNSSIPSALPEPMTASEAAARKSQIKARITDT +IGPTETSIAPRQRPKANSATTATPSVLTIQSSATPVKVLAPGEFGNHRPKGALRPGNGPE +ILLGQGPPQQPPQQHRVLQQLQQGDWRLQQLHLQHRHPHQQQQQQQQQQQQQQQQQQQQQ +QQQQQQHHHHHHHHLLQDAYMQQYQHATQQQQMLQQQFLMHSVYQPQPSASQYPTMMPQY +QQAFFQQQMLAQHQPSQQQASPEYLTSPQEFSPALVSYTSSLPAQVGTIMDSSYSANRSV +ADKEAIANFTNQKNISNPPDMSGWNPFGEDNFSKLTEEELLDREFDLLRSNRLEERASSD +KNVDSLSAPHNHPPEDPFGSVPFISHSGSPEKKAEHSSINQENGTANPIKNGKTSPASKD +QRTGKKTSVQGQVQKGNDESESDFESDPPSPKSSEEEEQDDEEVLQGEQGDFNDDDTEPE +NLGHRPLLMDSEDEEEEEKHSSDSDYEQAKAKYSDMSSVYRDRSGSGPTQDLNTILLTSA +QLSSDVAVETPKQEFDVFGAVPFFAVRAQQPQQEKNEKNLPQHRFPAAGLEQEEFDVFTK +APFSKKVNVQECHAVGPEAHTIPGYPKSVDVFGSTPFQPFLTSTSKSESNEDLFGLVPFD +EITGSQQQKVKQRSLQKLSSRQRRTKQDMSKSNGKRHHGTPTSTKKTLKPTYRTPERARR +HKKVGRRDSQSSNEFLTISDSKENISVALTDGKDRGNVLQPEESLLDPFGAKPFHSPDLS +WHPPHQGLSDIRADHNTVLPGRPRQNSLHGSFHSADVLKMDDFGAVPFTELVVQSITPHQ +SQQSQPVELDPFGAAPFPSKQ +>sp|P06213|INSR_HUMAN Insulin receptor OS=Homo sapiens GN=INSR PE=1 SV=4 +MATGGRRGAAAAPLLVAVAALLLGAAGHLYPGEVCPGMDIRNNLTRLHELENCSVIEGHL +QILLMFKTRPEDFRDLSFPKLIMITDYLLLFRVYGLESLKDLFPNLTVIRGSRLFFNYAL +VIFEMVHLKELGLYNLMNITRGSVRIEKNNELCYLATIDWSRILDSVEDNYIVLNKDDNE +ECGDICPGTAKGKTNCPATVINGQFVERCWTHSHCQKVCPTICKSHGCTAEGLCCHSECL +GNCSQPDDPTKCVACRNFYLDGRCVETCPPPYYHFQDWRCVNFSFCQDLHHKCKNSRRQG +CHQYVIHNNKCIPECPSGYTMNSSNLLCTPCLGPCPKVCHLLEGEKTIDSVTSAQELRGC +TVINGSLIINIRGGNNLAAELEANLGLIEEISGYLKIRRSYALVSLSFFRKLRLIRGETL -------------------- >> begin captured stdout << --------------------- Uploaded file: four_human_proteins.fasta , ftype: fasta , extra: {'ftype': 'fasta', 'value': 'four_human_proteins.fasta', 'children': []} Uploaded file: blastp_four_human_vs_rhodopsin.tabular , ftype: tabular , extra: {'ftype': 'tabular', 'value': 'blastp_four_human_vs_rhodopsin.tabular', 'children': []} form 'tool_form' contains the following controls ( note the values ) control 0: <HiddenControl(tool_id=fasta_filter_by_id) (readonly)> control 1: <HiddenControl(tool_state=800255c1626362313139396662383639346664316133316565303361646336323163663764636566333538623a376232323566356637303631363736353566356632323361323033303263323032323639366537303735373435663636363137333734363132323361323032323331323232633230323236333666366337353664366537333232336132303232366537353663366332323263323032323639366537303735373435663734363136323735366336313732323233613230323233323232376471002e) (readonly)> control 2: <SelectControl(input_fasta=[*1])> control 3: <SelectControl(input_tabular=[*2])> control 4: <SelectControl(columns=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])> control 5: <SubmitControl(runtool_btn=Execute) (readonly)> page_inputs (0) {'input_fasta': ['four_human_proteins.fasta'], 'columns': ['1'], 'input_tabular': ['blastp_four_human_vs_rhodopsin.tabular']} --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- galaxy.tools.actions.upload_common: INFO: tool upload1 created job id 1 galaxy.jobs: DEBUG: dispatching job 1 to local runner galaxy.jobs: INFO: job 1 dispatched galaxy.jobs.runners.local: DEBUG: executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs.runners.local: DEBUG: execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmpOVlriT 1:/home/pjcock/repositories/galaxy-central/database/job_working_directory/1/dataset_1_files:/tmp/tmpu4uu_o/database/files/000/dataset_1.dat galaxy.jobs: DEBUG: job 1 ended galaxy.tools.actions.upload_common: INFO: tool upload1 created job id 2 galaxy.jobs: DEBUG: dispatching job 2 to local runner galaxy.jobs: INFO: job 2 dispatched galaxy.jobs.runners.local: DEBUG: executing: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs.runners.local: DEBUG: execution finished: python /home/pjcock/repositories/galaxy-central/tools/data_source/upload.py /home/pjcock/repositories/galaxy-central /home/pjcock/repositories/galaxy-central/datatypes_conf.xml /tmp/tmp6Z76yl 2:/home/pjcock/repositories/galaxy-central/database/job_working_directory/2/dataset_2_files:/tmp/tmpu4uu_o/database/files/000/dataset_2.dat galaxy.jobs: DEBUG: job 2 ended base.twilltestcase: DEBUG: In submit_form, continuing, but caught exception: cannot find value/label "blastp_four_human_vs_rhodopsin.tabular" in list control base.twilltestcase: DEBUG: ## files diff on /home/pjcock/repositories/galaxy-central/test-data/four_human_proteins_filter_a.fasta and /tmp/tmpijWxYGfour_human_proteins_filter_a.fasta lines_diff=0, found diff = 61 --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 1 test in 21.504s FAILED (failures=1) functional_tests.py INFO 2010-11-23 13:59:54,494 Shutting down functional_tests.py INFO 2010-11-23 13:59:54,494 Shutting down embedded web server functional_tests.py INFO 2010-11-23 13:59:54,497 Embedded web server stopped functional_tests.py INFO 2010-11-23 13:59:54,497 Shutting down app galaxy.jobs INFO 2010-11-23 13:59:54,497 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:54,498 job queue stopped galaxy.jobs.runners.local INFO 2010-11-23 13:59:54,498 sending stop signal to worker threads galaxy.jobs.runners.local INFO 2010-11-23 13:59:54,498 local job runner stopped galaxy.jobs INFO 2010-11-23 13:59:54,499 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:54,499 job stopper stopped functional_tests.py INFO 2010-11-23 13:59:54,499 Embedded Universe application stopped functional_tests.py INFO 2010-11-23 13:59:54,500 Cleaning up temporary files in /tmp/tmpu4uu_o galaxy.jobs INFO 2010-11-23 13:59:55,410 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:55,411 job queue stopped galaxy.jobs.runners.local INFO 2010-11-23 13:59:55,411 sending stop signal to worker threads galaxy.jobs.runners.local INFO 2010-11-23 13:59:55,411 local job runner stopped galaxy.jobs INFO 2010-11-23 13:59:55,411 sending stop signal to worker thread galaxy.jobs INFO 2010-11-23 13:59:55,412 job stopper stopped 'run_functional_tests.sh help' for help Note: galaxy test framework uses tool_conf.xml.sample, not tool_conf.xml

2 3

Tabular file metadata - columns names
by Peter 22 Feb '13

22 Feb '13

Hi all, I'd like to know more about Galaxy's column metadata for tabular files. In the workflow editor under "Edit Step Actions" you can pick "Assign Columns", and then give column numbers for five predefined cases: Chrom, Start, End, Strand, Name. Do these "named columns" get shown anywhere in the Galaxy UI? For example, in a column select parameter widget? Is it possible to assign these columns in a tool's wrapper XML file? >From http://bitbucket.org/galaxy/galaxy-central/wiki/ToolConfigSyntax I'm aware of the metadata_source attribute to *copy* the meta data from the input file, but that isn't always relevant. Can I somehow specify that my tool has tabular output where column 1 is "Name"? Is it possible to introduce additional column types? e.g. "evalue" or "Description". Thanks, Peter

3 7

run_functional_tests.sh -sid option
by Peter 11 Feb '13

11 Feb '13

Hi, I'm trying to use run_functional_tests.sh to run all the tests for a section (group of tools). I've read the output from: ./run_functional_tests.sh help For example, from the tools_conf.xml.sample we have <section name="ENCODE Tools" id="EncodeTools"> <tool file="encode/gencode_partition.xml" /> <tool file="encode/random_intervals.xml" /> </section> And looking at these tools' XML files, <tool id="gencode_partition1" name="Gencode Partition"> and: <tool id="random_intervals1" name="Random Intervals"> I'd like to run the functional tests for the ENCODE tools, Using the switch for an individual tool id (-id) works, ./run_functional_tests.sh -id gencode_partition1 ... Ran 1 test in 22.302s ... and so does this (well, it say the tool doesn't have any tests which is true in this example): ./run_functional_tests.sh -id random_intervals1 ... Ran 1 test in 0.027s ... However, I also tried using the section id switch (-sid), ./run_functional_tests.sh -sid EncodeTools ... Ran 0 tests in 0.000s ... I also tried using the section name, /run_functional_tests.sh -sid "ENCODE Tools" Is this (-sid not working) a known issue, or am I using it wrong? Thanks, Peter

4 5

Automatically removing items from history
by Kempenaar, M (med) 08 Sep '11

08 Sep '11

Hello, Is it possible to let a tool in a workflow manage the users history? For instance, I have a tool that takes a variable number of user-uploaded CSV files as input and merges them into one file. Before running this tool, the user thus has a number of CSV files in the history which I would like to 'replace' with the single merged file. I was hoping that when I remove a file from disk that it would also be removed from the history, but that only results in a 'The resource could not be found.' error. Alternatively, is it possible to create a composite datafile/history item from uploaded files? In any case, I would like to remove the separately uploaded files from the users history which makes selecting files for the next step(s) easier. (this is a bit of a followup for: http://gmod.827538.n3.nabble.com/Creating-new-datatype-with-variable-number…) One other question, is there (any) documentation available on the Galaxy API or a set of example files? Thanks for your input! Regards, Marcel. De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht. The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

4 4

Sharing tool definitions between XML files
by Peter 02 Jun '11

02 Jun '11

Hi all, I'm wondering if there is any established way to share parameter definitions between tool wrapper XML files? For example, I am currently working on NCBI BLAST+ wrappers, and these tools have a lot in common. For example, the output format option will be a select parameter, and it will be the same for blastn, blastp, tblastn, etc. I can just cut and paste the definition but this seems inelegant and will be a long term maintenance burden if it ever needs updating (lots of places would need to be updated the same way). I'd like to have a shared XML snippet defining this parameter which I could then reference/include from each tool wrapper than needs it. I'm thinking of something like XML's <!ENTITY ...> might work. Is this possible? Peter

2 2

Output file naming / dataset's label
by Tanguy Le Carrour 14 Apr '11

14 Apr '11

Dear Galaxy Team, I started using Galaxy few months ago. I'm now setting up my first "real" workflows. I would like to keep the name of my input file through the various steps as dataset's label, maybe also modify it a bit, and finally give a "proper" name to the result file downloaded. A simple example: 1) date-sample_name.gff3 --[step1]-> date-sample_name-step1.tsv 2) date-sample_name-step1.tsv --[step2]-> date-sample_name-step1-step2.txt 3 download date-sample_name-step1-step2.txt I've seen that I can use the 'label' attribute in the tool config "data" tag to do so. But I don't know how to access the ${input.label} (?) and how to modify it. I doesn't stop me using my workflows, but it would definitely help my users not to get lost when they use them. Best regards, Tanguy Le Carrour -- Tanguy LE CARROUR Functional Genomics Center Zurich ETH Zurich / University of Zurich Winterthurerstrasse 190 8057 Zurich Switzerland

4 8

Restricted user builds
by Dan MacLean (TSL) 28 Mar '11

28 Mar '11

Dear Galaxy, We have an internal installation of Galaxy running very nicely with a fair number of internally generated builds and plenty of data libraries, recently a user asked if it was possible to add a build and restrict to just a few users. Is this possible? I couldn't find any specific documentation on the subject. Thanks for any help Dan

2 1

swap issue
by Isabelle Phan 22 Mar '11

22 Mar '11

Hello, I am running galaxy, using a postgres backend. The galaxy server systematically gets killed whenever I try uploading a bam file. The samtools process is still running. This is the end of the log: 2010-08-26 18:08:11,371 INFO sqlalchemy.engine.threadlocal.TLEngine.0x...4610 {'param_1': 8} sqlalchemy.engine.threadlocal.TLEngine.0x...4610 INFO 2010-08-26 18:08:11,371 {'param_1': 8} 2010-08-26 18:08:11,378 INFO sqlalchemy.pool.QueuePool.0x...4ed0 Connection <connection object at 0x567bf30; dsn: 'dbname=galaxy host=/var closed: 0> being returned to pool sqlalchemy.pool.QueuePool.0x...4ed0 INFO 2010-08-26 18:08:11,378 Connection <connection object at 0x567bf30; dsn: 'dbname=galaxy host=/var closed: 0> being returned to pool 10.1.20.44 - - [26/Aug/2010:18:08:06 -0700] "POST /root/history_item_updates HTTP/1.1" 200 - "http://galaxy:8080/history" "Mozilla/5.0 (X1 (x86_64); en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" Killed I have the following options set in universe_wsgi.ini: database_engine_option_server_side_cursors = True database_engine_option_strategy = threadlocal It seems to be running out of swap. I monitored memory usage with 'top' and galaxy died when available swap hit 0k free. I am not superuser on the machine where galaxy is running and I have no other server available. Is there something I could do to prevent galaxy from dying? Thanks, Isabelle -- Isabelle Phan, DPhil Seattle Biomedical Research Institute +1(206)256 7113

3 4

requested number of bytes is more than a Python string can hold
by Glen Beane 01 Mar '11

01 Mar '11

one of our users is experiencing this error from the SAM-to-BAM tool: Error extracting alignments from (/hpcdata/galaxy-setup/galaxy-dist/database/files/000/dataset_85.dat), requested number of bytes is more than a Python string can hold The SAM file has 58 million lines of mapped paired end reads. Any suggestions? The galaxy jobs running on our cluster are currently using the version of Python installed with CentOS 5 (python 2.4.3, 64-bit). We have been considering setting up a new python virtualenv for the galaxy user on the cluster that uses a newer version of Python (probably 2.6 so it matches the version on the server running the galaxy web app) -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153

2 4

Re: [galaxy-dev] Error with setuptools version in Galaxy installation on Cluster
by Nate Coraor 25 Feb '11

25 Feb '11

Hi Sonali, Since this is a local installation question, I have moved the discussion to galaxy-dev. Further responses are inline below. Sonali Amonkar wrote: > Hi, > > First of all I would like to appreciate the clean and neat documentation written for the Galaxy setup for Cluster. > > I am attempting to install Galaxy on Penguin cluster with Torque PBS manager. > While following the steps I came across an error as below: > > =========================================================================================================================================================== > [user@server pbs_python]$ /galaxy/Python-2.4.6/python scramble.py > > --------------------------------------------------------------------------- > This script requires setuptools version 0.6c12 to run (even to display > help). I will attempt to download it for you (from > http://pypi.python.org/packages/2.4/s/setuptools/) but > you may need to enable firewall access for this script first. > I will start the download in 8 seconds. > > (Note: if this machine does not have network access, please obtain the file > > http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c12-py2.4.egg > > and place it in this directory before rerunning this script.) > --------------------------------------------------------------------------- > Downloading http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c12-py2.4.egg > Traceback (most recent call last): > File "scramble.py", line 14, in ? > from scramble_lib import * > File "../../../lib/scramble_lib.py", line 150, in ? > use_setuptools( download_delay=8, to_dir=os.path.dirname( __file__ ) ) > File "/galaxy/galaxy-dist/scripts/scramble/build/py2.4-linux-x86_64-ucs2/pbs_python/ez_setup.py", line 92, in use_setuptools > return do_download() > File "/galaxy/galaxy-dist/scripts/scramble/build/py2.4-linux-x86_64-ucs2/pbs_python/ez_setup.py", line 70, in do_download > egg = download_setuptools(version, download_base, to_dir, download_delay) > File "/galaxy/galaxy-dist/scripts/scramble/build/py2.4-linux-x86_64-ucs2/pbs_python/ez_setup.py", line 131, in download_setuptools > src = urllib2.urlopen(url) > File "/galaxy/Python-2.4.6/Lib/urllib2.py", line 130, in urlopen > return _opener.open(url, data) > File "/galaxy/Python-2.4.6/Lib/urllib2.py", line 364, in open > response = meth(req, response) > File "/galaxy/Python-2.4.6/Lib/urllib2.py", line 471, in http_response > response = self.parent.error( > File "/galaxy/Python-2.4.6/Lib/urllib2.py", line 402, in error > return self._call_chain(*args) > File "/galaxy/Python-2.4.6/Lib/urllib2.py", line 337, in _call_chain > result = func(*args) > File "/galaxy/Python-2.4.6/Lib/urllib2.py", line 480, in http_error_default > raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) > urllib2.HTTPError: HTTP Error 404: Not Found > =========================================================================================================================================================== > > > The scramble script here is trying to download an egg which does not exist at the location (try opening the link it is trying to download in a browser - you'll get 404, then see the parent directory - http://pypi.python.org/packages/2.4/s/setuptools/ you'll see last version available i.e. 0.6c11). > Hi Sonali, 0.6c12 is still in development and I'm not sure where this is coming from. The version of ez_setup.py in: /galaxy/galaxy-dist/scripts/scramble/lib/ references 0.6c11, and this file is copied to the build directory in: galaxy-dist/scripts/scramble/build/py2.4-linux-x86_64-ucs2/pbs_python/ What's the value of DEFAULT_VERSION in: /galaxy/galaxy-dist/scripts/scramble/build/py2.4-linux-x86_64-ucs2/pbs_python/ez_setup.py ? > If we try to download the last version of setuptools 0.6c11 and rename it to the latest version (desperate attempt to make it work), that doesn't work either. > > =========================================================================================================================================================== > [user@server pbs_python]$ mv setuptools-0.6c11-py2.4.egg setuptools-0.6c12-py2.4.egg > > [user@server pbs_python]$ /galaxy/Python-2.4.6/python scramble.py > checking for pbs-config... /usr/lib64/../bin/pbs-config > Found torque version: 2.4.0-snap.200812091621 > checking for python... /galaxy/Python-2.4.6/python > checking for python version... 2.4 > checking for python platform... linux2 > checking for python script directory... ${prefix}/lib/python2.4/site-packages > checking for python extension module directory... ${exec_prefix}/lib/python2.4/site-packages > configure: creating ./config.status > config.status: creating Makefile > config.status: creating setup.py > scramble(): Patching setup.py > Traceback (most recent call last): > File "scramble.py", line 49, in ? > execfile( "setup.py", globals(), locals() ) > File "setup.py", line 32, in ? > build_version = int(tmp[2]) > ValueError: invalid literal for int(): 0-snap > > =========================================================================================================================================================== This is a problem with your TORQUE version and pbs_python's assumptions about its version numbering. pbs_python expects it to be all-numeric whereas yours is a development snapshot with a non-integer value in the revision portion of the version (2.4.0-snap.200812091621). ^^^^^^^^^^^^^^^^^^^ There are a few ways to proceed. One would be to upgrade your TORQUE client to a release version. Another would be to try the drmaa job runner instead of pbs, since TORQUE provides a DRMAA C library, although I don't believe anyone has used it with TORQUE yet. The last would be to modify pbs_python's build process. Start by changing directories to: /galaxy/galaxy-dist/scripts/scramble/build/py2.4-linux-x86_64-ucs2/pbs_python Edit setup.py and comment all of the following: #VERSION = "2.4.0-snap.200812091621" #tmp = VERSION.split('.') #major_version = int(tmp[0]) #minor_version = int(tmp[1]) #build_version = int(tmp[2]) #if major_version >= 2 and minor_version >= 4 and build_version >= 7: # os.symlink('pbs_wrap_2.4.c', 'pbs_wrap.c') # os.symlink('pbs_2.4.py', 'pbs.py') # TORQUE_VERSION='TORQUE_2_4' #else: And then force the old version by un-indenting or doing something silly like: if True: os.symlink('pbs_wrap_2.1.c', 'pbs_wrap.c') os.symlink('pbs_2.1.py', 'pbs.py') TORQUE_VERSION='TORQUE_OLD' However, running scramble.py again will overwrite your changes, so you will also need to modify scramble.py (in the same directory, not the main scramble.py in /galaxy/galaxy-dist/scripts) and comment out: #run( 'sh configure --with-pbsdir=%s' % os.environ['LIBTORQUE_DIR'], os.getcwd(), 'Running pbs_python configure script' ) Once this is done you should be able to run the *LOCAL* scramble.py in the build directory: $ /galaxy/Python-2.4.6/python ./scramble.py If this succeeds, the egg will be built and placed in the dist/ subdirectory. The egg can then be copied to: /galaxy/galaxy-dist/eggs/ Please let us know how this goes. --nate > > Do let me know if anyone else has faced this issue of the scramble script trying to download a wrong version? Greatly appreciate your time. > > Warm Regards, > Sonali Amonkar > > > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. > _______________________________________________ > galaxy-user mailing list > galaxy-user(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-user

2 9