September 2013 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Tool Shed Tests failing with sniffer/ftype problem
by Peter Cock 24 Oct '13

24 Oct '13

Retitling thread to focus on a regression in the (Tool Shed) test framework to do with input file types and sniffers. On Thu, Aug 1, 2013 at 11:47 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > On Thu, Aug 1, 2013 at 11:43 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: >> On Mon, Jul 29, 2013 at 5:03 PM, Dave Bouvier <dave(a)bx.psu.edu> wrote: >>> Peter, >>> >>> The issue with seq_rename incorrectly failing functional tests has been >>> resolved in 10266:fe04978dadac, and the test results will be corrected the >>> next time the automated testing framework runs. I'm still looking into the >>> other issues you reported. >>> >>> --Dave B. >> >> Hi Dave, >> >> The seq_rename test results (apparently) from last night are still failing, >> http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/e440681dd17c >> >> AssertionError: Attempting to set field 'new_column' to value '['2']' >> in form 'tool_form' threw exception: cannot find value/label "2" in >> list control control: <SelectControl(new_column=[1])> If the above >> control is a DataToolparameter whose data type class does not include >> a sniff() method, make sure to include a proper 'ftype' attribute to >> the tag for the control within the <test> tag set. > > Same issue here?, > > http://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus/2be36fa7565e > > AssertionError: Attempting to set field 'fasta_file' to value > '['empty.fasta']' in form 'tool_form' threw exception: cannot find > value/label "empty.fasta" in list control control: > <SelectControl(fasta_file=[])> If the above control is a > DataToolparameter whose data type class does not include a sniff() > method, make sure to include a proper 'ftype' attribute to the tag for > the control within the <test> tag set. And another case which looks the same, http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/392279f2e120 AssertionError: Attempting to set field 'fasta_file' to value '['empty.fasta']' in form 'tool_form' threw exception: cannot find value/label "empty.fasta" in list control control: <SelectControl(fasta_file=[])> If the above control is a DataToolparameter whose data type class does not include a sniff() method, make sure to include a proper 'ftype' attribute to the tag for the control within the <test> tag set. Note that for the nlstradamus and effectivet3 tests this is using an empty FASTA file with no sequences in it, so the sniffer will not be able guess this is a FASTA file - which is why the tool text gives the ftype explicitly. I originally used a literally empty fasta file (0 bytes) but as I recall the (non-Tool Shed) framework at the time didn't like me doing that in the test framework, so this empty.fasta file just has a couple of blank lines in it. I am testing with empty FASTA files deliberately - they can often occur in gene/protein selection pipelines where an early part of the pipeline is stringent enough to reject all the sequences. However, the seq_rename sniffer problem is not to do with an 'empty' fasta file - it uses a non-empty tabular file. My guess is somewhere in the test framework the ftype attribute is being lost. This is still working locally, $ ./run_functional_tests.sh -id seq_rename $ ./run_functional_tests.sh -id nlstradamus $ ./run_functional_tests.sh -id effectiveT3 Regards, Peter

2 7

Test Toolshed Biopython package dependency & NumPy
by Peter Cock 22 Oct '13

22 Oct '13

On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote: > I updated R, numpy, scipy, scikit and removed the atlas dependency. It > seems to work fine for the ChemicalToolBox. I do not remove the lapack > dependency, because I did not get any complains until now. I'm seeing problems on the Test Tool Shed with NumPy :( The updated no-ATLAS numpy is revision f0490401cfff http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 My tools using NumPy via Biopython 1.62 do seem to be picking up this new NumPy package, which is good. However, the install of NumPy seems to be failing in strange and *different* ways - which I suspect points to a Tool Shed test framework bug, perhaps in sharing the dependencies between tools? ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id /9e9e3b860aa0 Looks like partial NumPy install? Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core' ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0 Apparently the same error: Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core' ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce Looks like a different partial NumPy install? Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 393, in <module> if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/core/__init__.py", line 42, in <module> __all__ += shape_base.__all__ AttributeError: 'module' object has no attribute '__all__' ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 Looks like yet another different partial NumPy install? Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Bio/Cluster/clustermodule.c: In function ‘py_median’: Bio/Cluster/clustermodule.c:2164:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) Bio/Cluster/clustermodule.c:2164:31: note: each undeclared identifier is reported only once for each function it appears in Bio/Cluster/clustermodule.c: In function ‘py_mean’: Bio/Cluster/clustermodule.c:2230:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb My old favourite, listed under "Latest revision: failing tool tests" but with no visible test results or install failure - but as this also depends on Biopython and NumPy so it probably failed installation too. Regards, Peter

4 18

Functional tests results in Invalid HTTP return code 404/500, allowed codes: 200
by Serrano Pereira 19 Oct '13

19 Oct '13

Dear Galaxy hackers, I'm having trouble with the tool test framework. I have a local installation of a Galaxy Tool Shed (latest stable from the repository) on my Linux server. I have an Apache proxy to my tool shed (per http://wiki.galaxyproject.org/HostingALocalToolShed) The Tool Shed seems to work fine. I'm able to create and maintain repositories and I'm able to install tools from the Tool Shed from remote Galaxy instances. Now I want to run the functional tests of the tools in the Tool Shed. I do that by following the instructions on the wiki (http://wiki.galaxyproject.org/AutomatedToolTests) It comes down to the following commands: ---------------------------------------------------------------------- cd /opt/galaxy/galaxy-dist/ export GALAXY_INSTALL_TEST_TOOL_SHED_API_KEY=2aeecfe2d9f28d0c5b6959a41886b5fd export GALAXY_INSTALL_TEST_TOOL_SHED_URL=http://toolshed.galaxy.domain.org/ python lib/tool_shed/scripts/check_repositories_for_functional_tests.py tool_shed_wsgi.ini sh install_and_test_tool_shed_repositories.sh ---------------------------------------------------------------------- The second script (install_and_test_tool_shed_repositories.sh) fails with the following error: ====================================================================== FAIL: install_repository_usearch (install_and_test_tool_shed_repositories.functional.test_install_repositories.TestInstallRepository_usearch) Install the repository usearch from http://toolshed.galaxy.domain.org/. ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/functional/test_install_repositories.py", line 48, in test_install_repository self.do_installation( repository_dict ) File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/functional/test_install_repositories.py", line 17, in do_installation self.install_repository( repository_info_dict ) File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/base/twilltestcase.py", line 61, in install_repository self.visit_url( '%s/repository/preview_tools_in_changeset?%s' % ( tool_shed_url, preview_params ) ) File "/opt/galaxy/galaxy-dist/test/install_and_test_tool_shed_repositories/base/twilltestcase.py", line 112, in visit_url ( return_code, url, ', '.join( str( code ) for code in allowed_codes ) ) AssertionError: Invalid HTTP return code 404 for http://toolshed.galaxy.domain.org//repository/preview_tools_in_changeset?re…, allowed codes: 200 ---------------------------------------------------------------------- (I've changed twilltestcase.py to also print the URL that is being browsed.) This is were it get's weird. The HTTP 404 response code indicates that the page was not found. However, when I browse to that URL, it works fine! What's more, when I try to access that page with `wget` from the same machine the HTTP response code is 200: ---------------------------------------------------------------------- wget http://toolshed.galaxy.domain.org//repository/preview_tools_in_changeset?re… ... HTTP request sent, awaiting response... 200 OK ... ---------------------------------------------------------------------- When I use the Tool Shed URL http://localhost:9009/ instead, I get a similar error, only the return code is 500 instead of 404. I suspect that the HTTP return code returned by the Python module that returns it (Mechanize?) is invalid. I found a note about this error on the wiki (http://wiki.galaxyproject.org/Admin/Running%20Tests) "Note: If you have another version of paste installed in your PYTHONPATH, it can cause problems running the functional tests. Many tests will error (instead of passing/failing) because of an HTML code 500 where a 200 was expected. There are two workarounds for this: uninstall paste or use virtualenv. " The thing is, I do not have another version of Paste installed. What can I do to fix this error? Regards, Serrano Pereira

2 5

Deploying LOC files for tool built-in data during a tool installation
by Jean-Frédéric Berthelot 17 Oct '13

17 Oct '13

Hi list, The tool I am currently wrapping has built-in data, which may be used by the tool users (through a relevant < from_data_table> + .LOC file configuration). They are .fasta databases which are rather small and are thus bundled in the tool distribution package. Thanks to the tool_dependencies.xml file, said distribution package is downloaded at install time, code is compiled, and since they are here, the data files are copied to $INSTAL L_DIR too , ready to be used. After that, the user still has to edit tool-data/my_fancy_data_files.loc ; but the thing is, during the install I know where these data files are (since I copied those there), so I would like to save the user the trouble and set up this file automagically. I would have two questions: 1/ Is it okay to have tool built-in data files in $INSTAL L_DIR, or would it be considered bad practice? 2/ Is there a way to set up the tool-data/my_fancy_data_files.loc during the install? Here are the options I though of: *shipping a “real” my_fancy_data_files.loc.sample with the good paths already set-up, which is going to be copied as the .loc file (a rather ugly hack) *using more <action type="shell_command"> during install to create my_fancy_data_files.loc (but deploying this file it is not part of the tool dependency install per se) *variant of the previous : shipping my_fancy_data_files.loc as part of the tool distribution package, and copy it through shell_command (same concern than above). Any thoughts? Cheers, -- Jean-Frédéric Bonsai Bioinformatics group

5 10

Can not find hg_g1k_v37.fa.
by shenwiyn 16 Oct '13

16 Oct '13

Hi everyone, I want to use the hg_g1k_v37.fa on my local Galaxy,but I can not find this data.When I use the command : rsync -avzP rsync://datacache.g2.bx.psu.edu/indexes/hg_g1k_v37/seq/hg_g1k_v37.fa the error occurred: receiving incremental file list rsync: link_stat "/hg_g1k_v37/seq/hg_g1k_v37.fa" (in indexes) failed: No such file or directory (2) sent 4 bytes received 6 bytes 4.00 bytes/sec total size is 0 speedup is 0.00 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1532) [receiver=3.0.6] Please any suggestions would be most welcome. Thanks. shenwiyn

3 4

cleanup_datasets.py with postgresql 8.1
by Jennifer Jackson 11 Oct '13

11 Oct '13

Posting to galaxy-dev(a)bx.psu.edu to give the question better exposure to the development community. Please _remove the galaxy-user(a)bx.psu.edu mailing list from all replies_ - we want to avoid double posts. > For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev On 9/26/13 7:29 AM, Malcolm Tobias wrote: > > All, > > My understanding is that the cleanup_datasets.py under > scripts/cleanup_datasets should be compatible with 'older' versions of > postgresql. I'm running 8.1 under CentOS 5. When I attempt to run the > scripts, it fails to clean any data sets. From the logs, I'm noticing > messages like: > > database_connection contains an unknown SQLAlchemy database dialect: > postgresql > > In my universe_wsgi.ini I've defined the database_connection as: > > database_connection = postgresql://galaxy:********@localhost:5432/galaxydb > > and am confident it's working as Galaxy works fine, other than this > problem with deleting older datasets. > > The pgcleanup.py script isn't an option, as it requires postgresql > >=9.1. I noticed yet another version of cleanup_datasets.py under cron > which looks like it might be compatible with postgres, but am leary of > trying it as I'm not sure what it means by "1: database directory to > clean". I'm not sure what directory to point this to, nor am I > confident that just deleting data from the directory is the safest thing. > > Any suggestions on how to proceed? > > Thanks! > > Malcolm > > -- > > Malcolm Tobias > > 314.362.1594 > > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org

2 2

Unicode in tool stderr crashing galaxy?
by Guest, Simon 10 Oct '13

10 Oct '13

My optimism was short lived. I think I've hit a galaxy bug in Unicode processing. Having managed to get the deseq tool not to crash galaxy in the sqlalchemy layer, I now get a crash when clicking on the info->stderr for the tool output dataset, probably because it contains Unicode. Here's the traceback from paster.log: ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 47124) Traceback (most recent call last): File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__ self.handle() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 442, in handle BaseHTTPRequestHandler.handle(self) File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle self.handle_one_request() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 437, in handle_one_request self.wsgi_execute() File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 290, in wsgi_execute self.wsgi_write_chunk(chunk) File "/home/galaxy-dev/galaxy/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py", line 150, in wsgi_write_chunk self.wfile.write(chunk) File "/usr/lib64/python2.6/socket.py", line 316, in write data = str(data) # XXX Should really reject non-string non-buffers UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 169: ordinal not in range(128) ---------------------------------------- I don't know where to look to find the actual stderr output. Any pointers? Is this actually a galaxy bug now? cheers, Simon ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================

3 4

configuring auth_ldap for galaxy
by Iry Witham 01 Oct '13

01 Oct '13

Hi Team, Has anyone had any experience with setting up auth_ldap for nginx? I am attempting to make this work, but nginx fails to start with the following message: Starting nginx nginx: [emerg] unknown directive "auth_ldap_url" in /usr/local/nginx/conf/nginx.conf:48 startproc: exit status of parent of /usr/local/nginx/sbin/nginx: 1 failed Any help would be appreciated. Thanks, Iry Witham The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 4

Download URL not accepted in tool_dependencies.xml
by Joachim Jacob | VIB | 01 Oct '13

01 Oct '13

Hi all, The download URL seems not to be accepted in tool_dependencies.xml. It is an URL from sourceforge, the direct link to a package. The error upon uploading the tool to my toolshed: ************** Metadata may have been defined for some items in revision '5665a799775d'. Correct the following problems if necessary and reset metadata. tool_dependencies.xml - Exception attempting to parse /mnt/toolsheddb/database/000/repo_26/tool_dependencies.xml: not well-formed (invalid token): line 6, column 149 ************** The tool_dependencies.xml file: ************** 1 <?xml version="1.0"?> 2 <tool_dependency> 3 <package name="transpose" version="2.0.0"> 4 <install version="1.0"> 5 <actions> 6 <action type="download_by_url">http://downloads.sourceforge.net/project/transpose/transpose/transpose-2.0/…</action> 7 <action type="shell_command">mkdir bin</action> 8 <action type="shell_command">unzip transpose-2.0.zip</action> <action type="shell_command">cd transpose-2.0/src</action> <action type="shell_command">gcc transpose.c -o transpose</action> <action type="move_file"> <source>$INSTALL_DIR/transpose-2.0/src/transpose</source> <destination>$INSTALL_DIR/bin</destination> </action> <action type="shell_command">chmod +x $INSTALL_DIR/bin/transpose</action> <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> <readme> Compiling transpose and putting in the path. </readme> </package> </tool_dependency> ************** Cheers, Joachim -- Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team

5 10

Dynamic data library
by Cole, Nathan (NIH/NCI) [C] 01 Oct '13

01 Oct '13

Hello, we've set up a local Galaxy instance in our genotyping and next-gen sequencing lab with local Apache LDAP (AD) integration, NFS mounts to a large NAS, and cluster integration coming. Do to the high volume of samples and staff that will be using the system, I want to set up data libraries (without copying to Galaxy). This is obviously no problem the first time, however I was wondering if there was a way to make a library, added from a system path, be dynamic so that it would stay synchronized with the underlying file structure? If a try dynamic library is not possible, is there a method for adding files to an existing library via that same system path that would not duplicate all of the original files in the data library? I did some scouring of the list and found some old unanswered questions and some things tangentially related topics, but I was unable to find a true answer or solution to my problem. Any information on how to do the tasks above or other solutions to provide the same functionality would be greatly appreciated. Thanks, Nathan

3 2