Test Toolshed Biopython package dependency & NumPy
On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote:
I updated R, numpy, scipy, scikit and removed the atlas dependency. It seems to work fine for the ChemicalToolBox. I do not remove the lapack dependency, because I did not get any complains until now.
I'm seeing problems on the Test Tool Shed with NumPy :( The updated no-ATLAS numpy is revision f0490401cfff http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 My tools using NumPy via Biopython 1.62 do seem to be picking up this new NumPy package, which is good. However, the install of NumPy seems to be failing in strange and *different* ways - which I suspect points to a Tool Shed test framework bug, perhaps in sharing the dependencies between tools? ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id /9e9e3b860aa0 Looks like partial NumPy install? Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core' ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0 Apparently the same error: Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core' ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce Looks like a different partial NumPy install? Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 393, in <module> if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/core/__init__.py", line 42, in <module> __all__ += shape_base.__all__ AttributeError: 'module' object has no attribute '__all__' ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 Looks like yet another different partial NumPy install? Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Bio/Cluster/clustermodule.c: In function ‘py_median’: Bio/Cluster/clustermodule.c:2164:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) Bio/Cluster/clustermodule.c:2164:31: note: each undeclared identifier is reported only once for each function it appears in Bio/Cluster/clustermodule.c: In function ‘py_mean’: Bio/Cluster/clustermodule.c:2230:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------- http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb My old favourite, listed under "Latest revision: failing tool tests" but with no visible test results or install failure - but as this also depends on Biopython and NumPy so it probably failed installation too. Regards, Peter
Hi, it does not mean anything, but for me its working locally on a Fedora (stable) machine. If I can anything do to fix that, please let me know. Bjoern
On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote:
I updated R, numpy, scipy, scikit and removed the atlas dependency. It seems to work fine for the ChemicalToolBox. I do not remove the lapack dependency, because I did not get any complains until now.
I'm seeing problems on the Test Tool Shed with NumPy :(
The updated no-ATLAS numpy is revision f0490401cfff http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7
My tools using NumPy via Biopython 1.62 do seem to be picking up this new NumPy package, which is good.
However, the install of NumPy seems to be failing in strange and *different* ways - which I suspect points to a Tool Shed test framework bug, perhaps in sharing the dependencies between tools?
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id /9e9e3b860aa0 Looks like partial NumPy install?
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core'
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0 Apparently the same error:
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core'
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce Looks like a different partial NumPy install?
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 393, in <module> if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/core/__init__.py", line 42, in <module> __all__ += shape_base.__all__ AttributeError: 'module' object has no attribute '__all__'
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 Looks like yet another different partial NumPy install?
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Bio/Cluster/clustermodule.c: In function ‘py_median’: Bio/Cluster/clustermodule.c:2164:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) Bio/Cluster/clustermodule.c:2164:31: note: each undeclared identifier is reported only once for each function it appears in Bio/Cluster/clustermodule.c: In function ‘py_mean’: Bio/Cluster/clustermodule.c:2230:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb My old favourite, listed under "Latest revision: failing tool tests" but with no visible test results or install failure - but as this also depends on Biopython and NumPy so it probably failed installation too.
Regards,
Peter
On Mon, Sep 30, 2013 at 12:23 PM, Bjoern Gruening <bjoern.gruening@gmail.com> wrote:
Hi,
it does not mean anything, but for me its working locally on a Fedora (stable) machine.
If I can anything do to fix that, please let me know. Bjoern
Thanks Bjoern - I'm pretty sure this is a Tool Shed issue, rather than the Biopython or NumPy package definitions themselves. Peter
Peter, It looks like there may be a version conflict between package_numpy and either a system numpy or the Galaxy numpy egg. I'll investigate and keep you updated on my progress. --Dave B. On 09/30/2013 05:08 AM, Peter Cock wrote:
On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote:
I updated R, numpy, scipy, scikit and removed the atlas dependency. It seems to work fine for the ChemicalToolBox. I do not remove the lapack dependency, because I did not get any complains until now.
I'm seeing problems on the Test Tool Shed with NumPy :(
The updated no-ATLAS numpy is revision f0490401cfff http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7
My tools using NumPy via Biopython 1.62 do seem to be picking up this new NumPy package, which is good.
However, the install of NumPy seems to be failing in strange and *different* ways - which I suspect points to a Tool Shed test framework bug, perhaps in sharing the dependencies between tools?
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id /9e9e3b860aa0 Looks like partial NumPy install?
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core'
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0 Apparently the same error:
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 463, in <module> setup(**setup_args) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "setup.py", line 216, in run if check_dependencies_once(): File "setup.py", line 123, in check_dependencies_once _CHECKED = check_dependencies() File "setup.py", line 163, in check_dependencies if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/polynomial.py", line 11, in <module> import numpy.core.numeric as NX AttributeError: 'module' object has no attribute 'core'
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce Looks like a different partial NumPy install?
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Traceback (most recent call last): File "setup.py", line 393, in <module> if is_Numpy_installed(): File "setup.py", line 282, in is_Numpy_installed return bool(can_import("numpy")) File "setup.py", line 274, in can_import return __import__(module_name) File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/__init__.py", line 137, in <module> import add_newdocs File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/ToolDependencies/numpy/1.7.1/iuc/package_numpy_1_7/f0490401cfff/lib/python/numpy/core/__init__.py", line 42, in <module> __all__ += shape_base.__all__ AttributeError: 'module' object has no attribute '__all__'
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 Looks like yet another different partial NumPy install?
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error Bio/Cluster/clustermodule.c: In function ‘py_median’: Bio/Cluster/clustermodule.c:2164:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) Bio/Cluster/clustermodule.c:2164:31: note: each undeclared identifier is reported only once for each function it appears in Bio/Cluster/clustermodule.c: In function ‘py_mean’: Bio/Cluster/clustermodule.c:2230:31: error: ‘PyArray_NOTYPE’ undeclared (first use in this function) error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------
http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb My old favourite, listed under "Latest revision: failing tool tests" but with no visible test results or install failure - but as this also depends on Biopython and NumPy so it probably failed installation too.
Regards,
Peter
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Mon, Sep 30, 2013 at 1:58 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
It looks like there may be a version conflict between package_numpy and either a system numpy or the Galaxy numpy egg. I'll investigate and keep you updated on my progress.
--Dave B.
Thanks Dave, If Tool authors can assume NumPy is already present via the Galaxy numpy egg and/or the system Python's numpy, that might be simpler... Peter
Am Montag, den 30.09.2013, 14:02 +0100 schrieb Peter Cock:
On Mon, Sep 30, 2013 at 1:58 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
It looks like there may be a version conflict between package_numpy and either a system numpy or the Galaxy numpy egg. I'll investigate and keep you updated on my progress.
--Dave B.
Thanks Dave,
If Tool authors can assume NumPy is already present via the Galaxy numpy egg and/or the system Python's numpy, that might be simpler...
The question is how to import the eggs into the tools? Is that possible at all? For example pysam is shipped with Galaxy right? But is there a way to use it inside of a tool? But I think even if that is possible we need separate pysam/numpy repositories for reproducibility. Cheers, Bjoern
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I've done some investigating, and it looks like there is an irregularity with the installation of package_numpy_1_7, and somehow package_biopython_1_62 is not finding the (successfully) installed numpy dependency. I duplicated the dependency chain for seq_select_by_id on my local tool shed, and got the following error for biopython's tool dependency record when installing seq_select_by_id into my local Galaxy instance: export PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python && export PATH=$PATH:$PATH_NUMPY && export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY && python setup.py install --install-lib /var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python STDOUT running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org Note that when I installed seq_select_by_id, numpy installed correctly, and PYTHONPATH was populated with the right values. --Dave B. On 09/30/2013 09:10 AM, Bjoern Gruening wrote:
Am Montag, den 30.09.2013, 14:02 +0100 schrieb Peter Cock:
On Mon, Sep 30, 2013 at 1:58 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter,
It looks like there may be a version conflict between package_numpy and either a system numpy or the Galaxy numpy egg. I'll investigate and keep you updated on my progress.
--Dave B.
Thanks Dave,
If Tool authors can assume NumPy is already present via the Galaxy numpy egg and/or the system Python's numpy, that might be simpler...
The question is how to import the eggs into the tools? Is that possible at all? For example pysam is shipped with Galaxy right? But is there a way to use it inside of a tool? But I think even if that is possible we need separate pysam/numpy repositories for reproducibility.
I agree with Björn, it's probably not a good idea to use Galaxy eggs in tools and their dependencies, since the egg versions can change at any time, which might impact reproducibilty.
Cheers, Bjoern
Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On Tue, Oct 1, 2013 at 9:18 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
I've done some investigating, and it looks like there is an irregularity with the installation of package_numpy_1_7, and somehow package_biopython_1_62 is not finding the (successfully) installed numpy dependency. I duplicated the dependency chain for seq_select_by_id on my local tool shed, and got the following error for biopython's tool dependency record when installing seq_select_by_id into my local Galaxy instance:
export PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python && export PATH=$PATH:$PATH_NUMPY && export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY && python setup.py install --install-lib /var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python
STDOUT running install
Numerical Python (NumPy) is not installed.
This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython.
You can find NumPy at http://numpy.scipy.org
Note that when I installed seq_select_by_id, numpy installed correctly, and PYTHONPATH was populated with the right values.
--Dave B.
Hi Dave, Bjeorn, Right now NumPy isn't installing at all on the Test Tool Shed. Dave - is is possible that the system was updated and the Fortran compiler removed? e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 reports: Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion lapack package 3.4.2 Error CMake Error: your Fortran compiler: "CMAKE_Fortran_COMPILER-NOTFOUND" was not found. Please set CMAKE_Fortran_COMPILER to a valid compiler path or name. CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed Since this seems to be breaking lapack, perhaps we should further simplify package_numpy_1_7 not to use lapack or ATLAS? What do you think Bjoern? http://toolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 Regards, Peter
Am Montag, den 07.10.2013, 14:07 +0100 schrieb Peter Cock:
On Tue, Oct 1, 2013 at 9:18 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
I've done some investigating, and it looks like there is an irregularity with the installation of package_numpy_1_7, and somehow package_biopython_1_62 is not finding the (successfully) installed numpy dependency. I duplicated the dependency chain for seq_select_by_id on my local tool shed, and got the following error for biopython's tool dependency record when installing seq_select_by_id into my local Galaxy instance:
export PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python && export PATH=$PATH:$PATH_NUMPY && export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY && python setup.py install --install-lib /var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python
STDOUT running install
Numerical Python (NumPy) is not installed.
This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython.
You can find NumPy at http://numpy.scipy.org
Note that when I installed seq_select_by_id, numpy installed correctly, and PYTHONPATH was populated with the right values.
--Dave B.
Hi Dave, Bjeorn,
Right now NumPy isn't installing at all on the Test Tool Shed.
Dave - is is possible that the system was updated and the Fortran compiler removed? e.g.
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 reports:
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion lapack package 3.4.2 Error CMake Error: your Fortran compiler: "CMAKE_Fortran_COMPILER-NOTFOUND" was not found. Please set CMAKE_Fortran_COMPILER to a valid compiler path or name. CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed CMake Error at /usr/share/cmake-2.8/Modules/CMakeFortranInformation.cmake:27 (get_filename_component): get_filename_component called with incorrect number of arguments Call Stack (most recent call first): CMakeLists.txt:2 (project) CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage CMake Error: Internal CMake error, TryCompile configure of cmake failed
Since this seems to be breaking lapack, perhaps we should further simplify package_numpy_1_7 not to use lapack or ATLAS? What do you think Bjoern?
I'm a little bit sceptic to remove features only because the Test Utility can not handle it. On the other hand I understand that numpy should just work and that your test failing since month because of my dependencies. I will get rid of it tomorrow. As a result we can setup_virtualenv and install numpy from pip directly I think. Is that ok for you? Cheers, Bjoern
http://toolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_numpy_1_7
Regards,
Peter
On Mon, Oct 7, 2013 at 8:52 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Hi Dave, Bjeorn,
Right now NumPy isn't installing at all on the Test Tool Shed.
Dave - is is possible that the system was updated and the Fortran compiler removed? e.g.
...
Since this seems to be breaking lapack, perhaps we should further simplify package_numpy_1_7 not to use lapack or ATLAS? What do you think Bjoern?
I'm a little bit sceptic to remove features only because the Test Utility can not handle it. On the other hand I understand that numpy should just work and that your test failing since month because of my dependencies. I will get rid of it tomorrow. As a result we can setup_virtualenv and install numpy from pip directly I think.
Is that ok for you? Cheers, Bjoern
Hi Bjoern, I think the real problem here is we don't yet have a clear base installation: https://trello.com/c/gYlRdqwr/774-toolshed-define-a-base-set-of-utilities-th... https://trello.com/c/7VTlX9rD/844-toolshed-tool-dependency-installation-requ... What is worse is the system used by the ToolShed for running the nightly tests appears to change without notice, e.g. here it looks like something with the Fortran compiler has changed, for the BLAST+ wrappers the default shell seems to have changed between bash and sh). I think the ToolShed system should be a role model of the agreed standard set of system level dependencies. Given that, it seems your lapack package has some dependencies which are not necessarily present, and this failure is cascading up to break NumPy and then Biopython, and thus multiple tools on the Tool Shed :( I do think that in the short term the best option is to further simplify package_numpy_1_7 not to use lapack or ATLAS - although this is not ideal :( On the positive side, your hard work setting up packages for NumPy, lapack and Altas is drawing attention to the need for a clearer set of base packages. :) Thanks, Peter
Peter, Björn, I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment. --Dave B. On 10/08/2013 05:25 AM, Peter Cock wrote:
On Mon, Oct 7, 2013 at 8:52 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Hi Dave, Bjeorn,
Right now NumPy isn't installing at all on the Test Tool Shed.
Dave - is is possible that the system was updated and the Fortran compiler removed? e.g.
...
Since this seems to be breaking lapack, perhaps we should further simplify package_numpy_1_7 not to use lapack or ATLAS? What do you think Bjoern?
I'm a little bit sceptic to remove features only because the Test Utility can not handle it. On the other hand I understand that numpy should just work and that your test failing since month because of my dependencies. I will get rid of it tomorrow. As a result we can setup_virtualenv and install numpy from pip directly I think.
Is that ok for you? Cheers, Bjoern
Hi Bjoern,
I think the real problem here is we don't yet have a clear base installation: https://trello.com/c/gYlRdqwr/774-toolshed-define-a-base-set-of-utilities-th... https://trello.com/c/7VTlX9rD/844-toolshed-tool-dependency-installation-requ...
What is worse is the system used by the ToolShed for running the nightly tests appears to change without notice, e.g. here it looks like something with the Fortran compiler has changed, for the BLAST+ wrappers the default shell seems to have changed between bash and sh).
I think the ToolShed system should be a role model of the agreed standard set of system level dependencies.
Given that, it seems your lapack package has some dependencies which are not necessarily present, and this failure is cascading up to break NumPy and then Biopython, and thus multiple tools on the Tool Shed :(
I do think that in the short term the best option is to further simplify package_numpy_1_7 not to use lapack or ATLAS - although this is not ideal :(
On the positive side, your hard work setting up packages for NumPy, lapack and Altas is drawing attention to the need for a clearer set of base packages. :)
Thanks,
Peter
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :) Bjoern - it looks like we can leave the NumPy definition as is... Regards, Peter
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :)
Bjoern - it looks like we can leave the NumPy definition as is...
Ah sorry, totally forgot about it :(
Regards,
Peter
On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :)
Bjoern - it looks like we can leave the NumPy definition as is...
Ah sorry, totally forgot about it :(
Progress, Fortran is working and lapack seems to have installed, but: e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org TypeNameVersion numpy package 1.7.1 Error Running from numpy source directory. /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) error: Command "/usr/bin/gfortran -Wall -Wall -shared build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o -L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib -Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so" failed with exit status 1 Most of that is warnings about not using ATLAS (which we expect) but I'm not quite sure why it fails... Peter
On Thu, Oct 10, 2013 at 10:00 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :)
Bjoern - it looks like we can leave the NumPy definition as is...
Ah sorry, totally forgot about it :(
Progress, Fortran is working and lapack seems to have installed, but: e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org TypeNameVersion numpy package 1.7.1 Error Running from numpy source directory. /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) error: Command "/usr/bin/gfortran -Wall -Wall -shared build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o -L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib -Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so" failed with exit status 1
Most of that is warnings about not using ATLAS (which we expect) but I'm not quite sure why it fails...
Hi Guys, Bjoern - good news for you, the installation of NumPy can work on the current Test Tool Shed. Dave - bad news for you, last night I had 3 NumPy install failures (all different) and 2 successes - which to me suggests a possible race condition in the nightly testing? (1) Last night on the Test Tool Shed this failed as in email above: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14 (2) This failed with the simpler message: http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org (3) Here the installation seems to have failed silently, leading to a test failure at the import stage: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0 Tests that failed Tool id: seq_select_by_id Tool version: seq_select_by_id Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6) Stderr: Fatal error: Exit code 1 () Biopython 1.54 or later is required (4, 5) However these also depend on Biopython 1.62 and worked: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce (failed last night) http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/8a34c565a473 My guess is a race condition where multiple attempts are being made in parallel to install NumPy (in the same place), and this causes some of these attempts to fail. But that is without looking at the code at all ;) Regards, Peter
Peter, I've made some changes to the way the test VM handles tool dependencies, which should reduce the frequency of this type of error, and will be running further tests today to see how I can improve the framework's handling of these situations. --Dave B. On 10/11/2013 04:51 AM, Peter Cock wrote:
On Thu, Oct 10, 2013 at 10:00 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :)
Bjoern - it looks like we can leave the NumPy definition as is...
Ah sorry, totally forgot about it :(
Progress, Fortran is working and lapack seems to have installed, but: e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org TypeNameVersion numpy package 1.7.1 Error Running from numpy source directory. /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) error: Command "/usr/bin/gfortran -Wall -Wall -shared build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o -L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib -Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so" failed with exit status 1
Most of that is warnings about not using ATLAS (which we expect) but I'm not quite sure why it fails...
Hi Guys,
Bjoern - good news for you, the installation of NumPy can work on the current Test Tool Shed.
Dave - bad news for you, last night I had 3 NumPy install failures (all different) and 2 successes - which to me suggests a possible race condition in the nightly testing?
(1) Last night on the Test Tool Shed this failed as in email above:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14
(2) This failed with the simpler message:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org
(3) Here the installation seems to have failed silently, leading to a test failure at the import stage: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0
Tests that failed Tool id: seq_select_by_id Tool version: seq_select_by_id Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6) Stderr: Fatal error: Exit code 1 () Biopython 1.54 or later is required
(4, 5) However these also depend on Biopython 1.62 and worked:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce (failed last night)
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/8a34c565a473
My guess is a race condition where multiple attempts are being made in parallel to install NumPy (in the same place), and this causes some of these attempts to fail. But that is without looking at the code at all ;)
Regards,
Peter
Björn, Peter, In the course of my investigation, I've discovered a possible improvement to the lapack build recipe. If you change the cmake command from: cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR/lapack to: cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR/lapack -DCMAKE_Fortran_FLAGS='-O2 -fPIC' the need for the earlier sed command is eliminated, which may reduce the likelihood of environment-specific failures. Specifically, I attempted to install get_orfs_or_cdss on a local (linux) system, and the numpy installation's error message informed me that lapack was built without the -fPIC flag. --Dave B. On 10/11/2013 04:51 AM, Peter Cock wrote:
On Thu, Oct 10, 2013 at 10:00 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :)
Bjoern - it looks like we can leave the NumPy definition as is...
Ah sorry, totally forgot about it :(
Progress, Fortran is working and lapack seems to have installed, but: e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org TypeNameVersion numpy package 1.7.1 Error Running from numpy source directory. /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) error: Command "/usr/bin/gfortran -Wall -Wall -shared build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o -L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib -Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so" failed with exit status 1
Most of that is warnings about not using ATLAS (which we expect) but I'm not quite sure why it fails...
Hi Guys,
Bjoern - good news for you, the installation of NumPy can work on the current Test Tool Shed.
Dave - bad news for you, last night I had 3 NumPy install failures (all different) and 2 successes - which to me suggests a possible race condition in the nightly testing?
(1) Last night on the Test Tool Shed this failed as in email above:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14
(2) This failed with the simpler message:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org
(3) Here the installation seems to have failed silently, leading to a test failure at the import stage: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0
Tests that failed Tool id: seq_select_by_id Tool version: seq_select_by_id Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6) Stderr: Fatal error: Exit code 1 () Biopython 1.54 or later is required
(4, 5) However these also depend on Biopython 1.62 and worked:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce (failed last night)
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/8a34c565a473
My guess is a race condition where multiple attempts are being made in parallel to install NumPy (in the same place), and this causes some of these attempts to fail. But that is without looking at the code at all ;)
Regards,
Peter
Hi Dave, thanks for the pointer. I updated the testtoolshed package. For me installing fine. I will wait one more day and have a look at Peter's Testcases. I will than migrate it to the main toolshed. Changes are now also in github. Thanks, Björn
Björn, Peter,
In the course of my investigation, I've discovered a possible improvement to the lapack build recipe. If you change the cmake command from:
cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR/lapack
to:
cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR/lapack -DCMAKE_Fortran_FLAGS='-O2 -fPIC'
the need for the earlier sed command is eliminated, which may reduce the likelihood of environment-specific failures. Specifically, I attempted to install get_orfs_or_cdss on a local (linux) system, and the numpy installation's error message informed me that lapack was built without the -fPIC flag.
--Dave B.
On 10/11/2013 04:51 AM, Peter Cock wrote:
On Thu, Oct 10, 2013 at 10:00 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
On Wed, Oct 9, 2013 at 10:01 PM, Björn Grüning <bjoern.gruening@pharmazie.uni-freiburg.de> wrote:
Am Mittwoch, den 09.10.2013, 19:15 +0100 schrieb Peter Cock:
On Tue, Oct 8, 2013 at 9:57 PM, Dave Bouvier <dave@bx.psu.edu> wrote:
Peter, Björn,
I have restored gfortran to the repository test host. I believe it was inadvertently removed, possibly as a side effect of updating packages or removing packages that were causing conflicts with the Galaxy eggs. Numpy and biopython should now install successfully, as they do in my local environment.
--Dave B.
Thanks Dave, fingers crossed for tonight's test run :)
Bjoern - it looks like we can leave the NumPy definition as is...
Ah sorry, totally forgot about it :(
Progress, Fortran is working and lapack seems to have installed, but: e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org TypeNameVersion numpy package 1.7.1 Error Running from numpy source directory. /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1494: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmp3kNwuR/tmp-toolshed-mtdGJ0d5H/numpy-1.7.1/numpy/distutils/system_info.py:1408: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) error: Command "/usr/bin/gfortran -Wall -Wall -shared build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o -L/ToolDependencies/lapack/3.4.2/iuc/package_lapack_3_4/60957bd68fe2/lapack/lib -Lbuild/temp.linux-x86_64-2.7 -llapack -lblas -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so" failed with exit status 1
Most of that is warnings about not using ATLAS (which we expect) but I'm not quite sure why it fails...
Hi Guys,
Bjoern - good news for you, the installation of NumPy can work on the current Test Tool Shed.
Dave - bad news for you, last night I had 3 NumPy install failures (all different) and 2 successes - which to me suggests a possible race condition in the nightly testing?
(1) Last night on the Test Tool Shed this failed as in email above:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip/834cf6618a14
(2) This failed with the simpler message:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss/308ec20faefb
Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion biopython package 1.62 Error running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org
(3) Here the installation seems to have failed silently, leading to a test failure at the import stage: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0
Tests that failed Tool id: seq_select_by_id Tool version: seq_select_by_id Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6) Stderr: Fatal error: Exit code 1 () Biopython 1.54 or later is required
(4, 5) However these also depend on Biopython 1.62 and worked:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename/d08767ae73ce (failed last night)
http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id/8a34c565a473
My guess is a race condition where multiple attempts are being made in parallel to install NumPy (in the same place), and this causes some of these attempts to fail. But that is without looking at the code at all ;)
Regards,
Peter
Hi Dave, The nightly tests for my tools using Biopython are mostly working, e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename However, I had an odd failure last night: http://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id/9e9e3b860aa0 Automated tool test results Tool test results Automated test environment Time tested: ~ 11 hours ago System: Linux 3.0.0-1-amd64 Architecture: x86_64 Python version: 2.7.2+ Galaxy revision: 11038:315018626d87 Galaxy database version: 117 Tool shed revision: 11020:95815478f355 Tool shed database version: 21 Tool shed mercurial version: 2.2.3 Tests that failed Tool id: seq_select_by_id Tool version: seq_select_by_id Test: test_tool_000000 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/seq_select_by_id/seq_select_by_id/0.0.6) Stderr: Fatal error: Exit code 1 () Biopython 1.54 or later is required Traceback: Traceback (most recent call last): File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", line 102, in do_it self.verify_dataset_correctness( outfile, hid=elem_hid, maxseconds=testdef.maxseconds, attributes=attributes, shed_tool_id=shed_tool_id ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 782, in verify_dataset_correctness self._assert_dataset_state( elem, 'ok' ) File "/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/base/twilltestcase.py", line 606, in _assert_dataset_state raise AssertionError( errmsg ) AssertionError: Expecting dataset state 'ok', but state is 'error'. Dataset blurb: error There are no installation problems shown, yet the Biopython import line inside seq_select_by_id.py is failing. Could you send me the logs from this failure? Perhaps we have a silent installation failure? Thanks, Peter
participants (4)
-
Bjoern Gruening -
Björn Grüning -
Dave Bouvier -
Peter Cock