On Wed, Feb 19, 2014 at 9:33 PM, Greg Von Kuster <greg@bx.psu.edu> wrote:
Hi Peter,
Dave B and I just discovered that issue that causes your tests to fail.
The problem lies with our current implementation supporting the
<move_directory_files> action tag, which is used in your recipe.
This tag ultimately uses Python's os.listdir via shutil.move.
It turns out that certain Python versions produce different results
from os.listdir - the order is different. This is a problem with MIRA
because during the instlllation it is moving a directory of files where
the directory includes symlinks to other files in the directory. When
the symlinks are moved after the files to which they link, problems
occur. Again, this behavior is intermittent depending on the Pyhton
version ( and perhaps the os ).
In any case, we have a fix for out move_directory_files which
Dave is now committing. So your tests should pass with tonight's
run - Let's hope!
Thanks Peter!
Well done Greg & Dave,
That explanation makes perfect sense to me - but I can
see it took some digging to solve it!
There is actually one MIRA binary (mira) and several aliases
(e.g. mirabait) which are really symlinks back to mira. Magic.
My next query would be: why wasn't the failing action
<move_directory_files> being caught as a failed install?
Your installation recipe for mira attempts to download a binary and if that fails, it echoes an error, but still performs <set_environment> actions..
The tool dependency installation process will fall back to attempting the installation using the action tags to handle install from source and compile if downloading a pre-compiled binary fails. However, your recipe does not have include installing from source, so the installation process simply assumed your <set_environment> tags were all that was necessary. If the recipe for installing from source is not too complex, maybe you could add it to your recipe. It's probabluy not critical though since we now have the fix to the framework.
Here is the entire log of the installtion process that helped us uncover the problem - notice that upon failure of the initial binary installation, the process "proceeds with install and compile recipe for tool dependency MIRA".
tool_shed.galaxy_install.tool_dependencies.fabric_util DEBUG 2014-02-19 04:50:42,285 Successfully downloaded from url: https://downloads.sourceforge.net/project/mira-assembler/MIRA/stable/mira_4.0_linux-gnu_x86_64_static.tar.bz2
tool_shed.galaxy_install.tool_dependencies.install_util ERROR 2014-02-19 04:50:42,318 Error installing tool dependency MIRA version 4.0.
Traceback (most recent call last):
File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py", line 289, in install_and_build_package_via_fabric
tool_dependency = fabric_util.install_and_build_package( app, tool_dependency, actions_dict )
File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py", line 581, in install_and_build_package
destination_dir=os.path.join( action_dict[ 'destination_directory' ] ) )
File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py", line 309, in move_directory_files
shutil.move( source_file, destination_file )
File "/usr/lib/python2.7/shutil.py", line 301, in move
copy2(src, real_dst)
File "/usr/lib/python2.7/shutil.py", line 130, in copy2
copyfile(src, dst)
File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpdr4lDp/tmp-toolshed-mtdV8W9PM/mira_4.0_linux-gnu_x86_64_static/bin/mirabait'
tool_shed.galaxy_install.tool_dependencies.install_util DEBUG 2014-02-19 04:50:42,541 Error downloading binary for tool dependency MIRA version 4.0: File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/install_util.py", line 289, in install_and_build_package_via_fabric
tool_dependency = fabric_util.install_and_build_package( app, tool_dependency, actions_dict )
File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py", line 581, in install_and_build_package
destination_dir=os.path.join( action_dict[ 'destination_directory' ] ) )
File "/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py", line 309, in move_directory_files
shutil.move( source_file, destination_file )
File "/usr/lib/python2.7/shutil.py", line 301, in move
copy2(src, real_dst)
File "/usr/lib/python2.7/shutil.py", line 130, in copy2
copyfile(src, dst)
File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
[Errno 2] No such file or directory: '/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpdr4lDp/tmp-toolshed-mtdV8W9PM/mira_4.0_linux-gnu_x86_64_static/bin/mirabait'
tool_shed.galaxy_install.tool_dependencies.install_util DEBUG 2014-02-19 04:50:42,541 Proceeding with install and compile recipe for tool dependency MIRA.
tool_shed.util.tool_dependency_util DEBUG 2014-02-19 04:50:42,549 Removed tool dependency installation directory: /ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40
/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py:173: DeprecationWarning: With-statements now directly support multiple context managers
with settings( warn_only=True ):
/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py:646: DeprecationWarning: With-statements now directly support multiple context managers
with settings( warn_only=True ):
/var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py:173: DeprecationWarning: With-statements now directly support multiple context managers
with settings( warn_only=True ):
Warning: local() encountered an error (return code 2) while executing 'echo Your machine details (the output from 'uname' and 'arch'):'
tool_shed.galaxy_install.tool_dependencies.install_util DEBUG 2014-02-19 04:50:43,683 Proceeding with install and compile recipe for tool dependency MIRA.
tool_shed.util.tool_dependency_util DEBUG 2014-02-19 04:50:43,690 Removed tool dependency installation directory: /ToolDepsTest/MIRA/4.0/peterjc/mira4_assembler/133b863a8a40
tool_shed.util.tool_dependency_util DEBUG 2014-02-19 04:50:43,698 Changing status for tool dependency MIRA from Uninstalled to Installed.
Thanks,
Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/