I solved this problem by deleting the virtualenv in $GALAXY_ROOT and recreating it from one of my cluster worker nodes. System python on the worker nodes is 2.7.15rc1 from the Ubuntu 18.04 repository. System python on the Galaxy server is 2.7.12 from the Ubuntu 16.04 repository. Since the virtual environment used by Galaxy contains symbolic links to system python modules, it seems there is a dependency between the virtual environment and system python? I tried different methods for creating the virtual env but nothing worked except for creating it on one of the cluster nodes. I look forward to testing the virtualenv-in-conda method that Marius PRed in here ( https://github.com/galaxyproject/galaxy/pull/6975) in the future.

One final "gotcha" I discovered - creating the virtualenv with `bash run.sh` does not install `watchdog`, a module which is necessary if watching the tool directories for changes is enabled. In certain cases this can lead to an exception.

Peter

On Tue, 11 Dec 2018 at 17:15, Peter van Heusden pvh@sanbi.ac.za wrote:

Hi there

I'm trying to run Windowmasker in Galaxy and receiving the following error:

"computing the genome length pass 1 pass 2 optimizing the data structure Traceback (most recent call last): File "/tools/software/galaxy/galaxy1/database/jobs/006/6476/set_metadata_bUlLkt.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/tools/software/galaxy/galaxy1/lib/galaxy_ext/metadata/set_metadata.py", line 14, in <module> import logging File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module> import sys, os, time, cStringIO, traceback, warnings, weakref, collections File "/usr/lib/python2.7/weakref.py", line 14, in <module> from _weakref import ( ImportError: cannot import name _remove_dead_weakref"

The usual suggested fix for this is to remove the Galaxy virtual environment (.venv) and recreated it. I have done that but the error persists. Galaxy is running here on Ubuntu 16.04 with Python 2.7.12. Any advice?

Thanks! Peter