I think I've figured this out in the end, posting my solution in case anyone else comes across it in future.

The issue was due to a change in the way that Python setuptools adds its installed modules to the search path (sys.path) via easy-install.pth files. Later versions of setuptools (>=25.0.0) append to rather than rewrite sys.path, so modules installed as eggs don't take precedence over older versions in the Galaxy core (for example "galaxy-ops" and "galaxy_sequence_utils").

One fix is to set the SETUPTOOLS_SYS_PATH_TECHNIQUE environment variable to "rewrite" and reinstall the modules (to update the module's "easy-install.pth" file). This should work for all recent versions of setuptools (and fixed the problems below for me), but might not work for future versions of setuptools - see http://setuptools.readthedocs.io/en/latest/history.html#v25-0-0.

Alternatively: the "galaxy_sequence_utils" package (used by "fastq_paired_end_joiner", maybe others?) is available in bioconda, so using that is probably a better long-term fix (not tested yet though). But I don't think "galaxy-ops" (used by "join" and other tools) is there yet.

Best wishes

Peter

--
Peter Briggs peter.briggs@manchester.ac.uk
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482


________________________________________
From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Peter Briggs [peter.briggs@manchester.ac.uk]
Sent: Wednesday, December 14, 2016 4:13 PM
To: Galaxy Dev List
Subject: [galaxy-dev] Tools pick up Python modules from Galaxy core rather than installed tool dependencies

Dear devs

I was wondering if anyone had any suggestions as to the cause of a
peculiar error I'm seeing when running the latest versions of the
devteam "join" and "fastq_paired_end_joiner".

The tools appear to be picking up Python modules from the Galaxy
library, rather than from the installed dependencies. The PYTHONPATH
appears to be correct (i.e. tool dependency appears before Galaxy
library path) but sys.path shows the dependency's Python egg appearing
after the Galaxy library path.

E.g. for the paired-end joiner:

PYTHONPATH=/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python:/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib

sys.path = ['',
'/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python',
'/mnt/rvmi/centaurus/galaxy/devel/galaxy-dist/lib',
'/mnt/rvmi/centaurus/galaxy/devel/tool_dependencies/galaxy_sequence_utils/1.0.1/iuc/package_galaxy_sequence_utils_1_0_1/c1ab450748ba/lib/python/galaxy_sequence_utils-1.0.1-py2.7.egg',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python27.zip',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/plat-linux2',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-tk',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-old',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/lib-dynload',
'/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7',
'/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/plat-linux2',
'/mnt/rvmi/centaurus/galaxy/python/2.7.10/lib/python2.7/lib-tk',
'/home/centaurus-galaxy/fastq_paired_end_joiner_problem/venv/lib/python2.7/site-packages']

Error:

Traceback (most recent call last):
   File
"/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py",
line 158, in <module>
     main()
   File
"/mnt/rvmi/centaurus/galaxy/devel/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_joiner/6a7f5da7c76d/fastq_paired_end_joiner/fastq_paired_end_joiner.py",
line 136, in main
     joiner = fq.fastqJoiner( input1_type, paste=paste )
TypeError: __init__() got an unexpected keyword argument 'paste'

I know it's a long shot but can anyone suggest where to start looking at
where this is going wrong?

Thanks,

Peter

Ps I'm using Galaxy v16.07 and the problem occurs on our production
platform but not in a local Galaxy install on my desktop.

--
Peter Briggs peter.briggs@manchester.ac.uk
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/