On Mon, Jan 25, 2016 at 5:50 PM, John Chilton <jmchilton@gmail.com> wrote:
On Mon, Jan 25, 2016 at 3:44 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Thanks John,
On Mon, Jan 25, 2016 at 3:29 PM, John Chilton <jmchilton@gmail.com> wrote:
The script generated to call Galaxy is here:
https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/metada...
The job template stuff that setups of the environment the job runs in is here:
https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/uti...
This second file changes in a large way with 16.01 which ditches eggs for virtual environments and wheels.
We were trying with v15.10 (I think), but since its late January 2016, can we expect a v16.01 release shortly? That might be quite timely as we're not going to be working on our new VM server and its cluster integration till next week...
If I was doing something new and doing configuration testing - I would target 16.01 - the release will probably happen this week - it has been running on main for some time now. More than other recent releases I would target this one a little ahead of time because it makes some biggish changes to how Galaxy is configured - uwsgi needs to be tweaked, we are swapping from eggs to wheels, there are some changes to job script generation to isolate tool dependency evaluation. More than other recent upgrade processes (15.04 -> 15.07 and 15.07 -> 15.10) I think there is the potential for little hiccups.
Useful food for thought - thank you. We'll give that a go in Feb 2016, hopefully early next week :)
We don't explicitly support running different versions of Python on the worker and handler it seems - but I have seen it work before. You could probably hack up DEFAULT_JOB_FILE_TEMPLATE.sh to point at a different instance of Galaxy for your version. I'd hope there was something easier though.
So there should probably be a note on the cluster wiki page about recommending having the same version of Python on the cluster nodes and Galaxy sever?
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
If you can find a clear place to add that note, I'd go for it ;).
Done: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?action=diff&rev1=52&rev2=53
In general though it is a broader problem - all tool shed installations happen on the web servers - not on the cluster workers. I know many institutions run different OSes between those machines - but if you think about it the tool shed doesn't really support it - it just sort of happens that usually there isn't problems.
Good point - that's quite a minefield with consistent versions of Python on the Galaxy server and cluster just the tip of the iceberg. That does strongly suggest sticking with the same version of Linux on both if possible... perhaps another note on the wiki? Thanks, Peter