On Mon, Feb 22, 2016 at 5:54 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Hi all,

Last week at the Galaxy Admins video hangout Nate gave an
overview of the way Python dependency handling is changing
from using egg files to using wheel files instead - slide links at:

https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18

I understand the Galaxy v16.01 (2016 January feature freeze)
release which switches from eggs to wheels is due out shortly -
hopefully this week? 

Recently we've been trying to setup a replacement Galaxy instance
where jobs are submitted to our SGE cluster as the linux user
requesting the job. This has been a bumpy road, e.g.

http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-after-jobs-complete-tt4668736.html

Right now we've run into a problem with the metadata scripts
trying to access ~/.python-eggs and going back over the archives
it looks like we may need to set $PYTHON_EGG_CACHE to a
temp folder on a per-node per-user basis, see e.g.

http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.html#a4656394

Sadly the wiki does not cover this aspect of running jobs as the
real user (but the egg settings are about to be obsolete):

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User

Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE
will go away. What happens with wheels, Python virtual environments,
and running jobs as the real user?

Hi Peter,

Unlike eggs, which can be installed in their package format (as a single .egg file) and used in place, wheels are unpacked and their contents reside in the virtualenv's site-packages just as if you had `python setup.py install`ed them, there's no cache directory. The "real user" should be able to use Galaxy's virtualenv as long as the appropriate read/execute permissions are set on a common group or other.

Galaxy's virtualenv is passed down to the jobs when possible:


If you need to create a separate virtualenv for jobs to use, you can do this with an <env> tag on the destination:

  https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#galaxy-job-handlers

There are instructions in that document on how to create your own Galaxy virtualenv:

  https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#managing-dependencies-manually

I have not used the "real user" stuff in a very long time so I haven't tested this, but I can't think of any major roadblocks that should prevent it from working.

--nate
 

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/