On Mon, Feb 22, 2016 at 5:54 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:

Hi all,

Last week at the Galaxy Admins video hangout Nate gave an overview of the way Python dependency handling is changing from using egg files to using wheel files instead - slide links at:

https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18

I understand the Galaxy v16.01 (2016 January feature freeze) release which switches from eggs to wheels is due out shortly - hopefully this week?

Recently we've been trying to setup a replacement Galaxy instance where jobs are submitted to our SGE cluster as the linux user requesting the job. This has been a bumpy road, e.g.

http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-aft...

Right now we've run into a problem with the metadata scripts trying to access ~/.python-eggs and going back over the archives it looks like we may need to set $PYTHON_EGG_CACHE to a temp folder on a per-node per-user basis, see e.g.

http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.ht...

Sadly the wiki does not cover this aspect of running jobs as the real user (but the egg settings are about to be obsolete):

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_J...

Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE will go away. What happens with wheels, Python virtual environments, and running jobs as the real user?

Hi Peter, Unlike eggs, which can be installed in their package format (as a single .egg file) and used in place, wheels are unpacked and their contents reside in the virtualenv's site-packages just as if you had `python setup.py install`ed them, there's no cache directory. The "real user" should be able to use Galaxy's virtualenv as long as the appropriate read/execute permissions are set on a common group or other. Galaxy's virtualenv is passed down to the jobs when possible: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/uti... If you need to create a separate virtualenv for jobs to use, you can do this with an <env> tag on the destination: https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies... There are instructions in that document on how to create your own Galaxy virtualenv: https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies... I have not used the "real user" stuff in a very long time so I haven't tested this, but I can't think of any major roadblocks that should prevent it from working. --nate

Thanks,

Peter ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/