Dear list,
I am running galaxy-dist on Amazon EC2 through Cloudman, and am using the enable_tasked_jobs to run jobs in parallel. Yes, I know it's not recommended in production. My jobs usually get split in 72 parts, and sometimes (but not always, maybe in 30-50% of cases), errors are returned concerning the python egg cache, usually:
[Errno 17] File exists: '/home/galaxy/.python-eggs'
or something like
[Errno 17] File exists: '/home/galaxy/.python-eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg-tmp'
The errors arise AFAIK from when scripts/extract_dataset_part.py is run. I am guessing that the tmp python egg dir is created for every task of the mentioned 72, that they sometimes coincide and that this leads to an error.
I would like to solve this problem, but before doing so, I'd like to know if someone else has already fixed it in a galaxy-central changeset.
cheers, jorrit