I recently ran into odd behavior when jobs were submitted at the same time to our cluster (jobs wouldn't return correctly, e.g. ). I changed "cleanup_job" in galaxy.ini to "never" to see what was going on in my job_working_directory (/scratch/galaxy/job_working/).

It looks like subdirectories are no longer being created in my parent job_working_directory (e.g. the tool_script.sh along with the outputs are always placed and executed in /scratch/galaxy/job_working). Isn't there usually a bunch of subdirectories created (e.g. 000/XXX, etc...) 

I suspect my issues arise because script execution is happening in the parent directory rather than in temporary ones that get created within the parent directory.

Could this be the issue and if so does anyone know how this could have happened or how I can fix this?

I've also noticed that my environment variables are pointing to the parent directory rather than a subdirectory:

SGE_CWD_PATH=/scratch/galaxy/job_working

As an example, the contents of /scratch/galaxy/job_working look like this (note that there are no usual subdirectories like 000/XXX, etc...):

metadata_in_HistoryDatasetAssociation_46704_NHpqc0

Any help would be greatly appreciated!!