Hi Galaxy-devs,
I recently ran into odd behavior when jobs were submitted at the same time to our cluster (jobs wouldn't return correctly, e.g. ). I changed "cleanup_job" in galaxy.ini to "never" to see what was going on in my job_working_directory (/scratch/galaxy/job_working/).
It looks like subdirectories are no longer being created in my parent job_working_directory (e.g. the tool_script.sh along with the outputs are always placed and executed in /scratch/galaxy/job_working). Isn't there usually a bunch of subdirectories created (e.g. 000/XXX, etc...)
I suspect my issues arise because script execution is happening in the parent directory rather than in temporary ones that get created within the parent directory.
Could this be the issue and if so does anyone know how this could have happened or how I can fix this?
I've also noticed that my environment variables are pointing to the parent directory rather than a subdirectory:
SGE_CWD_PATH=/scratch/galaxy/job_working
As an example, the contents of /scratch/galaxy/job_working look like this (note that there are no usual subdirectories like 000/XXX, etc...):
[galaxy@galaxy job_working]$ ls | more __instrument_core_epoch_end __instrument_core_epoch_start __instrument_core_galaxy_slots conda-env conda-metadata-env galaxy_20771.e galaxy_20771.ec galaxy_20771.o galaxy_20771.sh galaxy_20772.e galaxy_20772.ec galaxy_20772.o galaxy_20772.sh metadata_in_HistoryDatasetAssociation_46704_NHpqc0 metadata_in_HistoryDatasetAssociation_46705_nUT573 metadata_kwds_HistoryDatasetAssociation_46704_5IADnM metadata_kwds_HistoryDatasetAssociation_46705__z5cTO metadata_out_HistoryDatasetAssociation_46704_G8phbW metadata_out_HistoryDatasetAssociation_46705_qTOxoE metadata_override_HistoryDatasetAssociation_46704_v0f_Wl metadata_override_HistoryDatasetAssociation_46705_8pzLX_ metadata_results_HistoryDatasetAssociation_46704_fSFQsP metadata_results_HistoryDatasetAssociation_46705_r4vUOP set_metadata_2A6wuD.py set_metadata_2Dasxn.py tool_script.sh working
Any help would be greatly appreciated!!