Actually the failure turned out to be from a different change I made. The "default_cluster_job_runner" change seemed to be fine.
(For the curious, the change that caused these errors was that I split the job_environment_setup_file PATH changes into two files and had job_environment_setup_file simply source the two files. I guess Galaxy didn't like that.)
-Greg
On Fri, Dec 14, 2012 at 2:07 PM, greg margeemail@gmail.com wrote:
Hi guys,
I'm really stuck on this one.
In my galaxy config file I changed the setting it uses to launch jobs from:
default_cluster_job_runner = drmaa://-V/ (which worked fine)
To default_cluster_job_runner = drmaa://-V -pe batch 8/
Now whenever I run a job I see a message like this:
"Job output not returned from cluster"
(Is there any other information I can use to debug this?)
I see the information below in the log file if any of that is useful:
galaxy.jobs.manager DEBUG 2012-12-14 13:27:36,961 (104) Job assigned to handler 'main' galaxy.jobs DEBUG 2012-12-14 13:27:37,584 (104) Working directory for job is: /home/galaxy/job_working_directory/000/104 galaxy.jobs.handler DEBUG 2012-12-14 13:27:37,586 dispatching job 104 to drmaa runner galaxy.jobs.handler INFO 2012-12-14 13:27:37,772 (104) Job dispatched galaxy.jobs.runners DEBUG 2012-12-14 13:27:38,078 Copying /home/galaxy/job_working_directory/000/104/global_model.txt to /home/galaxy/000/dataset_155.dat as directed by from_work_dir galaxy.jobs.runners DEBUG 2012-12-14 13:27:38,111 Copying /home/galaxy/job_working_directory/000/104/genes.fpkm_tracking to /home/galaxy/000/dataset_152.dat as directed by from_work_dir galaxy.jobs.runners DEBUG 2012-12-14 13:27:38,135 Copying /home/galaxy/job_working_directory/000/104/isoforms.fpkm_tracking to /home/galaxy/000/dataset_153.dat as directed by from_work_dir galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:38,819 (104) submitting file /home/galaxy/pbs/galaxy_104.sh galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:38,820 (104) command is: python /misc/local/galaxy/galaxy-dist/tools/ngs_rna/cufflinks_wrapper.py --input=/home/galaxy/000/dataset_93.dat --assembled-isoforms-output=/home/galaxy/000/dataset_154.dat --num-threads="4" -I 300000 -F 0.1 -j 0.15; cp /home/galaxy/job_working_directory/000/104/global_model.txt /home/galaxy/000/dataset_155.dat; cp /home/galaxy/job_working_directory/000/104/genes.fpkm_tracking /home/galaxy/000/dataset_152.dat; cp /home/galaxy/job_working_directory/000/104/isoforms.fpkm_tracking /home/galaxy/000/dataset_153.dat; cd /misc/local/galaxy/galaxy-dist; /misc/local/galaxy/galaxy-dist/set_metadata.sh /home/galaxy /home/galaxy/job_working_directory/000/104 . /misc/local/galaxy/galaxy-dist/universe_wsgi.ini /home/galaxy/job_working_directory/tmppjMj2j /home/galaxy/job_working_directory/000/104/galaxy.json /home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_176_zK8nkf,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_176_Dkq7kA,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_176_jR0e_Z,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_176_jl8W2w,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_176_M6ctS5 /home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_177_Q1vGSc,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_177_MG96_w,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_177_NWTSVQ,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_177_iTNC5T,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_177_MO0SIc /home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_174_HcoiD4,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_174_4j5SRC,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_174_irlquD,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_174_jfK1zk,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_174_YQzY4e /home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_175_rjQ2Gp,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_175_HMeDGS,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_175_7sjyvl,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_175_cJ_Sw6,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_175_JY4ngx galaxy.jobs.runners.drmaa INFO 2012-12-14 13:27:38,884 (104) queued as 4076946 galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:39,822 (104/4076946) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:42,830 (104/4076946) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:44,065 (104/4076946) state change: job finished normally galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:44,111 Job output not returned from cluster galaxy.jobs DEBUG 2012-12-14 13:27:44,147 Tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2012-12-14 13:27:44,187 setting dataset state to ERROR galaxy.jobs DEBUG 2012-12-14 13:27:44,206 setting dataset state to ERROR galaxy.jobs DEBUG 2012-12-14 13:27:44,226 setting dataset state to ERROR galaxy.jobs DEBUG 2012-12-14 13:27:44,250 setting dataset state to ERROR galaxy.jobs DEBUG 2012-12-14 13:27:44,432 job 104 ended galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,432 Cleaning up external metadata files galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,452 Failed to cleanup MetadataTempFile temp files from /home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_176_jR0e_Z: No JSON object could be decoded: line 1 column 0 (char 0) galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,475 Failed to cleanup MetadataTempFile temp files from /home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_177_NWTSVQ: No JSON object could be decoded: line 1 column 0 (char 0) galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,491 Failed to cleanup MetadataTempFile temp files from /home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_175_7sjyvl: No JSON object could be decoded: line 1 column 0 (char 0) galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,510 Failed to cleanup MetadataTempFile temp files from /home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_174_irlquD: No JSON object could be decoded: line 1 column 0 (char 0) galaxy.objectstore CRITICAL 2012-12-14 13:27:44,544 /home/galaxy/job_working_directory/000/104 delete error [Errno 39] Directory not empty: '/home/galaxy/job_working_directory/000/104'
Thanks,
-Greg