"default_cluster_job_runner" setting changed - no longer getting job output from cluster - galaxy-dev

14 Dec 2012

      Hi guys,

I'm really stuck on this one.

In my galaxy config file I changed the setting it uses to launch jobs from:

default_cluster_job_runner = drmaa://-V/
(which worked fine)

To
default_cluster_job_runner = drmaa://-V -pe batch 8/

Now whenever I run a job I see a message like this:

"Job output not returned from cluster"

(Is there any other information I can use to debug this?)

I see the information below in the log file if any of that is useful:

galaxy.jobs.manager DEBUG 2012-12-14 13:27:36,961 (104) Job assigned
to handler 'main'
galaxy.jobs DEBUG 2012-12-14 13:27:37,584 (104) Working directory for
job is: /home/galaxy/job_working_directory/000/104
galaxy.jobs.handler DEBUG 2012-12-14 13:27:37,586 dispatching job 104
to drmaa runner
galaxy.jobs.handler INFO 2012-12-14 13:27:37,772 (104) Job dispatched
galaxy.jobs.runners DEBUG 2012-12-14 13:27:38,078 Copying
/home/galaxy/job_working_directory/000/104/global_model.txt to
/home/galaxy/000/dataset_155.dat as directed by from_work_dir
galaxy.jobs.runners DEBUG 2012-12-14 13:27:38,111 Copying
/home/galaxy/job_working_directory/000/104/genes.fpkm_tracking to
/home/galaxy/000/dataset_152.dat as directed by from_work_dir
galaxy.jobs.runners DEBUG 2012-12-14 13:27:38,135 Copying
/home/galaxy/job_working_directory/000/104/isoforms.fpkm_tracking to
/home/galaxy/000/dataset_153.dat as directed by from_work_dir
galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:38,819 (104)
submitting file /home/galaxy/pbs/galaxy_104.sh
galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:38,820 (104) command
is: python /misc/local/galaxy/galaxy-dist/tools/ngs_rna/cufflinks_wrapper.py
             --input=/home/galaxy/000/dataset_93.dat
--assembled-isoforms-output=/home/galaxy/000/dataset_154.dat
  --num-threads="4"             -I 300000             -F 0.1
  -j 0.15; cp /home/galaxy/job_working_directory/000/104/global_model.txt
/home/galaxy/000/dataset_155.dat; cp
/home/galaxy/job_working_directory/000/104/genes.fpkm_tracking
/home/galaxy/000/dataset_152.dat; cp
/home/galaxy/job_working_directory/000/104/isoforms.fpkm_tracking
/home/galaxy/000/dataset_153.dat; cd /misc/local/galaxy/galaxy-dist;
/misc/local/galaxy/galaxy-dist/set_metadata.sh /home/galaxy
/home/galaxy/job_working_directory/000/104 .
/misc/local/galaxy/galaxy-dist/universe_wsgi.ini
/home/galaxy/job_working_directory/tmppjMj2j
/home/galaxy/job_working_directory/000/104/galaxy.json
/home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_176_zK8nkf,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_176_Dkq7kA,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_176_jR0e_Z,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_176_jl8W2w,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_176_M6ctS5
/home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_177_Q1vGSc,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_177_MG96_w,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_177_NWTSVQ,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_177_iTNC5T,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_177_MO0SIc
/home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_174_HcoiD4,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_174_4j5SRC,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_174_irlquD,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_174_jfK1zk,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_174_YQzY4e
/home/galaxy/job_working_directory/000/104/metadata_in_HistoryDatasetAssociation_175_rjQ2Gp,/home/galaxy/job_working_directory/000/104/metadata_kwds_HistoryDatasetAssociation_175_HMeDGS,/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_175_7sjyvl,/home/galaxy/job_working_directory/000/104/metadata_results_HistoryDatasetAssociation_175_cJ_Sw6,,/home/galaxy/job_working_directory/000/104/metadata_override_HistoryDatasetAssociation_175_JY4ngx
galaxy.jobs.runners.drmaa INFO 2012-12-14 13:27:38,884 (104) queued as 4076946
galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:39,822 (104/4076946)
state change: job is queued and active
galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:42,830 (104/4076946)
state change: job is running
galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:44,065 (104/4076946)
state change: job finished normally
galaxy.jobs.runners.drmaa DEBUG 2012-12-14 13:27:44,111 Job output not
returned from cluster
galaxy.jobs DEBUG 2012-12-14 13:27:44,147 Tool did not define exit
code or stdio handling; checking stderr for success
galaxy.jobs DEBUG 2012-12-14 13:27:44,187 setting dataset state to ERROR
galaxy.jobs DEBUG 2012-12-14 13:27:44,206 setting dataset state to ERROR
galaxy.jobs DEBUG 2012-12-14 13:27:44,226 setting dataset state to ERROR
galaxy.jobs DEBUG 2012-12-14 13:27:44,250 setting dataset state to ERROR
galaxy.jobs DEBUG 2012-12-14 13:27:44,432 job 104 ended
galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,432 Cleaning up
external metadata files
galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,452 Failed to
cleanup MetadataTempFile temp files from
/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_176_jR0e_Z:
No JSON object could be decoded: line 1 column 0 (char 0)
galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,475 Failed to
cleanup MetadataTempFile temp files from
/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_177_NWTSVQ:
No JSON object could be decoded: line 1 column 0 (char 0)
galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,491 Failed to
cleanup MetadataTempFile temp files from
/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_175_7sjyvl:
No JSON object could be decoded: line 1 column 0 (char 0)
galaxy.datatypes.metadata DEBUG 2012-12-14 13:27:44,510 Failed to
cleanup MetadataTempFile temp files from
/home/galaxy/job_working_directory/000/104/metadata_out_HistoryDatasetAssociation_174_irlquD:
No JSON object could be decoded: line 1 column 0 (char 0)
galaxy.objectstore CRITICAL 2012-12-14 13:27:44,544
/home/galaxy/job_working_directory/000/104 delete error [Errno 39]
Directory not empty: '/home/galaxy/job_working_directory/000/104'

Thanks,

-Greg

"default_cluster_job_runner" setting changed - no longer getting job output from cluster

greg

greg

tags

participants (1)