Hi all,

I’ve configured one of our tools to submit jobs to our condor cluster.  I can see the job is routed to the condor runner:

==> handler4.log <==

galaxy.jobs.handler DEBUG 2015-01-30 09:14:58,092 (508) Dispatching to condor runner

galaxy.jobs DEBUG 2015-01-30 09:14:58,204 (508) Persisting job destination (destination id: condor)

I can see that indeed the job is submitted to the condor cluster:

[root@galaxy galaxy-dist]# condor_q

-- Submitter: galaxy.local : <10.177.61.90:55265> : galaxy.local

ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

  21.0   galaxy          1/30 09:15   0+00:00:02 R  0   0.0  galaxy_509.sh

The job begins to run:

==> handler4.log <==

galaxy.jobs.runners.condor DEBUG 2015-01-30 09:15:03,827 (508/20) job is now running

galaxy.jobs.runners.condor DEBUG 2015-01-30 09:15:05,183 (508/20) job has completed

Galaxy is almost immediately removing the job working directory:

Here is a snippet of the errors:

==> handler4.log <==

galaxy.jobs.runners DEBUG 2015-01-30 09:15:06,372 (508/20) Unable to cleanup /panfs/storage.local/opt/galaxy-dist/database/pbs/galaxy_508.ec: [Errno 2] No such file or directory: '/panfs/storage.local/opt/galaxy-dist/database/pbs/galaxy_508.ec'

galaxy.jobs DEBUG 2015-01-30 09:15:06,816 setting dataset state to ERROR

galaxy.datatypes.metadata DEBUG 2015-01-30 09:15:06,996 Failed to cleanup MetadataTempFile temp files from /panfs/storage.local/galaxy-data/job_working_directory/000/508/metadata_out_HistoryDatasetAssociation_717_zrzVqh: No JSON object could be decoded

Is it possible galaxy is attempting to query condor and see if the job is running, not finding anything and deciding that the job is not running and bailing out?

I’ve reconstructed the process step by step using the logs but I have not been able to see exactly where the condor_submit command is shown so I can try to submit the same job manually.

Does anyone have a suggestion for debugging this?

Thanks,

Don

Florida State University

Research Computing Center