I have 2 job handlers...
It seems that when I restart them only a single job is queued, it has an error and then sticks on "Cleaning up external metadata files” forever.
I have 700+ jobs in the queue and would like to get them running asap...
here’s the log from the other.
Starting server in PID 4873.
galaxy.jobs DEBUG 2014-08-05 17:34:08,388 (125206) Working directory for job is: /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206
galaxy.jobs.handler DEBUG 2014-08-05 17:34:08,410 (125206) Dispatching to gridengine runner
galaxy.jobs DEBUG 2014-08-05 17:34:08,479 (125206) Persisting job destination (destination id: gridengine)
galaxy.jobs.handler INFO 2014-08-05 17:34:08,498 (125206) Job dispatched
galaxy.tools.deps DEBUG 2014-08-05 17:34:08,862 Building dependency shell command for dependency 'bedtools2'
galaxy.tools.deps WARNING 2014-08-05 17:34:08,865 Failed to resolve dependency on 'bedtools2', ignoring
galaxy.jobs.runners DEBUG 2014-08-05 17:34:08,966 (125206) command is: bedtools --version > /mnt/galaxy/data/galaxy/tmp/GALAXY_VERSION_STRING_125206 2>&1; bedtools intersect -v -a /mnt/galaxy/data/galaxy/user-data/000/170/dataset_170995.dat -b /mnt/galaxy/data/galaxy/user-data/000/169/dataset_169046.dat
> "/mnt/galaxy/data/galaxy/user-data/000/170/dataset_170999.dat"; return_code=$?; cd /mnt/galaxy/data/galaxy/galaxy-dist; /mnt/galaxy/data/galaxy/galaxy-dist/set_metadata.sh /mnt/galaxy/data/galaxy/user-data /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206
. /mnt/galaxy/data/galaxy/galaxy-dist/universe_wsgi.ini /mnt/galaxy/data/galaxy/tmp/tmpLCBPvF /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy.json /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_in_HistoryDatasetAssociation_194826_qIy33T,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_kwds_HistoryDatasetAssociation_194826_SJhToq,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_out_HistoryDatasetAssociation_194826_4Lf956,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_results_HistoryDatasetAssociation_194826_YzEZK8,,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_override_HistoryDatasetAssociation_194826_19nZAm;
sh -c "exit $return_code"
galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:08,994 (125206) submitting file /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy_125206.sh
galaxy.jobs.runners.drmaa INFO 2014-08-05 17:34:09,023 (125206) queued as 1539118
galaxy.jobs DEBUG 2014-08-05 17:34:09,048 (125206) Persisting job destination (destination id: gridengine)
galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:09,451 (125206/1539118) state change: job is queued and active
galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:11,510 (125206/1539118) state change: job is running
galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:31,208 (125206/1539118) state change: job finished normally
galaxy.jobs DEBUG 2014-08-05 17:34:31,440 setting dataset state to ERROR
galaxy.jobs INFO 2014-08-05 17:34:31,652 Collecting job metrics for <galaxy.model.Job object at 0x7f42e030f850>
galaxy.jobs DEBUG 2014-08-05 17:34:31,664 job 125206 ended
galaxy.datatypes.metadata DEBUG 2014-08-05 17:34:31,665 Cleaning up external metadata files
--
Bradley W. Langhorst, Ph.D.
Applications and Product Development Scientist