I’m not aware of any recent changes to the configuration of galaxy…
It seems that jobs are not being distributed at the moment.
I have 2 job handlers...
It seems that when I restart them only a single job is queued, it has an error and then sticks on "Cleaning up external metadata files” forever.
any ideas? I have 700+ jobs in the queue and would like to get them running asap...
one hanlder just stays at Starting server in PID 4868. serving on http://127.0.0.1:8090
here’s the log from the other.
Starting server in PID 4873. serving on http://127.0.0.1:8091 galaxy.jobs DEBUG 2014-08-05 17:34:08,388 (125206) Working directory for job is: /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 galaxy.jobs.handler DEBUG 2014-08-05 17:34:08,410 (125206) Dispatching to gridengine runner galaxy.jobs DEBUG 2014-08-05 17:34:08,479 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.handler INFO 2014-08-05 17:34:08,498 (125206) Job dispatched galaxy.tools.deps DEBUG 2014-08-05 17:34:08,862 Building dependency shell command for dependency 'bedtools2' galaxy.tools.deps WARNING 2014-08-05 17:34:08,865 Failed to resolve dependency on 'bedtools2', ignoring galaxy.jobs.runners DEBUG 2014-08-05 17:34:08,966 (125206) command is: bedtools --version > /mnt/galaxy/data/galaxy/tmp/GALAXY_VERSION_STRING_125206 2>&1; bedtools intersect -v -a /mnt/galaxy/data/galaxy/user-data/000/170/dataset_170995.dat -b /mnt/galaxy/data/galaxy/user-data/000/169/dataset_169046.dat > "/mnt/galaxy/data/galaxy/user-data/000/170/dataset_170999.dat"; return_code=$?; cd /mnt/galaxy/data/galaxy/galaxy-dist; /mnt/galaxy/data/galaxy/galaxy-dist/set_metadata.sh /mnt/galaxy/data/galaxy/user-data /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 . /mnt/galaxy/data/galaxy/galaxy-dist/universe_wsgi.ini /mnt/galaxy/data/galaxy/tmp/tmpLCBPvF /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy.json /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_in_HistoryDatasetAssociation_194826_qIy33T,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_kwds_HistoryDatasetAssociation_194826_SJhToq,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_out_HistoryDatasetAssociation_194826_4Lf956,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_results_HistoryDatasetAssociation_194826_YzEZK8,,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_override_HistoryDatasetAssociation_194826_19nZAm; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:08,994 (125206) submitting file /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy_125206.sh galaxy.jobs.runners.drmaa INFO 2014-08-05 17:34:09,023 (125206) queued as 1539118 galaxy.jobs DEBUG 2014-08-05 17:34:09,048 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:09,451 (125206/1539118) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:11,510 (125206/1539118) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:31,208 (125206/1539118) state change: job finished normally galaxy.jobs DEBUG 2014-08-05 17:34:31,440 setting dataset state to ERROR galaxy.jobs INFO 2014-08-05 17:34:31,652 Collecting job metrics for <galaxy.model.Job object at 0x7f42e030f850> galaxy.jobs DEBUG 2014-08-05 17:34:31,664 job 125206 ended galaxy.datatypes.metadata DEBUG 2014-08-05 17:34:31,665 Cleaning up external metadata files
-- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist