I’m not aware of any recent changes to the configuration of galaxy… It seems that jobs are not being distributed at the moment. I have 2 job handlers... It seems that when I restart them only a single job is queued, it has an error and then sticks on "Cleaning up external metadata files” forever. any ideas? I have 700+ jobs in the queue and would like to get them running asap... one hanlder just stays at Starting server in PID 4868. serving on http://127.0.0.1:8090 here’s the log from the other. Starting server in PID 4873. serving on http://127.0.0.1:8091 galaxy.jobs DEBUG 2014-08-05 17:34:08,388 (125206) Working directory for job is: /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 galaxy.jobs.handler DEBUG 2014-08-05 17:34:08,410 (125206) Dispatching to gridengine runner galaxy.jobs DEBUG 2014-08-05 17:34:08,479 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.handler INFO 2014-08-05 17:34:08,498 (125206) Job dispatched galaxy.tools.deps DEBUG 2014-08-05 17:34:08,862 Building dependency shell command for dependency 'bedtools2' galaxy.tools.deps WARNING 2014-08-05 17:34:08,865 Failed to resolve dependency on 'bedtools2', ignoring galaxy.jobs.runners DEBUG 2014-08-05 17:34:08,966 (125206) command is: bedtools --version > /mnt/galaxy/data/galaxy/tmp/GALAXY_VERSION_STRING_125206 2>&1; bedtools intersect -v -a /mnt/galaxy/data/galaxy/user-data/000/170/dataset_170995.dat -b /mnt/galaxy/data/galaxy/user-data/000/169/dataset_169046.dat > "/mnt/galaxy/data/galaxy/user-data/000/170/dataset_170999.dat"; return_code=$?; cd /mnt/galaxy/data/galaxy/galaxy-dist; /mnt/galaxy/data/galaxy/galaxy-dist/set_metadata.sh /mnt/galaxy/data/galaxy/user-data /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 . /mnt/galaxy/data/galaxy/galaxy-dist/universe_wsgi.ini /mnt/galaxy/data/galaxy/tmp/tmpLCBPvF /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy.json /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_in_HistoryDatasetAssociation_194826_qIy33T,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_kwds_HistoryDatasetAssociation_194826_SJhToq,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_out_HistoryDatasetAssociation_194826_4Lf956,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_results_HistoryDatasetAssociation_194826_YzEZK8,,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_override_HistoryDatasetAssociation_194826_19nZAm; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:08,994 (125206) submitting file /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy_125206.sh galaxy.jobs.runners.drmaa INFO 2014-08-05 17:34:09,023 (125206) queued as 1539118 galaxy.jobs DEBUG 2014-08-05 17:34:09,048 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:09,451 (125206/1539118) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:11,510 (125206/1539118) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:31,208 (125206/1539118) state change: job finished normally galaxy.jobs DEBUG 2014-08-05 17:34:31,440 setting dataset state to ERROR galaxy.jobs INFO 2014-08-05 17:34:31,652 Collecting job metrics for <galaxy.model.Job object at 0x7f42e030f850> galaxy.jobs DEBUG 2014-08-05 17:34:31,664 job 125206 ended galaxy.datatypes.metadata DEBUG 2014-08-05 17:34:31,665 Cleaning up external metadata files -- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist
Nothing comes to mind immediately, can you post a link to your job_conf.xml file? What is your `cleanup_job` option set to in universe_wsgi.ini? Can you try changing it to 'never' and see if the jobs continue? -John On Tue, Aug 5, 2014 at 5:54 PM, Langhorst, Brad <Langhorst@neb.com> wrote:
I’m not aware of any recent changes to the configuration of galaxy…
It seems that jobs are not being distributed at the moment.
I have 2 job handlers...
It seems that when I restart them only a single job is queued, it has an error and then sticks on "Cleaning up external metadata files” forever.
any ideas? I have 700+ jobs in the queue and would like to get them running asap...
one hanlder just stays at Starting server in PID 4868. serving on http://127.0.0.1:8090
here’s the log from the other.
Starting server in PID 4873. serving on http://127.0.0.1:8091 galaxy.jobs DEBUG 2014-08-05 17:34:08,388 (125206) Working directory for job is: /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 galaxy.jobs.handler DEBUG 2014-08-05 17:34:08,410 (125206) Dispatching to gridengine runner galaxy.jobs DEBUG 2014-08-05 17:34:08,479 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.handler INFO 2014-08-05 17:34:08,498 (125206) Job dispatched galaxy.tools.deps DEBUG 2014-08-05 17:34:08,862 Building dependency shell command for dependency 'bedtools2' galaxy.tools.deps WARNING 2014-08-05 17:34:08,865 Failed to resolve dependency on 'bedtools2', ignoring galaxy.jobs.runners DEBUG 2014-08-05 17:34:08,966 (125206) command is: bedtools --version > /mnt/galaxy/data/galaxy/tmp/GALAXY_VERSION_STRING_125206 2>&1; bedtools intersect -v -a /mnt/galaxy/data/galaxy/user-data/000/170/dataset_170995.dat -b /mnt/galaxy/data/galaxy/user-data/000/169/dataset_169046.dat > "/mnt/galaxy/data/galaxy/user-data/000/170/dataset_170999.dat"; return_code=$?; cd /mnt/galaxy/data/galaxy/galaxy-dist; /mnt/galaxy/data/galaxy/galaxy-dist/set_metadata.sh /mnt/galaxy/data/galaxy/user-data /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206 . /mnt/galaxy/data/galaxy/galaxy-dist/universe_wsgi.ini /mnt/galaxy/data/galaxy/tmp/tmpLCBPvF /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy.json /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_in_HistoryDatasetAssociation_194826_qIy33T,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_kwds_HistoryDatasetAssociation_194826_SJhToq,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_out_HistoryDatasetAssociation_194826_4Lf956,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_results_HistoryDatasetAssociation_194826_YzEZK8,,/mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/metadata_override_HistoryDatasetAssociation_194826_19nZAm; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:08,994 (125206) submitting file /mnt/galaxy/data/galaxy/job_working_directory/000/125/125206/galaxy_125206.sh galaxy.jobs.runners.drmaa INFO 2014-08-05 17:34:09,023 (125206) queued as 1539118 galaxy.jobs DEBUG 2014-08-05 17:34:09,048 (125206) Persisting job destination (destination id: gridengine) galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:09,451 (125206/1539118) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:11,510 (125206/1539118) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2014-08-05 17:34:31,208 (125206/1539118) state change: job finished normally galaxy.jobs DEBUG 2014-08-05 17:34:31,440 setting dataset state to ERROR galaxy.jobs INFO 2014-08-05 17:34:31,652 Collecting job metrics for <galaxy.model.Job object at 0x7f42e030f850> galaxy.jobs DEBUG 2014-08-05 17:34:31,664 job 125206 ended galaxy.datatypes.metadata DEBUG 2014-08-05 17:34:31,665 Cleaning up external metadata files
-- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (2)
-
John Chilton -
Langhorst, Brad