We experienced an issue where some of the galaxy jobs were sitting in the 'new'
state for a quite long time. They were not waiting for cluster resources to become
available, but haven't been even queued up through DRMAA. We are currently using
non-debug mode and following were my observations:
* No indication of new jobs in paster.log file
* database/pbs script didn't contain any associated job scripts
* in backend database - job table contained their galaxy job id but no command_line input
Also, not all the jobs are waiting in the 'new' state. Many jobs submitted after
above waiting jobs got completed successfully on the cluster. Is there any job submission
logic within galaxy which is being used for submitting jobs? Any clues on how to debug
this issue will be really helpful.