Hello all,

I'm using a local install of Galaxy connected to a UGE cluster and Python 2.6.7. Most of my tools are reporting failures in the history with messages like:

    An error occurred running this job: (34) Job output not returned from cluster; exit status = "0"

With the relevant lines from Galaxy's log file looking like:

galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,586 job 34 working directory is /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,636 job 34 input = [["/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat", "fastq", "5230830"]]
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,671 (34) submitting file /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/uge/galaxy_34.sh
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,672 (34) command is: memtimepro -q -o /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/34.drmmt perl /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/tools/lanl/readMapping/fastqSplitter/separate_paired_end_reads.pl        -i /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat       -l /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_53.dat       -r /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_54.dat ; cd /opt/galaxy/dev/Galaxy-JGI_galaxy-dev; /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/set_metadata.sh ./database/files /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34 . /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/universe_wsgi.ini /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/tmp/tmp831aRx /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/galaxy.json /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_in_HistoryDatasetAssociation_59_fpBxRp,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_kwds_HistoryDatasetAssociation_59_MLg1Aw,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_out_HistoryDatasetAssociation_59_ZVYBIT,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_results_HistoryDatasetAssociation_59_q4FpIq,,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_override_HistoryDatasetAssociation_59_3HXPLq /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_in_HistoryDatasetAssociation_60_Qw0GpA,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_kwds_HistoryDatasetAssociation_60_ZF6Zk_,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_out_HistoryDatasetAssociation_60_eEItGr,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_results_HistoryDatasetAssociation_60_bMa6HN,,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_override_HistoryDatasetAssociation_60_X1El1z
galaxy.jobs.runners.drmaa INFO 2012-10-25 10:31:26,720 (34) queued as 553774
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:28,947 (34/553774) state change: job is running

-- snip --

galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:32:29,411 (34/553774) state change: job finished normally
128.165.72.57 - - [25/Oct/2012:10:32:31 -0600] "POST /root/history_item_updates HTTP/1.0" 200 - "http://galaxy-dev.lanl.gov/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:15.0) Gecko/20100101 Firefox/15.0.1"
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:32:31,241 (34) Job output not returned from cluster; exit status = "0
"
galaxy.jobs DEBUG 2012-10-25 10:32:31,285 The tool did not define exit code or stdio handling; checking stderr for success
galaxy.jobs DEBUG 2012-10-25 10:32:31,427 setting dataset state to ERROR
galaxy.jobs DEBUG 2012-10-25 10:32:31,471 setting dataset state to ERROR
galaxy.jobs DEBUG 2012-10-25 10:32:31,728 job 34 ended
galaxy.datatypes.metadata DEBUG 2012-10-25 10:32:31,729 Cleaning up external metadata files


And yet, for this example, the tool did actually run successfully with the expected output files where they should be in the galaxy/database/files directory.

Further, the 34.drmec file has just the exit code 0, the 34.drmout file is empty and the 34.drmmt file is:

{
    "program": "perl",
    "arguments": [
        "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/tools/lanl/readMapping/fastqSplitter/separate_paired_end_reads.pl",
        "-i",
        "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat",
        "-l",
        "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_53.dat",
        "-r",
        "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_54.dat"
    ],
    "exit_status": 0,
    "start_time": 1351182686,
    "wall_time": 0.223,
    "usage": {
        "utime": 0.04,
        "stime": 0.08,
        "max_vmem_kb": 11920,
        "max_rss_kb": 156,
        "start_time": 1351182686,
        "wall_time": 0.223
    }
}


So everything looks good to me but Galaxy is reporting a failed run. What piece am I missing?

Thanks!