Hello all,
I'm using a local install of Galaxy connected to a UGE cluster and Python 2.6.7. Most of my tools are reporting failures in the history with messages like:
An error occurred running this job: (34) Job output not returned from cluster; exit status = "0"
With the relevant lines from Galaxy's log file looking like:
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,586 job 34 working directory is /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34 galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,636 job 34 input = [["/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat", "fastq", "5230830"]] galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,671 (34) submitting file /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/uge/galaxy_34.sh galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,672 (34) command is: memtimepro -q -o /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/34.drmmt perl /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/tools/lanl/readMapping/fastqSplitter/separate_paired_end_reads.pl -i /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat -l /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_53.dat -r /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_54.dat ; cd /opt/galaxy/dev/Galaxy-JGI_galaxy-dev; /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/set_metadata.sh ./database/files /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34 . /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/universe_wsgi.ini /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/tmp/tmp831aRx /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/galaxy.json /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_in_HistoryDatasetAssociation_59_fpBxRp,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_kwds_HistoryDatasetAssociation_59_MLg1Aw,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_out_HistoryDatasetAssociation_59_ZVYBIT,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_results_HistoryDatasetAssociation_59_q4FpIq,,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_override_HistoryDatasetAssociation_59_3HXPLq /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_in_HistoryDatasetAssociation_60_Qw0GpA,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_kwds_HistoryDatasetAssociation_60_ZF6Zk_,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_out_HistoryDatasetAssociation_60_eEItGr,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_results_HistoryDatasetAssociation_60_bMa6HN,,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_override_HistoryDatasetAssociation_60_X1El1z galaxy.jobs.runners.drmaa INFO 2012-10-25 10:31:26,720 (34) queued as 553774 galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:28,947 (34/553774) state change: job is running
-- snip --
galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:32:29,411 (34/553774) state change: job finished normally 128.165.72.57 - - [25/Oct/2012:10:32:31 -0600] "POST /root/history_item_updates HTTP/1.0" 200 - "http://galaxy-dev.lanl.gov/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:15.0) Gecko/20100101 Firefox/15.0.1" galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:32:31,241 (34) Job output not returned from cluster; exit status = "0 " galaxy.jobs DEBUG 2012-10-25 10:32:31,285 The tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2012-10-25 10:32:31,427 setting dataset state to ERROR galaxy.jobs DEBUG 2012-10-25 10:32:31,471 setting dataset state to ERROR galaxy.jobs DEBUG 2012-10-25 10:32:31,728 job 34 ended galaxy.datatypes.metadata DEBUG 2012-10-25 10:32:31,729 Cleaning up external metadata files
And yet, for this example, the tool did actually run successfully with the expected output files where they should be in the galaxy/database/files directory.
Further, the 34.drmec file has just the exit code 0, the 34.drmout file is empty and the 34.drmmt file is:
{ "program": "perl", "arguments": [ "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/tools/lanl/readMapping/fastqSplitter/separate_paired_end_reads.pl", "-i", "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat", "-l", "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_53.dat", "-r", "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_54.dat" ], "exit_status": 0, "start_time": 1351182686, "wall_time": 0.223, "usage": { "utime": 0.04, "stime": 0.08, "max_vmem_kb": 11920, "max_rss_kb": 156, "start_time": 1351182686, "wall_time": 0.223 } }
So everything looks good to me but Galaxy is reporting a failed run. What piece am I missing?
Thanks!