On Oct 25, 2012, at 4:34 PM, Yilk, Todd A wrote:

Hello all,

I'm using a local install of Galaxy connected to a UGE cluster and Python 2.6.7. Most of my tools are reporting failures in the history with messages like:

An error occurred running this job: (34) Job output not returned from cluster; exit status = "0"

With the relevant lines from Galaxy's log file looking like:

galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,586 job 34 working directory is /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34 galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,636 job 34 input = [["/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat", "fastq", "5230830"]] galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,671 (34) submitting file /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/uge/galaxy_34.sh galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:26,672 (34) command is: memtimepro -q -o /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/34.drmmt perl /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/tools/lanl/readMapping/fastqSplitter/separate_paired_end_reads.pl -i /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat -l /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_53.dat -r /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_54.dat ; cd /opt/galaxy/dev/Galaxy-JGI_galaxy-dev; /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/set_metadata.sh ./database/files /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34 . /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/universe_wsgi.ini /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/tmp/tmp831aRx /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/galaxy.json /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_in_HistoryDatasetAssociation_59_fpBxRp,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_kwds_HistoryDatasetAssociation_59_MLg1Aw,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_out_HistoryDatasetAssociation_59_ZVYBIT,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_results_HistoryDatasetAssociation_59_q4FpIq,,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_override_HistoryDatasetAssociation_59_3HXPLq /opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_in_HistoryDatasetAssociation_60_Qw0GpA,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_kwds_HistoryDatasetAssociation_60_ZF6Zk_,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_out_HistoryDatasetAssociation_60_eEItGr,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_results_HistoryDatasetAssociation_60_bMa6HN,,/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/job_working_directory/000/34/metadata_override_HistoryDatasetAssociation_60_X1El1z galaxy.jobs.runners.drmaa INFO 2012-10-25 10:31:26,720 (34) queued as 553774 galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:31:28,947 (34/553774) state change: job is running

-- snip --

galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:32:29,411 (34/553774) state change: job finished normally 128.165.72.57 - - [25/Oct/2012:10:32:31 -0600] "POST /root/history_item_updates HTTP/1.0" 200 - "http://galaxy-dev.lanl.gov/history" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:15.0) Gecko/20100101 Firefox/15.0.1" galaxy.jobs.runners.drmaa DEBUG 2012-10-25 10:32:31,241 (34) Job output not returned from cluster; exit status = "0 " galaxy.jobs DEBUG 2012-10-25 10:32:31,285 The tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2012-10-25 10:32:31,427 setting dataset state to ERROR galaxy.jobs DEBUG 2012-10-25 10:32:31,471 setting dataset state to ERROR galaxy.jobs DEBUG 2012-10-25 10:32:31,728 job 34 ended galaxy.datatypes.metadata DEBUG 2012-10-25 10:32:31,729 Cleaning up external metadata files

And yet, for this example, the tool did actually run successfully with the expected output files where they should be in the galaxy/database/files directory.

Further, the 34.drmec file has just the exit code 0, the 34.drmout file is empty and the 34.drmmt file is:

Hi Todd, I've moved this over to galaxy-dev since it concerns a local installation. There should be a 34.drmerr file in addition to 34.drmout and 34.drmec. If any of the three are missing or unreadable when the job finishes, you'll encounter this error. You might want to try increasing the value of "retry_job_output_collection". Also, the changes in bfbd456ebbbe will show exactly which file is causing the problem, and the error message from the OS. --nate

{ "program": "perl", "arguments": [ "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/tools/lanl/readMapping/fastqSplitter/separate_paired_end_reads.pl", "-i", "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_48.dat", "-l", "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_53.dat", "-r", "/opt/galaxy/dev/Galaxy-JGI_galaxy-dev/database/files/000/dataset_54.dat" ], "exit_status": 0, "start_time": 1351182686, "wall_time": 0.223, "usage": { "utime": 0.04, "stime": 0.08, "max_vmem_kb": 11920, "max_rss_kb": 156, "start_time": 1351182686, "wall_time": 0.223 } }

So everything looks good to me but Galaxy is reporting a failed run. What piece am I missing?

Thanks!

___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:

http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/