Hi Bjorn and Hans,

We are running Galaxy on our local webserver, so there is no job scheduler.  Instead, we are using the localJobRunner configuration in the job_conf.xml

<?xml version="1.0"?>
<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="8"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
    </plugins>
    <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore6" runner="multilocal">
          <param id="local_slots">6</param>
        </destination>
    </destinations>
    <tools>
    <tool id="bowtie2" destination="multicore6" />
    <tool id="spades"  destination="multicore6" />
    <tool id="bbmap_1"  destination="multicore6" />
    <tool id="iuc_pear"  destination="multicore6" />
    <tool id="abyss-pe"  destination="multicore6" />
    <tool id="fastq_groomer_parallel"  destination="multicore6" />
    </tools>

    <handlers>
        <handler id="main"/>
    </handlers>
</job_conf>

Also, normally, we run galaxy in the daemon mode, but recently to help debug this issue, we have been running galaxy in interactive mode in a screen session.  

@Hans - The processes run under the galaxy user and everything seems to run fine.  I am trying to get a concrete example, but run.sh usually crashes during our trimming/assembly steps for NGS data.  Sometimes these workflows run to completion, but sometimes they crash the run.sh process.  When run.sh crashes, the individual running programs keep running as the galaxy user.  We cannot restart galaxy until we manually kill those running processes.  Here are what the running processes look like on the system.

galaxy         21162  99.4  0.0  2432784    616 s014  R     2:08PM 1425:10.57 seqtk seq -q 0 -X 255 -l 0 -Q 33 -s 11 -f 1.0 -L 0 -1 /Users/galaxy/data_galaxy/test_BACs/all-merged.interleaved.fq
galaxy         21159  99.4  0.0  2432784    616 s014  R     2:08PM 1425:16.72 seqtk seq -q 0 -X 255 -l 0 -Q 33 -s 11 -f 1.0 -L 0 -2 /Users/galaxy/data_galaxy/test_BACs/all-merged.interleaved.fq
galaxy         21118  92.5  0.5  3015188 367852 s014  R+    2:07PM 791:21.54 python ./scripts/paster.py serve universe_wsgi.ini
galaxy         21160   0.0  0.0  2433640   1044 s014  S     2:08PM   0:00.01 /bin/sh /Users/galaxy/galaxy-dist/database/job_working_directory/002/2180/galaxy_2180.sh
galaxy         21157   0.0  0.0  2433640   1044 s014  S     2:08PM   0:00.01 /bin/sh /Users/galaxy/galaxy-dist/database/job_working_directory/002/2179/galaxy_2179.sh
galaxy         21113   0.0  0.0  2433640   1000 s014  S+    2:07PM   0:00.00 sh run.sh


Thank you for the help,
-Hans