excellent, thanks for the correction!<br><br><div class="gmail_quote">On Wed, Jan 18, 2012 at 8:59 AM, Shantanu Pavgi <span dir="ltr">&lt;<a href="mailto:pavgi@uab.edu">pavgi@uab.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div style="word-wrap:break-word">
<div><br>
</div>
<div>Ed,</div>
<div><br>
</div>
<div>I think you may want to add job with DrmCommunicationException error back into the watched_jobs list.  </div>
<div>{{{</div>
<div><div class="im">
<div>except ( drmaa.DrmCommunicationException ), e:</div>
</div><div>  # Catch drmaa communication exception, log a warning message and</div>
<div>  # continue to monitor jobs.</div>
<div>  log.warning(&quot;(%s/%s) Couldn&#39;t communicate with the cluster scheduler to check job status.&quot; %( galaxy_job_id, job_id ))</div>
<div>  # keep old job state</div>
<div>  new_watched.append( drm_job_state )</div>
<div>  continue</div>
<div>}}}</div>
</div>
<div><br>
</div>
<div>Here is a patch and related discussion on it: <a href="https://bitbucket.org/galaxy/galaxy-central/pull-request/23/catch-drmcommunication-exception" target="_blank">https://bitbucket.org/galaxy/galaxy-central/pull-request/23/catch-drmcommunication-exception</a> </div>

<span class="HOEnZb"><font color="#888888">
<div><br>
</div>
<div>--</div>
<div>Shantanu</div></font></span><div><div class="h5">
<div><br>
</div>
<br>
<div>
<div>On Jan 13, 2012, at 2:11 PM, Edward Kirton wrote:</div>
<br>
<blockquote type="cite">
<div>i had seen the job process die with this error:
<div><br>
<div>
<div>    if state != old_state:</div>
<div>UnboundLocalError: local variable &#39;state&#39; referenced before assignment</div>
<div><br>
</div>
<div>since the DRM timeout is an intermittent error, i&#39;m not absolutely positive i have nailed it (a bit more time will tell -- i&#39;m keeping an eye on it with debug log messages), but it seems so.  i intended to share this as a patched clone when i became sure
 but when i read your email i just sent what i had immediately.  let us know if that seems to solve the problem for you, so we&#39;ll have some confirmation.</div>
<div><br>
</div>
<div>glad to help,</div>
<div>ed</div>
<div><br>
</div>
<div>p.s. i have another patch for &quot;output not returned from cluster&quot; errors that i&#39;m also still validating, due to NFS race since we don&#39;t have inode metadata caching turned off as the galaxy developers suggest.</div>


<br>
<div class="gmail_quote">On Fri, Jan 13, 2012 at 8:06 AM, Ann Black <span dir="ltr">
&lt;<a href="mailto:annblack@eng.uiowa.edu" target="_blank">annblack@eng.uiowa.edu</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>Thanks so much! I have applied the fix to our env.  </div>
<div><br>
</div>
<div>In looking over the logic, there was an existing exception block that would have caught the communication exception generically – but the job moved (in this scenario erroneously) the job into a failure workflow.  I would like to understand what ended up
 hanging galaxy – so it must be related to a valid job being moved into failure state ? Did you follow it down the rabbit hole by any chance to see what caused the hang in your env ? </div>
<div><br>
</div>
<div>Thanks again,</div>
<div><br>
</div>
<div>Ann</div>
<div><br>
</div>
<span>
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">


<span style="font-weight:bold">From: </span>Edward Kirton &lt;<a href="mailto:eskirton@lbl.gov" target="_blank">eskirton@lbl.gov</a>&gt;<br>
<span style="font-weight:bold">Date: </span>Thu, 12 Jan 2012 13:00:27 -0800<br>
<span style="font-weight:bold">To: </span>Ann Black &lt;<a href="mailto:annblack@eng.uiowa.edu" target="_blank">annblack@eng.uiowa.edu</a>&gt;<br>
<span style="font-weight:bold">Cc: </span>&quot;<a href="mailto:galaxy-dev@lists.bx.psu.edu" target="_blank">galaxy-dev@lists.bx.psu.edu</a>&quot; &lt;<a href="mailto:galaxy-dev@lists.bx.psu.edu" target="_blank">galaxy-dev@lists.bx.psu.edu</a>&gt;<br>


<span style="font-weight:bold">Subject: </span>Re: [galaxy-dev] Galaxy Hang after DrmCommunicationException<br>
</div>
<div>
<div>
<div><br>
</div>
<div>sometimes the scheduler can&#39;t keep up with all the work in it&#39;s 15sec cycle, so it doesn&#39;t respond to some messages.  here&#39;s a fix i&#39;ve been trying that seems to work.</div>
<div><br>
</div>
in lib/galaxy/jobs/runners/drmaa.py:
<div><br>
</div>
<div>
<div>    def check_watched_items( self ):</div>
<div>        &quot;&quot;&quot;</div>
<div>        Called by the monitor thread to look at each watched job and deal</div>
<div>        with state changes.</div>
<div>        &quot;&quot;&quot;</div>
<div>        new_watched = []</div>
<div>        for drm_job_state in self.watched:</div>
<div>            job_id = drm_job_state.job_id</div>
<div>            galaxy_job_id = drm_job_state.job_wrapper.job_id</div>
<div>            old_state = drm_job_state.old_state</div>
<div>            try:</div>
<div>                state = self.ds.jobStatus( job_id )</div>
<div>            # InternalException was reported to be necessary on some DRMs, but</div>
<div>            # this could cause failures to be detected as completion!  Please</div>
<div>            # report if you experience problems with this.</div>
<div>            except ( drmaa.InvalidJobException, drmaa.InternalException ), e:</div>
<div>                # we should only get here if an orphaned job was put into the queue at app star</div>
<div>tup</div>
<div>                log.debug(&quot;(%s/%s) job left DRM queue with following message: %s&quot; % ( galaxy_jo</div>
<div>b_id, job_id, e ) )</div>
<div>                self.work_queue.put( ( &#39;finish&#39;, drm_job_state ) )</div>
<div>                continue</div>
<div>            # BEGIN DRM TIMEOUT: Don&#39;t fail on scheduler communication error (probably just too busy)</div>
<div>            except ( drmaa.DrmCommunicationException ), e:</div>
<div>                log.warning(&quot;(%s/%s) DRM Communication Exception&quot; % ( galaxy_job_id, job_id</div>
<div> ))</div>
<div>                continue</div>
<div>            # END DRM TIMEOUT</div>
<br>
<div class="gmail_quote">On Wed, Jan 11, 2012 at 9:18 AM, Ann Black <span dir="ltr">
&lt;<a href="mailto:annblack@eng.uiowa.edu" target="_blank">annblack@eng.uiowa.edu</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>Good Morning galaxy group!</div>
<div><br>
</div>
<div>I was hoping that someone might have some ideas on a problem we have experienced a handful of times running galaxy on our local cluster.  </div>
<div><br>
</div>
<div>Occasionally we experience some communication timeouts between out cluster head node and a compute node which will self heal. However, this in turn will hang galaxy.  Below you will see output from our galaxy log file.  When the ERROR happens (which is
 not often) it consistently seems to hang galaxy.  We have to kill it off and restart it. We are running galaxy as a single PID at this time (we are still just testing it out, etc) and it is running on our head node (which we plan to move off of in the future). </div>


<div><br>
</div>
<div>
<div>galaxy.jobs.runners.drmaa DEBUG <a href="tel:2012-01-10%2019" value="+12012011019" target="_blank">
2012-01-10 19</a>:19:58,800 (1654/698075) state change: job is running</div>
<div>galaxy.jobs.runners.drmaa ERROR <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:57:47,021 (1654/698075) Unable to check job status</div>
<div>Traceback (most recent call last):</div>
<div>  File &quot;/data/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py&quot;, line 236, in check_watched_items</div>
<div>    state = self.ds.jobStatus( job_id )</div>
<div>  File &quot;/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/__init__.py&quot;, line 522, in jobStatus</div>
<div>    _h.c(_w.drmaa_job_ps, jobName, _ct.byref(status))</div>
<div>  File &quot;/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/helpers.py&quot;, line 213, in c</div>
<div>    return f(*(args + (error_buffer, sizeof(error_buffer))))</div>
<div>  File &quot;/data/galaxy-dist/eggs/drmaa-0.4b3-py2.7.egg/drmaa/errors.py&quot;, line 90, in error_check</div>
<div>    raise _ERRORS[code-1](&quot;code %s: %s&quot; % (code, error_buffer.value))</div>
<div>DrmCommunicationException: code 2: failed receiving gdi request response for mid=24442 (got syncron message receive timeout error).</div>
<div>galaxy.jobs.runners.drmaa WARNING <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:58:05,090 (1654/698075) job will now be errored</div>
<div>galaxy.jobs.runners.drmaa DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:06,396 (1654/698075) User killed running job, but error encountered removing from DRM queue: code 2: failed receiving gdi request response for mid=24444 (got syncron message receive timeout error).</div>


<div>galaxy.datatypes.metadata DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:06,896 Cleaning up external metadata files</div>
<div>galaxy.datatypes.metadata DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:06,947 Failed to cleanup MetadataTempFile temp files from database/tmp/metadata_out_HistoryDatasetAssociation_2913_ZUTgBy: No JSON object could be decoded: line 1 column 0 (char 0)</div>
<div>galaxy.datatypes.metadata DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:09,640 Cleaning up external metadata files</div>
<div><a href="http://galaxy.jobs/" target="_blank">galaxy.jobs</a> INFO <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:09,697 job 1656 unable to run: one or more inputs in error state</div>
<div>galaxy.datatypes.metadata DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:10,121 Cleaning up external metadata files</div>
<div><a href="http://galaxy.jobs/" target="_blank">galaxy.jobs</a> INFO <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:10,159 job 1655 unable to run: one or more inputs in error state</div>
<div>galaxy.datatypes.metadata DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:12,076 Cleaning up external metadata files</div>
<div><a href="http://galaxy.jobs/" target="_blank">galaxy.jobs</a> INFO <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:12,126 job 1657 unable to run: one or more inputs in error state</div>
<div>galaxy.datatypes.metadata DEBUG <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:13,601 Cleaning up external metadata files</div>
<div><a href="http://galaxy.jobs/" target="_blank">galaxy.jobs</a> INFO <a href="tel:2012-01-10%2020" value="+12012011020" target="_blank">
2012-01-10 20</a>:59:13,650 job 1658 unable to run: one or more inputs in error state</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Has anyone else experienced this or have some ideas on how we can further debug to figure out why galaxy hangs?</div>
<div><br>
</div>
<div>Thanks much!</div>
<div><br>
</div>
<div>Ann Black-Ziegelbein</div>
</div>
<br>
___________________________________________________________<br>
Please keep all replies on the list by using &quot;reply all&quot;<br>
in your mail client.  To manage your subscriptions to this<br>
and other Galaxy lists, please use the interface at:<br>
<br>
 <a href="http://lists.bx.psu.edu/" target="_blank">http://lists.bx.psu.edu/</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
___________________________________________________________<br>
Please keep all replies on the list by using &quot;reply all&quot;<br>
in your mail client.  To manage your subscriptions to this<br>
and other Galaxy lists, please use the interface at:<br>
<br>
 <a href="http://lists.bx.psu.edu/" target="_blank">http://lists.bx.psu.edu/</a></blockquote>
</div>
<br>
</div></div></div>

</blockquote></div><br>