Deferred DDNS updates on DHCP servers. GSS-TSIG TSIG [ Edited ]

[#1]

These are some reports \ alerts I have written around dynamic DNS updates pending on Infoblox DHCP members.   Most of the issues in our environment with this process involve Infoblox DHCP members sending DDNS updates to Microsoft DNS servers using GSS-TSIG.   These scripts will also catch Infoblox to Infoblox problems and Infoblox to random broken other DNS servers as well.   The "random other DNS servers" are generally a group of miss configured clients or servers that wind up asking to send DDNS updates to a typo'ed DNS suffix... etc. 

Service restarts on Infoblox and reboots of Domain Controllers on the Microsoft side can cause spikes in the DDNS update queues as the GSS-TSIG keys time out and are re-negotiated. The 2 hour average on the alerts takes care of these "normal" spikes and starts to high light the boxes that are really having problems for some reason.

These syslog field extractions are needed:

ib:syslog : EXTRACT-ServerIP	(?=[^D]*(?:Deferring GSS-TSIG DDNS updates to DNS server|D.*Deferring GSS-TSIG DDNS updates to DNS server))^\d+\-\d+\-\d+\w+\d+:\d+:\d+\-\d+:(\d+\s+)+\w+(\d+\s+)+\w+\d+\.\w+\.\w+\.(\w+\s+)+\w+\[\d+\]:\s+(\w+\s+)+\w+\-(\w+\s+)+(?P<ServerIP>[^ ]+)

ib:syslog : EXTRACT-Suffix,Error  (?=[^U]*(?:Unable to add forward map|U.*Unable to add forward map))^(?:[^\.\n]*\.){4}(?P<Suffix>[^ ]+)[^:\n]*:\s+(?P<Error>.+) 

This is the Home Dashboard status table I use.   It can also easily be written to do alerting as well.

<panel>
      <table>
        <title>DHCP Servers with Pending DDNS updates -- Last 2 hour ave &gt; 200 pending</title>
        <search>
          <query>index=* sourcetype=ib:syslog Processed  
           | bin _time span=2h                         
| stats avg(DeferredAgain) AS DeferredAgain by host _time 
| where DeferredAgain &gt; 200            
 | table host, DeferredAgain            
 | appendpipe [stats count           
  | eval Message="NO DHCP Members Found with High DDNS Updates"       
        | where count==0            
  | table Message]
         </query>
          <earliest>-2h@h</earliest>
          <latest>@h</latest>
        </search>
        <option name="wrap">undefined</option>
        <option name="rowNumbers">undefined</option>
        <drilldown>
          <link>2dhcpdeferredddnsupdates?form.members=$row.host$</link>
        </drilldown>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="count">10</option>
      </table>
    </panel>

Clicking on the "alerting" members in the dashboard will bring up this detailed Dashboard.

<form>
  <label>2-DHCP-Deferred-DDNS-Updates</label>
  <description></description>
  <fieldset submitButton="true" autoRun="true">
    <input type="time" token="time">
      <label>Time</label>
      <default>
        <earliest>-1d</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="members">
      <label>Members</label>
      <choice value="*">All</choice>
      <search>
        <query>index=ib_DHCP_summary 
               | stats count by orig_host</query>
        <earliest>-1d</earliest>
        <latest>now</latest>
      </search>
      <fieldForLabel>orig_host</fieldForLabel>
      <fieldForValue>orig_host</fieldForValue>
      <default>*</default>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <valuePrefix>host="</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> OR </delimiter>
      <fieldset>
        <input token="members"></input>
      </fieldset>
    </input>
    <input type="dropdown" token="ea_site">
      <label>Member Site</label>
      <choice value="All">All</choice>
      <default>All</default>
      <search>
        <query>| inputlookup __grouping_by_ea_tag_lookup
               | spath input=EA path=Site output=EA_Site
               | stats count by EA_Site</query>
        <earliest>$time.earliest$</earliest>
        <latest>$time.latest$</latest>
      </search>
      <fieldForLabel>EA_Site</fieldForLabel>
      <fieldForValue>EA_Site</fieldForValue>
      <change>
        <condition value="All">
          <set token="ea_site_str">| noop</set>
        </condition>
        <condition value="*">
          <set token="ea_site_str">|  lookup __grouping_by_ea_tag_lookup host | spath input=EA path=Site 

output=EA_Site |where EA_Site="$value$"</set>
        </condition>
      </change>
    </input>
    <input type="dropdown" token="ea_function">
      <label>Member Function</label>
      <choice value="All">All</choice>
      <default>All</default>
      <search>
        <query>| inputlookup __grouping_by_ea_tag_lookup
               | spath input=EA path=MemberFunction output=EA_function
               | stats count by EA_function</query>
        <earliest>$time.earliest$</earliest>
        <latest>$time.latest$</latest>
      </search>
      <fieldForLabel>EA_function</fieldForLabel>
      <fieldForValue>EA_function</fieldForValue>
      <change>
        <condition value="All">
          <set token="ea_function_str">| noop</set>
        </condition>
        <condition value="*">
          <set token="ea_function_str">|  lookup __grouping_by_ea_tag_lookup host | spath input=EA path=MemberFunction output=EA_function
                                    | where EA_function="$value$"</set>
        </condition>
      </change>
    </input>
    <input type="link" token="view" searchWhenChanged="true">
      <label>View</label>
      <choice value="chart">Line Chart</choice>
      <choice value="table">Table</choice>
      <choice value="all">All</choice>
      <default>chart</default>
      <change>
        <condition value="table">
          <set token="show_table">true</set>
          <unset token="show_chart"></unset>
          <unset token="show_stacked_area"></unset>
        </condition>
        <condition value="chart">
          <set token="show_chart">true</set>
          <unset token="show_table"></unset>
          <unset token="show_stacked_area"></unset>
        </condition>
        <condition value="stacked_area">
          <set token="show_stacked_area">true</set>
          <unset token="show_table"></unset>
          <unset token="show_chart"></unset>
        </condition>
        <condition value="all">
          <set token="show_chart">true</set>
          <set token="show_stacked_area">true</set>
          <set token="show_table">true</set>
        </condition>
      </change>
    </input>
  </fieldset>
  <search id="base_search">
    <query>index=* sourcetype=ib:syslog Processed 
           $members$
           $ea_site_str$
      	   $ea_function_str$
      
           
           | timechart sum(DeferredAgain) AS DeferredAgain sum(Abandoned) AS Abandoned sum(Sucesses) AS Sucesses sum(TotalProcessed) AS TotalProcessed</query>
    <earliest>$time.earliest$</earliest>
    <latest>$time.latest$</latest>
  </search>
  <row>
    <panel>
      <chart depends="$show_chart$">
        <search base="base_search">
          <query>| rename _time as Time
                 | eval Time=strftime(Time, "%m-%d %H:%M")</query>
        </search>
        <option name="height">500px</option>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">90</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.axisTitleX.text">Time</option>
        <option name="charting.axisTitleY.text">Queries Per Second</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table depends="$show_table$">
        <search base="base_search">
          <query>| sort -_time
                 | rename _time as Time
                 | eval Time=strftime(Time, "%Y-%m-%d %H:%M:%S %Z")</query>
        </search>
        <option name="wrap">true</option>
        <option name="rowNumbers">true</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="count">10</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Suffixes Failing</title>
        <search>
          <query>index=ib_syslog  "Deferring GSS-TSIG DDNS updates to DNS server" OR "Unable to add forward map" 
             $members$
           $ea_site_str$
      	   $ea_function_str$
            | lookup dnslookup clientip as ServerIP OUTPUT clienthost as ServerName 
            | eval Suffix=if(isnull(Suffix), ServerName , Suffix )  
            | eval Error=if(isnull(Error), "GSS-TSIG Key Pending" , Error )
            | stats count(Suffix) as Count by host Suffix Error 
            | sort - Count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="wrap">undefined</option>
        <option name="rowNumbers">undefined</option>
        <option name="drilldown">row</option>
      </table>
    </panel>
  </row>
</form>

The detailed dashboard gives you Info like this.

[#2]

Hello David,

Thank you for sharing this to the community. I’ve got several “Unable to add forward map*” messages in my syslogs which have been forwarded to my reporting server. But the dashboard doesn’t pull any data with the sourcecode shared below. Are there any other requisites ? Is that the complete sourcecode ?

Best regards,

Mohammed Alman.

[#3 · Accepted solution]

That is the complete source code.    My guess would be that the issue is in the field extractions.   Those make some assumptions about the formatting of the syslog messages and the servers involved that may not be true in your environment.   I'd focus on those first to see that they are actually getting the right fields out of the raw syslog data.