Reply

Easy Anomaly Detection for Home Dashboard -- DNS and DHCP QPS Increase

[ Edited ]
Expert
Posts: 181
2177     0

As written the code below will show you members where the DNS QPS or the DHCP LPS has increased 1.5 times over the last 2 hours vs the average of the same 2 hour window the previous 2 weeks.

This has a lot of noise in it.   You will need to tune it to your environment.   I’m still playing with some more filtering around this base and may turn these into an alert at some point but for now I just have them loaded on the home dashboard as a quick visual check when headed into the reporting tool.   

 

The first step to removing the noise is setting a low-end threshold.   If a small server goes from 10 QPS to 20 QPS its likely not something you want to alert on or maybe even see in a dashboard.  That code is easy to add.

This code is not nearly as sophisticated as the Splunk built in tools for anomaly detection, but at least for me, it was much easier to tune to our environment than the built-in tools.

<panel>
      <table>
        <title>DHCP Members where Message Rate is 1.5 * Greater than Last Week</title>
        <search>
          <query>sourcetype=ib:dhcp:message index=ib_dhcp earliest=-2h@m latest=now
            | eval subtotal = 0 
            | foreach dhcpv4* [ eval subtotal = subtotal + '&lt;&lt;FIELD&gt;&gt;']
            | stats sum(subtotal) as TOTAL, sum(dhcpv4*) as dhcpv4*_subtotal by host
            | join host [ search  sourcetype=ib:dhcp:message index=ib_dhcp earliest=-170h@m latest=-168h@m 
            | eval subtotalLastWeek = 0 
            | foreach dhcpv4* [ eval subtotalLastWeek = subtotalLastWeek + '&lt;&lt;FIELD&gt;&gt;'] 
            | stats sum(subtotalLastWeek) as TOTALLastWeek, sum(dhcpLastWeek*) as dhcpLastWeek*_subtotal by host ] 
            | join host [ search  sourcetype=ib:dhcp:message index=ib_dhcp earliest=-338h@m latest=-336h@m
            | eval subtotal2Weeks = 0 
            | foreach dhcpv4* [ eval subtotal2Weeks = subtotal2Weeks + '&lt;&lt;FIELD&gt;&gt;']
            | stats sum(subtotal2Weeks) as TOTAL2Weeks, sum(dhcp2Weeks*) as dhcp2Weeks*_subtotal by host ] 
            | eval Last2Avg=((TOTALLastWeek+TOTAL2Weeks)/2)
            |  where TOTAL &gt; 1.5 * Last2Avg AND Last2Avg &gt; 7200 
            | eval PercentChange=round(((TOTAL/Last2Avg) * 100),0) 
            | eval TodayRate=round((TOTAL/7200),1) 
            | eval RateLastWeek=round((Last2Avg/7200),1)  
            | rename host as Member  TodayRate as "Avg Rate Last Hour" RateLastWeek as "Avg QPS Last 2 Weeks Last Hour"
            | sort - PercentChange   
            | table Member, "Avg Rate Last Hour", "Avg QPS Last 2 Weeks Last Hour" 
			| appendpipe [stats count
            | eval Message="NO DHCP Members Found with an Increase in DHCP Rate"  
            | where count==0 
            | table Message]</query>
          <earliest>-26h@h</earliest>
          <latest>-1h@h</latest>
        </search>
        
      </table>
    </panel>
    <panel>
      <table>
        <title>DNS Members where QPS is 1.5 * Greater than Last Week</title>
        <search>
          <query>index=ib_dns  sourcetype=ib:dns:query:qps earliest=-2h@m latest=now 
            |stats sum(COUNT) as TodayLastHour by host 
            | join host [search index=ib_dns  sourcetype=ib:dns:query:qps earliest=-170h@m latest=-168h@m
            | stats sum(COUNT) AS LastWeekLastHour by host] 
            |join host [search index=ib_dns  sourcetype=ib:dns:query:qps earliest=-338h@m latest=-336h@m
            | stats sum(COUNT) AS TwoWeeksLastHour by host] 
            | eval Last2Avg=((LastWeekLastHour+TwoWeeksLastHour)/2) 
            | where TodayLastHour &gt;  1.5 * Last2Avg
            | eval PercentChange=round(((TodayLastHour/Last2Avg) * 100),0) 
            | eval TodayQPSLastHour=round((TodayLastHour/7200),1) 
            | eval LastWeekQPSLastHour=round((Last2Avg/7200),1) 
            | rename host as Member  TodayQPSLastHour as "Avg QPS Last Hour" LastWeekQPSLastHour as "Avg QPS Last 2 Weeks Last Hour"
            | sort - PercentChange
            | fields - TodayLastHour -  LastWeekLastHour - PercentChange - TwoWeeksLastHour - Last2Avg
            | table Member, "Avg QPS Last Hour", "Avg QPS Last 2 Weeks Last Hour"
            | appendpipe [stats count
            | eval Message="NO DNS Members Found with an Increase in DNS Rate"  
            | where count==0 
            | table Message]</query>
          <earliest>-7d</earliest>
          <latest>now</latest>
        </search>
        <option name="wrap">true</option>
        <option name="rowNumbers">false</option>
        <option name="dataOverlayMode">none</option>
        <drilldown>
          <link>dns_query_rate_by_query_type_real_time?form.members=$row.Member$</link>
        </drilldown>
        <option name="count">5</option>
      </table>
    </panel>

 

Showing results for 
Search instead for 
Do you mean 

Recommended for You