Reply
Highlighted

Syslog Based Query Logging Report

[ Edited ]
Adviser
Posts: 118
3858     3

The DNS Domains Queried by Client report that comes with Infoblox Reporting and Analytics provides query logging, but it requires the Infoblox Data Connector. The Data Connector is designed to offload the processing of query logs and provide filtering so that you can send only queries of interest to the reporting server. If you don't mind the overhead on your grid members (mainly from sending large volumes of syslog messages), and you have the indexing capacity for the query volume you'll be sending to reporting, this report allows you to report on query data from the syslog the same way you can with the Data Connector.

 

 

 

screencapture-10-60-16-25-ui-ItsVzR25p7czOqlcQUemZQ-Itsd7-emZ11-1490310144444.png

 

 

BEWARE!! Turning this on could cuase you to exceede your daily indexing limit. This highly depends on your qps rate and your license size. The average syslog message, done with some small sample size testing, is around 170 bytes per query. This means around 1300 qps will cause overages on a 20GB/day license (1300 qps * 170 bytes * 86400 seconds = 19GB/day). Plus there is the other NIOS reporting data being indexed. Do some math up front as a precautionary measure using the DNS Daily Query Rate by Member report as the baseline for your average QPS rate. It's worth point out that in addition to filtering, the Data Connector indexes the queries more efficiently than syslog, resulting in lower indexing for the same QPS rate.

 

But if you do go over your indexing rate for the day the system will display a warning. If this is done 5 times within a rolling 30 day period reports will not longer render until there is less than 5 violations in the 30 day rolling average. Even though reports aren't rendered, data will still be indexed. Infoblox support can reset the counter, and your account manager or Infoblox reseller can get you more indexing capacity.

 

Prerequisites:

  1. You must configure NIOS to send syslog messages to the repporting server (NIOS 8.0+ feature). This is found in the reporting properties under the "syslog data" tab.
  2. You must enable the ib_syslog index under grid reporting properties -> General and allocate some percentage of the indexex to it
  3. You must turn on query logging for each member you wish to get logs for. This is found in the DNS properties for each member in the "logging" tab.

 

Once complete you should see query data in the reporting server by searching for "index=ib_syslog" in the search tab.

 

Next you'll need to set up new extractions. To do this go to the reporting tab and click "settings" in the top right grey bar and navigate to Fields » Field extractions » New. Configure the settings as shown below. The Regex is also in the code box below for easy cut & paste.

Untitled.png

^(?:[^ \n]* ){6}(?P<src_ip>[^#]+)(?:[^ \n]* ){3}(?P<query>[^ ]+)\s+\w+\s+(?P<query_type>\w+)[^\(\n]*\((?P<host2>[^\)]+)

 

Once complete you can now add the new report/dashboard by going to the dashboards tab and creating a new dashboard and using the code below.

<form>
  <label>DNS Domains Queried by Client (syslog)</label>
  <description>System-created dashboard: Please clone before editing.</description>
  <fieldset submitButton="true" autoRun="true">
    <input type="time" token="time">
      <label>Time</label>
      <default>
        <earliest>-1d</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="text" token="src_ip">
      <label>Client IP Address (e.g. 192.168.1.2)</label>
      <default>All</default>
      <change>
        <condition value="All">
          <set token="src_ip_str">
            <![CDATA[ ]]>
          </set>
        </condition>
        <condition value="*">
          <set token="src_ip_str">(src_ip=$value$)</set>
        </condition>
      </change>
    </input>
    <input type="text" token="query">
      <label>Domain Name (e.g. www.company.com)</label>
      <default>All</default>
      <change>
        <condition value="All">
          <set token="query_str">
            <![CDATA[ ]]>
          </set>
        </condition>
        <condition value="*">
          <set token="query_str">(query=$value$)</set>
        </condition>
      </change>
    </input>
    <input type="dropdown" token="query_type">
      <label>Query Type</label>
      <choice value="all">All</choice>
      <choice value="A">A</choice>
      <choice value="AAAA">AAAA</choice>
      <choice value="CNAME">CNAME</choice>
      <choice value="DNAME">DNAME</choice>
      <choice value="DNSKEY">DNSKEY</choice>
      <choice value="DS">DS</choice>
      <choice value="NAPTR">NAPTR</choice>
      <choice value="NSEC">NSEC</choice>
      <choice value="NSEC3PARAM">NSEC3PARAM</choice>
      <choice value="NSEC3">NSEC3</choice>
      <choice value="RRSIG">RRSIG</choice>
      <choice value="SRV">SRV</choice>
      <choice value="TXT">TXT</choice>
      <choice value="PTR">PTR</choice>
      <choice value="NS">NS</choice>
      <choice value="MX">MX</choice>
      <choice value="SOA">SOA</choice>
      <default>all</default>
      <change>
        <condition value="all">
          <set token="query_type_str">
            <![CDATA[ ]]>
          </set>
        </condition>
        <condition value="*">
          <set token="query_type_str">query_type="$value$"</set>
        </condition>
      </change>
    </input>
    <input type="multiselect" token="members">
      <label>Hosts</label>
      <choice value="*">All</choice>
      <search>
        <query>index=ib_syslog "query:"                | stats count by host2</query>
        <earliest>$time.earliest$</earliest>
        <latest>$time.latest$</latest>
      </search>
      <fieldForLabel>host2</fieldForLabel>
      <fieldForValue>host2</fieldForValue>
      <default>*</default>
      <prefix>(</prefix>
      <suffix>)</suffix>
      <valuePrefix>host2="</valuePrefix>
      <valueSuffix>"</valueSuffix>
      <delimiter> OR </delimiter>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <search>
          <query>index=ib_syslog ": query:"
            $src_ip_str$
            $query_str$
            $query_type_str$
            $members$
            | rename _time as Time
            | eval Timestamp=strftime(Time, "%Y-%m-%d %H:%M:%S %Z")
            | rename src_ip as "Source IP Address", query as "Domain Name", query_type as "Query Type", host2 as "Host", display_name as "View"
            | table "Timestamp" "Source IP Address" "Domain Name" "Query Type" "Host"</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="rowNumbers">false</option>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
</form>

 

Re: DNS Domains Queried by Client - Syslog Version

[ Edited ]
aabouzaher
Techie
Posts: 2
3859     3

Got it working.

 

Thank you

Re: DNS Domains Queried by Client - Syslog Version

aabouzaher
Techie
Posts: 2
3859     3

When I put client IP Addess in the filter "Client IP Address", it doesn't display the data. 

Showing results for 
Search instead for 
Do you mean 

Recommended for You