Reporting

Reply
Highlighted

Parsing Syslog Before Sending to Reporter

[ Edited ]
Expert
Posts: 173
3827     1

 

      There is some granularity lacking in the control of what messages get forwarded on to the reporter.

 

https://community.infoblox.com/t5/Reporting/Reporting-Volume-Usage-Trend-per-Syslog-Category/m-p/107...

 

      In my case to get at a few hundred megs of syslog messages to make the Recursive DNS Performance & Troubleshoot Dashboard work correctly, I would have to send over 10 gigs of “un-categorized” syslog messages to the reporter.
        I already send all my syslogs to a central linux box running rsyslog. So parsing out only what is needed and sending only those records back to the reporter should be fairly easy. I was already doing part of this to get to the EA’s that Infoblox has not opened to the Repoter. I ran into a few issues and learned a few things so I thought I’d document them and ask for help with a problem I’ve run into with my HA pairs.


1. Setup a central syslog repository on a linux or other box that gives you lots of control on how to forward on messages. I recommend rsyslog on a linux box but there are lots of other ways to make this happen.
2. Forward all your grid syslog messages there using the standard Infoblox syslog forwarding.
3. On one grid member turn on syslog proxy. Listen for inbound syslog messages from your central logging server.
4. Setup the syslog proxy member to forward ALL syslog messages to the reporter under the reporting tab config. The forwared \ parsed messages will fall under the non catigorized heading.
5. BE VERY CAREFUL ON THIS MEMBER’S “normal” SYSLOG CONFIG. If the configuration to forward to the central syslog repository is a grid setting, and you start forwarding a subset of the syslog messages to this member via its syslog proxy, your syslog proxy grid member can forward a second copy of these messages back to the central repository. Your central repository will re-parse them and forward them back on to this member a second time, which will then forward them to the central repository and the reporting member again.. Just think very carefully about your environment and loops. This can generate a lot of traffic and syslog data very quickly. Like hypothetically enough data to flood a 100mb link into your testing environment in about 30 seconds, while other testing is going on…
6. On your central repository, use the rsyslog config to parse just the messages out that you need from the DNS and DHCP logs and forward only them on to the syslog proxy grid member.

So I have rsyslogl lines that look like this:

 

if ($msg contains 'Recursion') or ($msg contains 'clients per query') or ($msg contains 'too many simultaneous') or ($msg contains 'resolving') then

action (type="omfwd" target="IP OF SYSLOG PROXY GRID MEMBER" port="514" protocol="udp" queue.filename="area51" queue.type="LinkedList" queue.size="10000" template="ADDHostName")

&stop

 

        Everything is pretty straight forward, accept the template which is where I’m stuck. When the syslog proxy server forwards on the syslog messages, ONLY to the reporting member, it puts its own name in for the "host" field in the syslog messages. It doesn’t do this for the messages it forwards on via the syslog proxy to “normal” destination, only to the reporter.

     I kind of feel this is broken, it should keep the original host name on the messages forward on to the reporter, but I'm guessing we will likely get little help fixing this since we are using this feature to pre-parse logs.


     I added the template where I put back in the IP of the original host so that the syslog proxy grid member wouldn’t mess with it on the way to the reporter.

$template ADDHostName,"%timegenerated% %HOSTNAME% goodhost=%HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

       This works for all my stand alone members pretty well. I can do an extraction of the “goodhost” field and do a lookup for via the nios_member_ip table to get the grid host name.   However, for HA pairs, the IP is the LAN 1 IP not the VIP, and all that is in the nios_member_ip is the VIP IP.

 

       So getting my HA pairs back to a host name and keeping the data consistent after a HA fail over is a bit of an issue.

All the magic of how the HA pairs put their VIP IP \ name into the syslog message bound for the reporting member is all behind the scenes. I assume it happens on the real members before the message gets sent.

 

Has anyone found a table in the reporter that has ALL the IP’s for a member to get back to the “real” grid name?

 

How about an easy API pull that I can update once a day?

 

Any other ideas?

Re: Parsing Syslog Before Sending to Reporter

[ Edited ]
cnsieler
Techie
Posts: 11
3828     1

This is a great post.  It is one that I have to read again.  I love the general concept.   Before we has the Reporting Appliance we had the following setup we are off loading the syslog to syslog servers.  That has not changed.  I like your concept.   So I am doing an index analysis of what we are feeding the syslog.  I am supporting a large enterprise.  I am using the NIOS GUI to look at the Members syslogs.  I am building a matrix of Facility, Server, and examples of what is populating (example messages) Alert, Critical, Debug, Emergency, Error, Info, Notice and Warning.  The problem is reverse engineering the settings in the Infoblox Grid DNS Properties.  In those properties we had selected client, config, database, etc. 

Then when we we acquired the Reporting Appliance we set up the Grid Reporting Properties with all the percentages.  The setting on syslog data was set to Send all and we turned on a flood.  We found out quickly that their was a License limit.

So then I did what all  people do when backed into a corner and starting reading the manual. (RTM)

 

 

Page 1377

"Logging Category: Select one of the following logging categories:

Send all: Select this to log all syslog messages, irrespective of categories to which it belongs. When you

select this option, the appliance logs syslog messages for all the events, including all DNS and Infoblox

related events. "

Note the following

"However, the syslog messages are not prefixed when you select this option."

 

This is the one I want to do...

" Send selected categories: Select this to configure logging categories from the list of available logging

categories. Use the arrows to move logging categories from the Available table to the Selected table

and vice versa. The appliance sends syslog messages for the categories that are in the Selected table.

When you select this option, you must add at least one logging category. The syslog messages are

prefixed with a category name to which it belongs. Also, the RPZ events logged in the syslog messages

uses specific prefixes for the selected categories. Note that the syslog messages are prefixed when you

set logging categories for at least one external syslog server, even if you set other external syslog

servers as Send All."

 

page 1378 (NIOS 8.0 Admin Guide)

” When you select the Send selected categories option, the syslog messages are prefixed with a category name to which it belongs."

 

"DNS Logging Categories: All DNS related messages use the following prefixes: client, config, database,

dnssec, general, lame_servers, network, notify, queries, query_rewrite, resolver, responses, rpz,

security, update, update_security, xfer_in, and xfer_out."

 

page 1381-1382

Logging Facility: Select a facility from the drop-down list. This is the location on the syslog server to which

you want to sort the DNS logging messages.

Logging Category: Select one or more of these log categories:

— general: Records the BIND messages that are not specifically classified.

— client: Enables the logging of messages related to query processing, but not the queries themselves.

Examples of messages include exceeding recursive client quota, and other errors related to recursive

clients, blacklist and NXDOMAIN interception, query name rewrite, and others.

config: Records the configuration file parsing messages.

— database: Records BIND’s internal database processes.

— dnssec: Records the DNSSEC-signed responses.

— lame servers: Records bad delegation instances.

— network: Records the network operation messages.

— notify: Records the asynchronous zone change notification messages.

— queries: Records the DNS queries. Note that enabling the logging of queries and responses will

significantly affect system performance. Ensure that your system has sufficient CPU capacity before you

enable DNS query logging.

— rate-limit: Logs RRL (Response Rate Limiting) events. You must enable RRL in order for the appliance to

log RRL events to this logging category.

— resolver: Logs messages related to outgoing queries from the 'named' process, when it is acting as a

resolver on behalf of clients.

— responses: Records DNS responses. Note that enabling the logging of queries and responses will

significantly affect system performance. Ensure that your system has sufficient CPU capacity before you

enable DNS response logging.

— rpz: Records log messages when responses are modified through RPZs or for which explicit passthrus

were invoked in the RPZs. This check box is not selected by default.

— security: Logs miscellaneous messages that are related to security, such as denial or approval (mostly

denial) of certain operations.

— transfer-in: Records zone transfer messages from the remote name servers to the appliance.

— transfer-out: Records zone transfer messages from the NIOS appliance to remote name servers.

— update: Records the dynamic update instances.

— update-security: Records the security updates.

— DTC load balancing: Records information about which client is directed to which server.

— DTC health monitors: Records any changes to the health state of a monitored server.

  1. Save the configuration and click Restart if it appears at the top of the screen.

I am doing some research and writing because I have to convince before we test again.  So more to follow

Re: Parsing Syslog Before Sending to Reporter

Expert
Posts: 173
3828     1

I like where you are going with this.   I asked for exactly this documentation from Infoblox and they said it likely didn't exist and it would take a great deal of time to create.   I agreed it would take a lot of time to reverse engineer, but it would seem like someone would already have to have it to build the categories.  It looks like you have a great start on discovering how things are split up.  I look forward to seeing your results. 

Re: Parsing Syslog Before Sending to Reporter

cnsieler
Techie
Posts: 11
3828     1

Thank you for your kind words.  I am an old plow horse that does not know any better.  In order to get reports one has to know their data.  I hate the old "just send me everything I will know it when I see it." 

Re: Parsing Syslog Before Sending to Reporter

[ Edited ]
Expert
Posts: 173
3828     1

@DEvans wrote:

     I kind of feel this is broken, it should keep the original host name on the messages forward on to the reporter, but I'm guessing we will likely get little help fixing this since we are using this feature to pre-parse logs.


     I added the template where I put back in the IP of the original host so that the syslog proxy grid member wouldn’t mess with it on the way to the reporter.

$template ADDHostName,"%timegenerated% %HOSTNAME% goodhost=%HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

       This works for all my stand alone members pretty well. I can do an extraction of the “goodhost” field and do a lookup for via the nios_member_ip table to get the grid host name.   However, for HA pairs, the IP is the LAN 1 IP not the VIP, and all that is in the nios_member_ip is the VIP IP.

 

       So getting my HA pairs back to a host name and keeping the data consistent after a HA fail over is a bit of an issue.


 To get the host name of the actual VIP of the HA pairs,  change the configuration Infoblox grid to include the hostname and IP in the syslog messages.  The syslog message sent to the remote syslog server will now include the hostname of both LAN1 and the VIP \ member name.   This allows you to send the VIP hostname and keep the data consistant as you have HA fail overs.


 

If you do this it takes a bit of extra parsing.   Rsyslog only grabs the first name in what should be the hostname field.   Infoblox is putting in 3 fields, the LAN1 hostname, then the VIP hostname or the LAN1 host name again if its not a HA pair, then the VIP or LAN1 IP.   The second two fields are just part of the message of the syslog so you have to parse out the second name.   This is what my template looks like now.

$template Reporter-template,"%timegenerated% %HOSTNAME% goodhost=%msg:F,32:2% %syslogtag%%msg:::drop-last-lf%\n"

I had some problems with the  goodhost=%msg:F,32:2% field extraction and rsyslog.   Some versions of rsyslog were not stopping at the next space correctly, so if your getting a bunch of extra junk after the host name, try upgrading rsyslog.

Showing results for 
Search instead for 
Do you mean 

Recommended for You