Reporting

Reply
Highlighted

Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5233     0

There have been several questions around getting to all the EA's for all the data types from the reporting tool.  ( I know, most of them are from me)  I think there are RFE's already open for this but it appears that the ability to pull them is already there, with some creative scripting.

I got close to getting at them by setting up a new lookup table of type external.  But I don't know python and I'm not sure what paths are available to the script to write the CSV file out to so I was not sure what was failing.

The EA's that I'm looking for are not very dynamic so a CSV that updates something between hourly to daily is all that is needed.   I don't really want to have to make the WAPI call real time for the report generation.

 

Is this possible to do?  Create a new, dynamic lookup csv via the built in tools that pulls the EA's from a data type and puts them on the reporting member like the lookup table for the Member object?

Could we get an example of that if it is possible? 

Re: Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5234     0

Well I see why I was having problems.    The option is there in the GUI to use custom python scripts for both custom lookups and for alert events, however, my support case was just updated with the fact that the option is not supported.....   Even though it is in the GUI and a core feature of vanilla Splunk.   Disappointing.


@DEvans wrote:

There have been several questions around getting to all the EA's for all the data types from the reporting tool.  ( I know, most of them are from me)  I think there are RFE's already open for this but it appears that the ability to pull them is already there, with some creative scripting.

I got close to getting at them by setting up a new lookup table of type external.  But I don't know python and I'm not sure what paths are available to the script to write the CSV file out to so I was not sure what was failing.

The EA's that I'm looking for are not very dynamic so a CSV that updates something between hourly to daily is all that is needed.   I don't really want to have to make the WAPI call real time for the report generation.

 

Is this possible to do?  Create a new, dynamic lookup csv via the built in tools that pulls the EA's from a data type and puts them on the reporting member like the lookup table for the Member object?

Could we get an example of that if it is possible? 


 

Re: Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5234     0

For those looking to inject data that is currently missing from the reporting tool automatically and dynamically I think I have found a way.  This would be data that is available via SNMP pulls or API \ WAPI calls, like CPU temp, model for all members, recursive DNS latency, EA's for Networks or other objects, etc.

 

Its not pretty, but once its setup, I think it will be relatively easy to maintain and add and remove data as needed.

 

I already had collected this "missing"  data on an external linux box that gathers it via SNMP and API calls for our own custom alerting and monitoring.

 

So from there I further parsed the data and created a syslog message out of it.   Then I forwarded that syslog message to a grid member that has syslog proxy enabled.  If configured correctly, it will take the syslog message and forward it on to the reporting member.   From there you can build your own summary tables, or view it in real time as a supplement to the data already in the reporting member.

Its not as easy or as nice as a dynamic lookup table, but the same functionality should be able to be built with minial overhead.  Expecially since you can do all the summary work you want to before pushing the syslog data on.

I just have a one line POC running now, I'll update with some examples as I get them going.

Re: Getting to all the Extensible Attributes in Reporting

Adviser
Posts: 118
5234     0

Very creative workaround Dave, thanks for sharing!

Re: Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5234     0

Here are some code snips from the first few examples .

 

The first group are modifications of the ib-graph bloxtools SNMP pull code from Infoblox.  It simply pulls the SNMP values via a perl script.  The same values that it updates in the RRD databases this code puts into a syslog message.    It looks to me like if you are still running bloxtools and ibgraph on a grid member that these logs would get picked up and sent on to the reporting tool automatically.  You would just need to setup the field extraction in splunk.   I'm running this off grid on a linux box so I have these syslog messages forwarded on (via the underlying OS syslog.conf)  to a grid member running syslog proxy so that they eventually make it on to the reporting member.

.......

   $oid = '.1.3.6.1.4.1.7779.3.1.1.2.1.4.0';
    $result = get_request($session, $oid);
    if (defined $result) {
        $hardware_type = $result;
        print F "model\t$result\n";
    }
.......

   $oid = '.1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.41';
    $result = get_request($session, $oid);
    if (defined $result) {
      if ($result=~ m/$re/is){
        $temp = $1 * 1.8 + 32;
        }
        print F "temperature\t$temp\n";
    }

.........


        openlog('bloxtools', 'cons,pid', 'user');
         syslog('info', 'FORREPORTERSys %s,%s,%s',$host,$temp,$hardware_type);
                    closelog();

 

This is a more traditional perl API syslog injection of all the Extensible Attributes from all of our networks.  I added just the lines to run through the EA's of each network to a larger data import export script for the grid so there is very little overhead to generating one syslog message for each network each day.  I'm starting to write some network usage reports around these EA's now.

@network_objs = $session->get(
        object => "Infoblox::DHCP::Network",
        network_view => "default",

        );

foreach my $networkObj (@network_objs) {

        my $infobloxnetwork = $networkObj->network;
       my $exts = $networkObj->extensible_attributes();
      my $extsyslog;
         foreach my $field ( keys %{$exts} ){
                $extsyslog .= $field . ":" . $$exts{$field} . ",";
        }

   openlog('bloxtools', 'cons,pid', 'user');
         syslog('info', 'FORREPORTERNet %s,%s',$infobloxnetwork,$extsyslog);
                    closelog();

}

Re: Getting to all the Extensible Attributes in Reporting

GRudd
Techie
Posts: 5
5234     0

I have just got the same message back from support. Is your RFE # RFE-7152 ? and if so We might have another vote.

 

 

Re: Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5234     0

RFE-3512  is the RFE that I opened sometime around 2013 for this functionality, so you can add your vote to that one.

It was actually opened on the previous generation of the reporter and I was told that it would be included in this re-write.  Some more EA's were included but Infoblox seems very resistant to open up all of them for some reason.  I am not sure why this is.  They have scripts that already pull some EA's for some objects, it would be minutes worth of codeing for them to simply allow those same processes to pull the rest of the EA's.

Re: Getting to all the Extensible Attributes in Reporting

Adviser
Posts: 118
5234     0

There is no specific resistance to getting EAs in reporting. It's currently part of a larger plan to enable reporting to pull from NIOS APIs (which would result in many other data types being available to reporting as well). I can't discuss the roadmap details in the forums, but I'm happy to chat with you directly about that feature if you like.

Re: Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5234     0

I had some time to work on this some more.    Matching network to network worked with my above solution but as soon as I needed to match an IP in the reporting tool to the syslog with its EA, the searches took forever looping thorugh a CIDR match.   I thought the solution was going to be back to a CSV but a CSV CIDR match is not an option as we do not have access to the .conf files to set the CSV field match type to CIDR.  

This is my current work around and I've disabled the Syslog import of the EA's for now.


Without a CIDR match option, the ability to get to the EA's in the networks and use them within the reporting is very difficult.

My current compromise is to export all the networks out of Infoblox.  Trim down the CSV to just the EA's I'm interested in, then generate a new column with just the first 3 octets of the IPV4 address and de duplicate that information.   I upload this "limited" list just of the first three IPV4 octets as a CSV lookup into the reporting tools.  I can then match the first 3 octets of an IP in the reporting tool against this CSV.   This gets me half way to the goal.   I can now take a random IP in the reporting tool and at least get back to the physical site it is at, usually. (as we rarely split a /24 between physical sites)   I still cannot get to the subnet specific information for anything less than a /24.   And for anything that is a /24 or bigger, it breaks for all the clients beyond the first /24 in a /22 in the subnet for example.   The larger subnets could be fixed with some code before the CSV is imported or with some Splunk logic that subtracts 1 from the 3rd octet until a match is found.   Not sure which is a better solution yet.

This does allow me to now get some "iplocation" mapping on our internal private network.   I can also include items like physical site, contact information, wired vs wireless in things like RPZ reports.  It’s getting there, it’s just a very long way to go to pull marginally correct data from within a tool that is the safe source for that data.


Re: Getting to all the Extensible Attributes in Reporting

Expert
Posts: 170
5234     0

Can you comment on any updates on getting to the EA's from within the reporting tool?


@RBarlow wrote:

There is no specific resistance to getting EAs in reporting. It's currently part of a larger plan to enable reporting to pull from NIOS APIs (which would result in many other data types being available to reporting as well). I can't discuss the roadmap details in the forums, but I'm happy to chat with you directly about that feature if you like.


 

Showing results for 
Search instead for 
Do you mean 

Recommended for You