01-03-2017 01:23 PM
Using the IB-Graph tool to trouble shoot issues within our Infoblox grid, I’ve gotten very used to opening different “reports” in multiple browser tabs and having that data there updating in real time to quickly click back to as changes are made within the grid.
One example would be troubleshooting an anycast issue: I’ll open the DNS query rate on the different servers, maybe 3 to 10 servers, each on their own browser tab. I also might open a few more tabs with the same servers’ CPU utilization. As I make changes within the grid and or routing, I can quickly switch back to any member to see if the route \ traffic had had the expected result. The reports are all there with the server name listed in the browser tab. With IB-Graph running on a standalone Redhat box, I can have several admins looking at the real time reporting data of the grid, probably 20 or 30 real time graphs being updated every 2 minutes and the 2 CPU Redhat VM doesn’t have any issues.
I’ve been able to get the real time data with the upgrade to 7.3, and come up with some groups that help with this kind of trouble shooting using the EA’s but there are still some limitations. First, the servers I want to look at for a particular issue are not always in a group that I have defined. Adding the servers individually works but there is no way to save that group on the fly so that I can easily come back to it. I've stared opening a second browswer to keep a custom dashboard running and making my changes the first browser.(IE, Chrome, Firefox) In the past I've had issues with logging into the same grid mutiple times from the same browswer, I'm not sure if that is still an issue or not. Selecting multiple members to a report is fairly slow, it seems to take some time to populate the member list and I get type ahead issues where it can’t find a member if I type the name to soon. Finally, the real time data display seems to have a significant CPU hit on the reporting member. I understand that Splunk is a very different database that RRD(IB-Graph) but it seems to not perform well at this kind of real time data parsing.
I’m looking for something a quick as having a list of the DNS servers within the query rate dashboard, holding down CTRL and clicking on the ones I’m interested in. Each time I click, I get a new browser tab (or multiple graphs within the Splunk page) with the report / dashboard for that member. This data needs to stay running and current until I close the tabs. Is there a good way to open a bunch of dashboards within the Infoblox GUI and keep them running while troubleshooting an issue? Are others using the reporter for this kind of real time trouble shooting?