Poor API performance when searching on multiple fields

[#1]

Hi folks,

We are using the Infoblox API to do some MAC authorization integration with Aruba's ClearPass product. In the process of doing this we have discovered tht API performance is extremely poor when trying to search on multiple fields. This appears to be because the implementation of the search API isn't smart enough to run a search, and then filter the results of that search based on the remaining criteria.  This leads to extremely high CPU usage on the grid member servicing API calls because the searches are internally matching many records that are ultimately discarded from the answer.  Here's an example which returns 0 results since the DE:AD:00:00:BE:EF MAC isn't in Infoblox.  This is against a virtual 1415 grid master candidate (NIOS 8.2.7) doing nothing but read-only API calls

First, we search only on the MAC address.  This is fast.

Executing API call /wapi/v2.7/record:host?mac=DE:AD:00:00:BE:EF
[]
real 0m0.149s
user 0m0.035s
sys 0m0.078s

Now, we try to match only on a MAC address in a specific IP range.  There are 1000+ host records in this IP range, but none of them have this MAC address.  The CPU gets crushed and the API call takes well over 4 seconds to return.

Executing API call /wapi/v2.7/record:host?mac=DE:AD:00:00:BE:EF&ipv4addr~=^172\.26\.
[]
real 0m4.433s
user 0m0.038s
sys 0m0.063s

Let's run the same API call, but against a different IP range with very few records in it.  Notice it's much faster, which tells me that internally Infoblox is looking through all of those host records, even when it could have thrown them all away based on the first MAC filter criteria.

Executing API call /wapi/v2.7/record:host?mac=DE:AD:00:00:BE:EF&ipv4addr~=^172\.27\.
[]
real 0m0.138s
user 0m0.033s
sys 0m0.069s

Is there any workaround or NIOS update on the horizon that will fix this?  Thanks.

[#2]

What are you attempting to accomplish with this Clearpass integration?  Is this search run every time a station authenticates / asociates to the AP?  Or is the intention to regularly synchronise a list of permitted MACs?

Have you also reviewed the integration offered by Clearpass to manage DHCP mac address filters? 

https://www.arubanetworks.com/techdocs/ClearPass/6.6/PolicyManager/Content/CPPM_UserGuide/Admin/Endp...

[#3]

Yes, this search gets run for every MAC authorization.  We looked at DHCP MAC address filters, but didn't end up using them because we wanted to be able to register DNS names with the MACs, so had to use host records.  So the MAC auth looks to see if the MAC is already registered in Infoblox to determine whether the user ends up in a captive portal role or a registered role.

[#4]

Thanks for the explanation. 

I did some testing with your specific searches.  While simultaneous searches do not block each other, having simultaneous searches will affect each others performance.  The performance will not scale, and this is the case for slower searches (with IP Address regex) and faster searches (without IP address regex).

I wouldn't reccomend using Infoblox API directly for authentication, authorization, or directory information such as for MAC address authentication.  The Infoblox product wasn't designed for this use case.

I would reccomend instead that the MAC Address information, along with associated hostnames or metadata, be stored internally within Clearpass or externally in a lightweight general-purpose database that can be indexed and optimized for fast read access, perhaps in an LDAP or SQL database.  The data could be popluated from Infoblox IPAM on a regular basis, perhaps as a nightly job.

[#5]

Thanks for the information.  I might not like it, but at least it gives us a firm direction to go in.