Security logs REST API not returning consistent data

jaybhaliya123
jaybhaliya123
in Automation

I am calling the Infoblox Security Events REST API (https://csp.infoblox.com/security-events/v1/security_events) from my virtual machine to fetch the Security Logs from Infoblox Portal, using the following query parameters: _limit, _offset, _order_by, start_time, end_time.

I am facing an issue where the API does not return consistent data across subsequent calls, even when the request parameters remain the same.

For Example, when I use:

  • _limit=25
  • _offset increased by 25 for each request
  • _order_by=timestamp desc
  • the same start_time and end_time

I observe the following behavior:

  • 1st page returns 25 events
  • 2nd page returns 25 events
  • 3rd page returns 30 events, even though _limit=25

Since the API does not return consistent records for any _limit, pagination appears inconsistent, making it difficult to reliably retrieve all security events.


Is there a recommended or alternative approach (for example, a different pagination or ordering strategy) to query the Security Logs from Infoblox portal and ensure consistent and complete data retrieval.

Answers

  • Ingmar
    Ingmar

    You’re running into a known limitation of offset/limit paging on a changing event stream in CSP.

    For DNS/Event and related CSP log APIs the public docs state:

    • Results “are not displayed in any particular order.”
    • To preserve row order for paging, results are cached, with a default cache expiry of 5 minutes; queries after that “may not have the same order of rows.”
    • Paging uses _limit and _offset, with a 50,000-row cap, and in some cases “more rows of data can be returned” than the requested _limit because of that cap and offset interaction.

    The Security Events (/security_events) REST API follows this same pagination model, so when you use _order_by=timestamp desc over a time range where new events keep arriving, and page with _offset = 0, 25, 50, … Each request is evaluated against a slightly different underlying result set and possibly a different cache snapshot. That’s why you see pages with more than _limit rows and inconsistent content across repeated calls with the “same” parameters.


    Practical workarounds

    Prefer server-driven pagination via _page_token:

    First call: include your filters and _limit, but no _offset or _page_token.
    Subsequent calls: pass only _page_token=<value> (same filters), and do not mix _offset.
    This lets the backend maintain a stable snapshot and cursor across pages.
    For bulk export or “get everything for this time range and analyze locally,” use the streaming endpoint:

    GET /security_events/download (same filters as /security_events), then page/process client-side.

    If you must stick with _offset/_limit:
    Use bounded, mostly historical time windows where events are no longer changing.
    Try to complete paging within the cache window so ordering doesn’t reshuffle mid-walk.
    Always sort on a deterministic field (timestamp + tiebreaker).

Categories