Optimizing Bulk Retrieval of IPAM Data (IPv4/IPv6 Addresses, Leases, and Host Records)

vishalrajput
vishalrajput
in DNS, DHCP, IPAM (DDI)

Hi Team,

We are currently integrating with Infoblox DDI and need to retrieve a large volume of IPAM-related data. Below is the approximate scale of the data we are working with:

  • ~4.6 million IPAM IP addresses
  • ~500,000 DHCP leases
  • ~100,000 host records

We are using the following WAPI endpoints to fetch the data:

  • /<wapi version>/networkview
  • /<wapi version>/network
  • /<wapi version>/ipv4address
  • /<wapi version>/ipv6network
  • /<wapi version>/ipv6address
  • /<wapi version>/lease
  • /<wapi version>/record:host


Fetching data at this scale is taking a significant amount of time and is impacting overall performance. We are looking for ways to optimize or reduce the retrieval time.

Questions:

  1. Are there any recommended best practices for efficiently fetching large-scale IPAM data using WAPI?
  2. Is there any bulk or optimized endpoint available specifically for retrieving IPv4 and IPv6 addresses?
  3. Can pagination, batching, or specific query parameters (e.g., _return_fields, _max_results, etc.) significantly improve performance?


Any guidance, documentation references, or recommended approaches would be greatly appreciated.

Tagged:

Answers

  • Ingmar
    Ingmar

    1. Offload Read-Only API Calls to a Grid Master Candidate (GMC)

    This is probably the single biggest improvement you can make. Every GET request you're sending today likely hits your Grid Master, which is also responsible for replication, services, and write operations. That contention directly impacts your retrieval times.

    NIOS supports enabling read-only API access on a Grid Master Candidate (GMC). Once enabled, you can point all of your bulk read GET requests at the GMC IP instead of the GM, which dramatically reduces load on the Grid Master and improves overall API throughput.

    To enable it:

    • Grid Manager → Grid tab → Members → select the Grid Master Candidate → Edit
    • General tab → Basic tab → check Read Only API access
    • Save and restart services

    Key things to know:

    • The GMC only supports GET (read-only) operations — no POST or PUT. That's exactly what you need for bulk retrieval.
    • If you're making a very high volume of read-only calls, consider deploying a dedicated GMC purely for API workloads. You can have multiple GMCs in a grid, and they don't require any special licensing.

    2. Leverage CSV Export for Bulk Data Retrieval

    For datasets this large, individual paginated WAPI GET calls are not the most efficient approach. Instead, use the fileop CSV export function, which is purpose-built for bulk data extraction. CSV export runs server-side and generates a downloadable file — it avoids the overhead of JSON serialization on each paginated page and can be up to 3x faster than equivalent WAPI GET calls.

    The workflow is a 3-step process:

    Step 1 — Initiate the CSV export:

    curl -k -u <user>:<pass> -H 'content-type: application/json' \  -X POST "https://<gm-or-gmc>/wapi/v2.12/fileop?_function=csv_export" \  -d '{"_object": "record:host"}'

    This returns a token and a download url.

    Step 2 — Download the CSV file:

    curl -k -u <user>:<pass> -H 'content-type: application/force-download' \  "https://<gm-or-gmc>/http_direct_file_io/req_id-DOWNLOAD-XXXX/Hostrecords.csv" \  -o "Hostrecords.csv"

    Step 3 — Clean up the stored file:

    curl -k -u <user>:<pass> -H 'content-type: application/json' \  -X POST "https://<gm-or-gmc>/wapi/v2.12/fileop?_function=downloadcomplete" \  -d '{"token": "<token_from_step1>"}'

    You can use this for network, record:host, lease, and most other WAPI object types. You can also use the fileop?_function=read endpoint to export WAPI query results directly to a JSON file on the appliance.

    Hope this helps

Categories