Specify _max_records for a query?

[#1]

I'm getting an error where I'm getting an error that the result set is too large.  While I could probably specify subsets of the data to get it within the current maximum of 1000 records, my concern is that I still might miss certain records.  In other words, for example, if I broke it down by records beginning with numbers in the hostname, there are bound to be other records I'm missing that don't start with a hostname. 

Given that, I think my only hope of getting the code to work is to specify a value for max_records (or maybe _max_records) to 5000 or below, and I'd have to work out some other fix.  To that end, I have the following problem, where I'm running into the maximum value.   It works for records less than 1000.

This is what I have in my script log (sanitized):

2018-03-16 23:41:50,183 - _make_request - DEBUG - https://grid-master-ip-here:443 "GET /wapi/v1.0/record:host?name~=.%2Adrc%7Cidrac%7Coob.%2Asomething.%2Asomething-else.%2AFQDN.%2Anet HTTP/1.1" 400 None

2018-03-16 23:41:50,188 - GetHostAndIP - DEBUG - Object r  is <Response [400]>

2018-03-16 23:41:50,189 - GetHostAndIP - DEBUG - Printing srch_wrd record:host, Host Reference: {'Error': 'AdmConProtoError: Result set too large (> 1000)', 'code': 'Client.Ibap.Proto', 'text': 'Result set too large (> 1000)'}

Where the corresponding code is:

    r = requests.get('https://' + str(grid_master_ip) + '/wapi/v1.0/' + str(srch_wrd),

                 verify=False,

                 params={ 'name~' : str(match_string) },

                 _max_results=5000,

                 #params=params,

                 auth=(str(acct),str(passwd)))

    log.debug("Object r  is " + str(r))

And where when I run the script, I get the following output (I get the same results whether I use max_results or _max_results; the only difference is the value in the unexpected keyword argument):

Traceback (most recent call last):

  File "./inventory.py", line 271, in <module>

    GetIPAMRecords(data_for, cfg)

  File "./inventory.py", line 136, in GetIPAMRecords

    GetHostAndIP(args, data_for, srch_wrd)

  File "./inventory.py", line 99, in GetHostAndIP

    auth=(str(acct),str(passwd)))

  File "/usr/local/lib/python3.4/site-packages/requests/api.py", line 72, in get

    return request('get', url, params=params, **kwargs)

  File "/usr/local/lib/python3.4/site-packages/requests/api.py", line 58, in request

    return session.request(method=method, url=url, **kwargs)

TypeError: request() got an unexpected keyword argument 'max_results'

[#2]

To return all the results you need to request a paged response, and then page through all the sets of responses.

Sorry I'm not a python guy, perhaps others can respond with a Python example.  Or if you get it working in Python, feel free to share your example as a response.

Here's a paging example using curl, returning 100 objects per page.

curl -k -u admin:infoblox -X GET 'https://192.168.1.2/wapi/v2.2/record:a?_paging=1&_return_as_object=1&_max_results=100'


This returns:

{
    "next_page_id": "789c5...",
    "result": [
    ...
    
For the next call, use the paging id to requst the next page.

curl -k -u admin:infoblox -X GET 'https://192.168.1.2/wapi/v2.2/record:a?_return_as_object=1&_max_results=100&_paging=1&_page_id=789c5...'
    
This returns another next_page_id, which goes into the next page request.  Repeat until next_page_id is empty,
which means all pages have been returned.

[#3 · Accepted solution]

The _max_results argument determines the maximum number of objects to be returned. If set to a negative number the appliance will return an error when the number of returned objects would exceed the setting. The default is -1000. If this is set to a positive number, the results will be truncated when necessary.

When you have a large number of records, you need to use paging.

1. To start a paging request, the initial search request must have _paging and _return_as_object set to 1, and _max_results set to the desired page size (<=1000).
2. The returned results object will contain the next_page_id field and the result field set to the first page of results. Note that the next_page_id field only contains URL-safe characters so it can be used as is and no quotation characters are required for subsequent requests.
3. To get the subsequent results, you can re-send GET requests to the original object and set _page_id to the ID string returned in the previous page of results.
4. Keep re-iterating till no next_page_id is returned, indicating that all records have been returned.

Here is a sample python code.

import requests
requests.packages.urllib3.disable_warnings()
url = "https://grid-master/wapi/v2.7/record:host"
querystring = {"_max_results":"2","_paging":"1","_return_as_object":"1"}
response = requests.request("GET", url, auth=('admin', 'infoblox'), params=querystring,verify=False)
print(response.text)
next_page_id = response.json()['next_page_id']
while next_page_id:
        query = {"_page_id":next_page_id}
        newresponse = requests.request("GET", url, auth=('admin', 'infoblox'), params=query,verify=False)
        print(newresponse.text)
        try:
                next_page_id = newresponse.json()['next_page_id']
        except Exception:
                next_page_id = 0

[#4]

Thanks, that worked.  Appreciate the help.