- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Specify _max_records for a query?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content
03-19-2018 05:15 PM
I'm getting an error where I'm getting an error that the result set is too large. While I could probably specify subsets of the data to get it within the current maximum of 1000 records, my concern is that I still might miss certain records. In other words, for example, if I broke it down by records beginning with numbers in the hostname, there are bound to be other records I'm missing that don't start with a hostname.
Given that, I think my only hope of getting the code to work is to specify a value for max_records (or maybe _max_records) to 5000 or below, and I'd have to work out some other fix. To that end, I have the following problem, where I'm running into the maximum value. It works for records less than 1000.
This is what I have in my script log (sanitized):
2018-03-16 23:41:50,183 - _make_request - DEBUG - https://grid-master-ip-here:443 "GET /wapi/v1.0/record:host?name~=.%2Adrc%7Cidrac%7Coob.%2Asomething.%2Asomething-else.%2AFQDN.%2Anet HTTP/1.1" 400 None
2018-03-16 23:41:50,188 - GetHostAndIP - DEBUG - Object r is <Response [400]>
2018-03-16 23:41:50,189 - GetHostAndIP - DEBUG - Printing srch_wrd record:host, Host Reference: {'Error': 'AdmConProtoError: Result set too large (> 1000)', 'code': 'Client.Ibap.Proto', 'text': 'Result set too large (> 1000)'}
Where the corresponding code is:
r = requests.get('https://' + str(grid_master_ip) + '/wapi/v1.0/' + str(srch_wrd),
verify=False,
params={ 'name~' : str(match_string) },
_max_results=5000,
#params=params,
auth=(str(acct),str(passwd)))
log.debug("Object r is " + str(r))
And where when I run the script, I get the following output (I get the same results whether I use max_results or _max_results; the only difference is the value in the unexpected keyword argument):
Traceback (most recent call last):
File "./inventory.py", line 271, in <module>
GetIPAMRecords(data_for, cfg)
File "./inventory.py", line 136, in GetIPAMRecords
GetHostAndIP(args, data_for, srch_wrd)
File "./inventory.py", line 99, in GetHostAndIP
auth=(str(acct),str(passwd)))
File "/usr/local/lib/python3.4/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.4/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
TypeError: request() got an unexpected keyword argument 'max_results'
Solved! Go to Solution.
Re: Specify _max_records for a query?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content
03-19-2018 06:53 PM
To return all the results you need to request a paged response, and then page through all the sets of responses.
Sorry I'm not a python guy, perhaps others can respond with a Python example. Or if you get it working in Python, feel free to share your example as a response.
Here's a paging example using curl, returning 100 objects per page.
curl -k -u admin:infoblox -X GET 'https://192.168.1.2/wapi/v2.2/record:a?_paging=1&_return_as_object=1&_max_results=100'
This returns:
{
"next_page_id": "789c5...",
"result": [
...
For the next call, use the paging id to requst the next page.
curl -k -u admin:infoblox -X GET 'https://192.168.1.2/wapi/v2.2/record:a?_return_as_object=1&_max_results=100&_paging=1&_page_id=789c5...'
This returns another next_page_id, which goes into the next page request. Repeat until next_page_id is empty,
which means all pages have been returned.
Re: Specify _max_records for a query?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content
03-20-2018 05:32 AM
The _max_results argument determines the maximum number of objects to be returned. If set to a negative number the appliance will return an error when the number of returned objects would exceed the setting. The default is -1000. If this is set to a positive number, the results will be truncated when necessary.
When you have a large number of records, you need to use paging.
- To start a paging request, the initial search request must have _paging and _return_as_object set to 1, and _max_results set to the desired page size (<=1000).
- The returned results object will contain the next_page_id field and the result field set to the first page of results. Note that the next_page_id field only contains URL-safe characters so it can be used as is and no quotation characters are required for subsequent requests.
- To get the subsequent results, you can re-send GET requests to the original object and set _page_id to the ID string returned in the previous page of results.
- Keep re-iterating till no next_page_id is returned, indicating that all records have been returned.
Here is a sample python code.
import requests requests.packages.urllib3.disable_warnings() url = "https://grid-master/wapi/v2.7/record:host" querystring = {"_max_results":"2","_paging":"1","_return_as_object":"1"} response = requests.request("GET", url, auth=('admin', 'infoblox'), params=querystring,verify=False) print(response.text) next_page_id = response.json()['next_page_id'] while next_page_id: query = {"_page_id":next_page_id} newresponse = requests.request("GET", url, auth=('admin', 'infoblox'), params=query,verify=False) print(newresponse.text) try: next_page_id = newresponse.json()['next_page_id'] except Exception: next_page_id = 0
Re: Specify _max_records for a query?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content
03-23-2018 11:00 PM
Thanks, that worked. Appreciate the help.