Importing a CSV file to Infoblox using webAPI?

[#1]

Hi All,

I'm trying to import a csv file to infoblox via webapi.

I'm still not sure about the follwoing two fields.

Few things I’m not sure..

• File_name : Is it just a name or complete source ? I would imagine it to be complete path of the on windows.
• Where would the file go once its uploaded?

command:

https://mydomain/wapi/v2.1/csvimporttask?action=START&file_name="my_path_on_windows\test.csv"

Error:

{ "Error": "AdmConProtoError: Field is not searchable: action", 
  "code": "Client.Ibap.Proto", 
  "text": "Field is not searchable: action"
}

please advise
Nidhi

[#2]

Import a CSV file

 POST /wapi/v1.5/fileop?_function=uploadinit
 Content-Type: application/json

 You will get back a URL for the upload, with a path of

 /http_direct_file_io/...  and a token for the upload session.  This is

 where you will be sending the file.

 Upload the CSV file to the URL.  The 'filedata' field is the actual

 contents of the file.  So you need to load this file into memory with

 whatever api client you are using.

   POST /http_direct_file_io/...

   {
     "name" : "my-file.csv",
     "filedata" : my-file.csv
   }

 Start the CSV import job

 You just use the original token, as this will have all the internal

 references to the filedata and it's location on the infoblox appliance.

 POST /wapi/v1.5/fileop?_function=csv_import
 Content-Type: application/json

    {
        "action":"START",
        "operation":"INSERT",
        "token": "eJydULFuwyAQ3e9...=",
        "on_error": "CONTINUE"
    }

[#3 · Accepted solution]

To add to what Geoff wrote, below is a complete Python script to initiate upload of a CSV file, upload the contents of the file, and start the CSV import task. I've tested this on Mac OS X, but not on Windows or Linux/Unix.

Frank

#!/usr/local/bin/python

"""
Import CSV data from a file.

This script uploads a CSV-formatted file (in Infoblox CSV format) into
the grid and starts a CSV import task.

To use this script change the 'url' variable to contain the domain
name or IP address of the grid master, and change the 'id' variable to
contain a userid with WAPI access to the grid master. (The script will
prompt for the corresponding password when run.)

If your grid master uses a TLS/SSL certificate from a commercial CA
then set the variable 'valid_cert' to True.

This script should work for NIOS 6.12 and later (WAPI 1.7 and later).
"""


# Import the required Python modules.
import requests
import json
import csv
import getpass
import sys
import os


# Set parameters to access the NIOS WAPI.
url = 'https://gm.example.com/wapi/v1.7/'
id = 'api'  # Userid with WAPI access
valid_cert = False  # True if GM uses certificate from commercial CA


# Helper functions.
def sanitize_filename(pathname):
    """Return sanitized filename without path information."""

    # Get the base filename without the directory path, convert dashes
    # to underscores, and get rid of other special characters.
    filename = ''
    for c in os.path.basename(pathname):
        if c == '-':
            c = '_'
        if c.isalnum() or c == '_' or c == '.':
            filename += c
    return filename


# Prompt for the API user password.
pw = getpass.getpass('Password for user ' + id + ': ')

# If running on Windows avoid error due to a self-signed cert.
if sys.platform.startswith('win') and not valid_cert:
    requests.packages.urllib3.disable_warnings()

# The CSV file we want to import (in the local filesystem).
csv_data = '/Users/jdoe/csv-data.csv'

# Initiate a file upload operation, providing a filename (with
# alphanumeric, underscore, or periods only) for the CSV job manager.
req_params = {'filename': sanitize_filename(csv_data)}
r = requests.post(url + 'fileop?_function=uploadinit',
                  params=req_params,
                  auth=(id, pw),
                  verify=valid_cert)
if r.status_code != requests.codes.ok:
    print r.text
    exit_msg = 'Error {} initiating upload: {}'
    sys.exit(exit_msg.format(r.status_code, r.reason))
results = r.json()

# Save the authentication cookie for use in subsequent requests.
ibapauth_cookie = r.cookies['ibapauth']

# Save the returned URL and token for subsequent requests.
upload_url = results['url']
upload_token = results['token']


# Upload the data in the CSV file.

# Specify a file handle for the file data to be uploaded.
req_files = {'filedata': open(csv_data,'r')}

# Specify the name of the file (not used?).
req_params = {'name': sanitize_filename(csv_data)}

# Use the ibapauth cookie to authenticate instead of userid/password.
req_cookies = {'ibapauth': ibapauth_cookie}

# Perform the actual upload. (NOTE: It does NOT return JSON results.)
r = requests.post(upload_url,
                  params=req_params,
                  files=req_files,
                  cookies=req_cookies,
                  verify=valid_cert)
if r.status_code != requests.codes.ok:
    print r.text
    exit_msg = 'Error {} uploading file: {}'
    sys.exit(exit_msg.format(r.status_code, r.reason))


# Initiate the actual import task.
req_params = {'token': upload_token,
              'doimport': True,
              'on_error': 'STOP',
              'operation': 'INSERT',
              'update_method': 'OVERRIDE'}
r = requests.post(url + 'fileop?_function=csv_import',
                  params=req_params,
                  cookies=req_cookies,
                  verify=valid_cert)
if r.status_code != requests.codes.ok:
    print r.text
    exit_msg = 'Error {} starting CSV import: {}'
    sys.exit(exit_msg.format(r.status_code, r.reason))
results = r.json()

# Record cvsimporttask object reference for possible future use.
csvimporttask = results['csv_import_task']['_ref']
print csvimporttask

[#4]

 Hi 

Firstly, thank you so much for the detailed reply of my questions.

I have bene looking into archive but didnt get anything so useful.

i have been trying the same method as you have mentioned in your example but not getting anywhere.

POST /https://xyz/wapi/v2.1/fileop?_function=uploadinit

May sound a dumb question but how can I use the POST directly via browser?

OR

Has to be used with curl only as mentioned in the refrrence doc?

Please advice

Nidhi

[#5]

 Hi 
 
Firstly, thank you so much for the detailed reply of my questions.
I have bene looking into archive but didnt get anything so useful.
 
i have been trying the same method as you have mentioned in your example but not getting anywhere.
POST /https://xyz/wapi/v2.1/fileop?_function=uploadinit
 
May sound a dumb question but how can I use the POST directly via browser?
OR
Has to be used with curl only as mentioned in the refrrence doc?
 
  
Please advice
Nidhi
 

[#6]

Thanks Frank and Geoff!!

Appreciate your responses and help!

Will let you know if still there is any concerns,Would really appreciate if you can respond to my question.

thats is how to use POST directly in a URL.

Thanks again!

Nidhi

[#7]

You can't use a POST in a url/browser, you need a client like curl or postman or a browser plugin

[#8]

I use the Chrome Advanced REST Client extension to do in-browser REST queries, including POST. 

https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo?hl=e...

It's pretty easy to use and gives you a nice interface when testing the REST API.

[#9]

A minor error in the script I posted: The line

req_files = {'filedata': open(csv_data,'r')}

should be

req_files = {'filedata': open(csv_data,'rb')}

 Per the documentation at python-requests.org, you should explicitly open the file as binary. On Mac OS X this shouldn't make a difference, but it might on other platforms, including Windows.

Frank

[#10]

Thanks for the update Frank!

-Nidhi

[#11]

Thank you so much!

Appreciate your help for the in-browser REST quesries.

[#12]

Thanks for the clarification.

-Nidhi

[#13]

Hello Frank,

You have posted a very detailed solution for CSV import to Infoblox web API and I have one quick question in the same.  

In your solution you directly assigned the complete path of the csv file to import to variable "csv_data". Can we pass the complete data of the file into csv_data instead of passing the fath of the file ?

Looking forward for your response.

Thanks & Regards,

Mukul

[#14]

You cannot simply add data to that variable. The CSV import requires you to upload a file to the URL provided by the uploadinit funtion.

[#15]

To upload the CSV file to the URL provided by the uploadinit function, do we need to pass the local path of the CSV file on the local system (where it exist) or the actual CSV file data ? 

[#16]

I tried the above python code which died (due to user error). When I rerun I get:

{ "Error": "AdmConProtoError: Too many CSV import tasks are in the UPLOADED state. Only 1 UPLOADED tasks are allowed for each user.", 
  "code": "Client.Ibap.Proto", 
  "text": "Too many CSV import tasks are in the UPLOADED state. Only 1 UPLOADED tasks are allowed for each user."
}

Obiously I need to cancel the upload task so two questions: 1)

how do I get the task id for the upload task and 2) how do I cancel so that I can re-upload?

Thanks

[#17]

Sorry to bump such an old thread, but does anyone know if there's a way to check on the CSVImport status?

The reason being is, there's a way to pull down the CSV_ERROR file, but you have to wait until the CSVImport job is finished.

The Perl module has this method available to it: https://ipam.illinois.edu/api/doc/Infoblox/Grid/CSVImportStatus.html#description

I have the code that runs after the CSVImportTask is done to pull down the error file, but I can't seem to figure out how to code the proper waiting time after the _function=csv_import runs in the python script provided as the example here.

While dropping a time.sleep(N) in the script after the csvimporttask variable is possible, there are times when a larger (2000+ line) CSV import can take anywhere from 3-5 minutes to complete, whereas a smaller import may only take a few seconds and I'd prefer not to put a ~5 minute sleep for every single csvimport task if possible.

Any ideas how to achieve this?

[#18]

Hi,

When you run the csv_import function it returns an output with the reference of the CSV import task, as shown below

WAPI Call:
curl -k -u admin:infoblox -H 'content-type: application/json' -X POST "https://grid-master/wapi/v2.7/fileop?_function=csv_import" -d '{"operation":"UPDATE","action":"START","on_error":"CONTINUE","update_method":"MERGE","separator":"TAB","token":"eJylUMtOwzAQvPtH2kuTrNu8uBWVSkiooBbOq8R2ykqJbWwHtX+PXQRHLhxs7e7M7GOEMPaKTp1Z\n/ITRPrhZBOOYBbYUpAfTj+aSGa3SC1erPNt1ocOjGpjlTCD2M42BNCKTJAKza7aUdsNOC3Wx5K4Y\naFILZku2h3Jdl7DZNJA1RVtxzvxpMbsxolXkv4dg/V2eQ5FBW2W84BmHPFVRUtwu4ECjQjK5Ux9I\ncvX28vS83a2KlkNRceDQFDXUdU6TNe6bHVvXaQrJGDVxyN/SSGoTXcYTYwxFVOQ++tGdVR4m+/9t\nAH76o9LCSNLnVOVx0P3jIYXrX8Jk5E2SzNxtX7d4fNinvGQ+HC1UyWioIziQGqXHYFCYyXbupmrY\nIR3edxZJ2zngp3KejE5YG7E++wIsS5yX\n"}'

Output:
{
    "csv_import_task": {
        "_ref": "csvimporttask/b25lLmNzdl9pbXBvcnRfdGFzayQ1:5",
        "admin_name": "admin",
        "file_name": "import_file",
        "file_size": 215,
        "import_id": 5,
        "lines_failed": 0,
        "lines_processed": 0,
        "lines_warning": 0,
        "on_error": "CONTINUE",
        "operation": "UPDATE",
        "separator": "TAB",
        "start_time": 1537512228,
        "status": "PENDING",
        "update_method": "MERGE"
    }
}

Using the CSV import task reference, you can query and fetch the status of your task

WAPI Call:
curl -k -u admin:infoblox -X GET "https://grid-master/wapi/v2.7/csvimporttask/b25lLmNzdl9pbXBvcnRfdGFzayQ1:5"

Output:
{
    "_ref": "csvimporttask/b25lLmNzdl9pbXBvcnRfdGFzayQ1:5",
    "admin_name": "admin",
    "end_time": 1537512233,
    "file_name": "import_file",
    "file_size": 215,
    "import_id": 5,
    "lines_failed": 0,
    "lines_processed": 3,
    "lines_warning": 0,
    "on_error": "CONTINUE",
    "operation": "UPDATE",
    "separator": "TAB",
    "start_time": 1537512233,
    "status": "COMPLETED",
    "update_method": "MERGE"
}

You can even just fetch the status and keep polling it till it reaches the COMPLETED or FAILED state.

WAPI Call:
curl -k -u admin:infoblox -X GET "https://grid-master/wapi/v2.7/csvimporttask/b25lLmNzdl9pbXBvcnRfdGFzayQ1:5?_return_fields=status"

Output:
{
    "_ref": "csvimporttask/b25lLmNzdl9pbXBvcnRfdGFzayQ1:5",
    "status": "COMPLETED"
}

Hope this helps,

Krishna

[#19]

Thanks! That was easy enough.. If anyone is interested in the downloading of the csv_error file, please let me know and I'll post the additional code.

[#20]

Hi,

I'm trying to import Name server group that ' exported previously from GUI.

The error is the following:

{
"Error": "AdmConProtoError: Function csv_import\" is not valid for this object",
"code": "Client.Ibap.Proto",
"text": "Function csv_import\" is not valid for this object"
}

It seems that is not possible to import csv that contains  some kind of object (member, name server group ) 

Do you know is it correct?

Regards

Mirko

[#21]

Hello,

Thanks a lot for all yours sharing.

Is it possible to export objects from a view/zone, and to have it directly in the CSV format, ready to be imported ?

I would like perform an export from a specific view/zone, and import to a specific view/zone.

It will be nice if you have an idea For the moment, i'm only able to export it in a Json format, not CSV.

Thanks in advance, and have a nice day.

[#22]

Hi,

You can export all the host records within a view/zone into a CSV using the following set of WAPI calls.

Initiate a CSV export of all the host records in the zone 'demo.com' within the network view 'test':
curl -k -u admin:infoblox -H 'content-type: application/json' -X POST "https://grid-master/wapi/v2.9/fileop?_function=csv_export" -d '{"_object": "record:host","view":"default.test"}'

Sample output:
{
    "token": "eJytUEFuwyAQvPOR5BLb62A77i2VG7VSlUhJpR5XNuAUyQYKJEp+36VSe+2lB9Cys8PsjBDW3dGr\nM6NLWBOiv4hoPXPAlkKb0Q6TvWXWqHTi3anAuj72eFQjcyUTiMNFT1EbRCa1iMyt2VI6zk4LdXPa\n3zHqWS2Yq9gOKl40bcuhyXgFZV0BC6fFxU8E10T4iNGFhzyHIoO2zsqizErIUxelpvUijnpSqG3u\n1SdqueoO7/vXw7ZbQVG2RV0V0HC+WXOeP1tyQoa8DJkIVxJokpaWVG1I6q8PaKxNBEleqYaCOHmg\nYPqzyuPs/msrgB8VVEZYqc05dUuSe3zZp3L9OzBbmYKElG23fdvi8WmX3hUL8eigTrlDQ+Co1SQD\nRovCzq7336wN26cAht6hNu4S8ap80NYkrCVsyL4AsxyhcQ==\n",
    "url": "https://10.196.202.21/http_direct_file_io/req_id-DOWNLOAD-1029065017448344/Hostrecords.csv"
}

Download the CSV file:
curl -k -u admin:infoblox -H 'content-type: application/force-download' "https://grid-master/http_direct_file_io/req_id-DOWNLOAD-1029065017448344/Hostrecords.csv" -o "Hostrecords.csv"

Remove the stored file using the token:
curl -k -u admin:infoblox -H 'content-type: application/json' -X POST "https://grid-master/wapi/v2.9/fileop?_function=downloadcomplete" -d '{"token" : "eJytUEFuwyAQvPOR5BLb62A77i2VG7VSlUhJpR5XNuAUyQYKJEp+36VSe+2lB9Cys8PsjBDW3dGr\nM6NLWBOiv4hoPXPAlkKb0Q6TvWXWqHTi3anAuj72eFQjcyUTiMNFT1EbRCa1iMyt2VI6zk4LdXPa\n3zHqWS2Yq9gOKl40bcuhyXgFZV0BC6fFxU8E10T4iNGFhzyHIoO2zsqizErIUxelpvUijnpSqG3u\n1SdqueoO7/vXw7ZbQVG2RV0V0HC+WXOeP1tyQoa8DJkIVxJokpaWVG1I6q8PaKxNBEleqYaCOHmg\nYPqzyuPs/msrgB8VVEZYqc05dUuSe3zZp3L9OzBbmYKElG23fdvi8WmX3hUL8eigTrlDQ+Co1SQD\nRovCzq7336wN26cAht6hNu4S8ap80NYkrCVsyL4AsxyhcQ==\n"}'

Hope you find this useful,

Krishna

[#23]

Hi Krishna

Thanks for your answer, yes, it will helpful for my use case.

I have to now translate it in Python, like on this thread:
https://community.infoblox.com/t5/API-Integration/Importing-a-CSV-file-to-Infoblox-using-webAPI/m-p/...

I use this scrip to perform the import, it works nicely.

Don't hesitate to share if you already did it

Have a nice day,

Antoine

[#24]

Hey @jeffwarn, could you please post the code for downloading the error file?

Also, my status always shows as COMPLETED even if there is an error. Any ideas what's going on with that?

[#25]

Hi ghore,

Im pretty confused .so i would step by step how i implemented. Im using Postman :

1. POST : https://ipman/wapi/v2.6.1/fileop?_function=uploadinit

Output :

{"token": *******

   "url" :"https://ipaddress/http_direct_file_io/req_id-UPLOAD-670111/import_file"

}

2. POST : https://ipaddress/http_direct_file_io/req_id-UPLOAD-670111/import_file

body :{ "name" : "C:\Users\test.csv", "filedata" : C:\Users\test.csv }
shows errors. please let me know what am I supposed to give in body

3. Please check my csv format
#IP-Adresse Netzwerk/CIDR MAC-Adresse Hostname Domain ID-Nr. Geräteart Gerätetyp Standort Netzdose Verteiler Nutzer Struktureinheit Tel.-Nr. Applikationen Bemerkungen
#149.203.121.179 149.203.121.0/24 00:50:C2:07:2D:29 glt-68462 imed.uni-magdeburg.de 68462 Haustechnik Neuberger PMC H1/ 1/2.3/S/17 ev1-s1/s/p Fähse Thomas G4.2.2, Betriebstechnik 13441, 13442 glt Test


thanks

[#26]

I am observing the same behavior that tgraham2 mentions above. I notice after the first step, the URL that is provided back to me has path

/http_direct_file_io/req_id-UPLOAD-xxxx/

where xxxx is a four digit number that increments each time I make the call (currently the number is at 1034).

I use the URL and token in the second call as described above and get an empty 200 response. Then, upon running the third request described (function csv_import), I get a 400 status code and the response body is:

{ "Error": "AdmConProtoError: Too many CSV import tasks are in the UPLOADED state. Only 1 UPLOADED tasks are allowed for each user.", 
  "code": "Client.Ibap.Proto", 
  "text": "Too many CSV import tasks are in the UPLOADED state. Only 1 UPLOADED tasks are allowed for each user."
}

I am unable as of yet to find in the documentation how to clear this apparent queue of UPLOADED state tasks so that this WAPI CSV upload can be successful. Any direction on how to resolve this issue would be greatly appreciated.

[#27]

When I cut and paste FHecker's python code and substitute the address of my grid master / username, I get HTTP 503 errors with this POST...

# Perform the actual upload. (NOTE: It does NOT return JSON results.)
r = requests.post(upload_url, 
        params=req_params, 
        files=req_files, 
        cookies=req_cookies, 
        verify=valid_cert)

I'm running Python 2.7 and NIOS 8.2.7-372540.  Any insight into what the problem could be is appreciated.

[#28]

Hi Techies,

is it possible to use "next available IP address " in CSV import, if so how to do that ?

Because everytime I have to give IP address manually in the CSV file!!

thanks

[#29]

How do I export the current configuration of the Infoblox using GUI or WAPI can any one help on this ?

[#30]

Does this meet your needs?

https://community.infoblox.com/t5/API-Integration/Backup-and-restore-via-WAPI/m-p/14901