Custom Logic for IP Allocation [ Edited ]

[#1]

Hello,

We have a custom infoblox plugin for vRA that we have developed which uses internal business logic when selecting an IP address.  We assign IP addresses based on different criteria depending on the details of the VM request.  For example, we have builds that...

• Assign random available IP in subnet
• Assign next available IP from a subset of IP's within the subnet.  We call these IP Pools, and they are defined by putting a host record with the name "reserved" against each IP in the pool, and then assigning a poolName extensible attribute to the host record so that we can search for IP's in the pool by using the poolName EA as our search criteria.
• Assign reserved IP matching the name of the server being built.

There are some others as well, but those are the main 3 types of assignments we make.  Our scripts currently do the following...

1. Query the Infoblox API to find all addresses matching the required search criteria
2. Select one of the returned IP's at random
3. Add a host record for the VM to that IP

The problem is that if someone submits many simultaneous requests then we sometimes see that the same IP is used for multiple builds since there is no way to make vRA run the IP Allocation process in series.  As a result, if there are multiple allocations running at the exact same time then they will sometimes see the same IP as available, which then causes one of the VM builds to fail due to the IP conflict.  I've tried injecting some random sleep intervals in our IP allocation code, and it has improved things, but its still not perfect and I do still observe this happening from time to time.

In researching this, I have found that there is a way to offload IP selection to Infoblox using the "_next_available_ip" function, for example...

POST  https://infoblox.net.local/wapi/v2.6/record:host

Body:  

{
    "name": "carlosTestHostRecord",
    "ipv4addrs": [
        {
            "ipv4addr": {
                "_object_function": "next_available_ip",
                "_result_field": "ips",
                "_object": "network",
                "_object_parameters": {
                    "network": "10.1.2.0/24"
                }
            }
        }
    ],
    "configure_for_dns": false
}

I have tested this, and this works great for scenarios where we just want the next available IP in a subnet with no additional criteria.  However, this does not solve the need for selecting an IP from an IP Pool within the subnet.

So I have the following questions...

• Is it possible to add more search params to the above operation?  Specifically, we'd want to search for IP's in the specified subnet, with a host record named "reserved", which also have a specific value for the poolName EA.
• Can we write our own functions that we can consume via the API, so that we can insert our own business logic?
• If we can write custom functions, then how would we pass parameters to those functions?
• Is there perhaps a better way to segregate multiple blocks of IP's within a single subnet aside from the host record with extensible attribute option that we are already using?

Many thanks.

Carlos

[#2]

Hello.  I'm surprised noone has attempted a stab at this.  My thoughts are to use the blocking feature of the Extensibility Subscription you created.  "Blockable Event Topics allow Blocking Subscriptions. Marking a subscription as Blocking will block any other subscriptions of this Blockable Event Topic and Condition, until this Action/workflow has been finished." 

My question back to you, why are you not using the Infoblox provided vRA/vRO plugins available from the Infoblox support site downloads? 

[#3]

Thanks for the response.  If you are referring to the blocking event subscription in vRA, then that is only applicable to the context of the deployment that the event has fired for.  So, if I were to assign many workflows to the same event topic, then I can use blocking along with priority to control the order in which they run for each deployment.  But in this case, there are many simultaneous deployments, so using event blocking would not prevent multiple IP allocation workflows from running simultaneously.

Unfortunately this is all besides the point though since the IPAM providers in vRA are not executed via event topics like this to begin with.  They are executed via ABX and there is no subscription or anything else to configure priority or blocking for when you setup an IPAM provider.

Finally, why are we using a custom IPAM provider for Infoblox instead of the one provided by Infoblox?  We are doing this so that we can include custom business logic IP allocation logic.  We have 3 primary methods we use for IP allocation:

• Some deployments just need the next available IP - easy, and supported out of the box.
• Some deployments need the next IP whose PoolName extensible attribute matches the value of the associated input on the vRA request.  Not supported in Infoblox's implementation.
• Some deployments need an address that has been reserved in Infoblox, whose reservation name matches the name of the VM being deployed.  Not supported in Infoblox's implementation.

So, we clearly need custom code for our internal business logic to assist in IP allocation, which brings me back to the original question.  Is there a way to customize the "next_available_ip" _object_function in Infoblox?  If there was, then we could put all of our business logic there and Infoblox would ensure that only one IP was allocated at a time, eliminating the opportunity for multiple deployments to get the same IP.

[#4]

Thank you for the response and that makes sense. 

Each of the deployment types you listed sound as if they would each come from a different design blueprint.   My first thought is that you could have tag constrained network profiles for each of those types.  That will work for your first type.  For the second type, in the plugin for vRA7, I was able to declare a dhcp pool and select next available from that, but that doesn't appear available in the plugin any more.  And for the 3rd, definitely won't work if the 2nd type won't work.

So you could isolate those deployment types through three different action ABX subscriptions and use blocking to prevent overrun of IP requests.   When creating the subscribtion, specify "Filter Events in Topic"

"Create a javascript syntax boolean expression using the 'event' object and its properties. Use 'event.data' to access the event's payload (data) according to the specified topic schema, or any of the event's header properties: sourceType, sourceIdentity, timeStamp, eventType, eventTopicId, correlationType, correlationId, description, targetType, targetId, userName, orgId." 

In your case filter on requesting blueprint / catalog item type.

In theory, the vRO workflows might assist, but as I'm learning more and more, there isn't a lot of support for the Infoblox provided plugins an writing your own appears to be the requirement.

[#5]

It's actually all one blueprint in vRA, not separate.  As for the ABX, the IPAM plugins use the ABX engine under the hood to run the code, but there is no subscription object to configure anything on.  The IPAM plugins "hook into" whatever internal events vRA has for IP allocation.

Maybe you are suggesting to use pure ABX for IP allocation instead of an IPAM plugin, but that would still not fix the original issue of multiple IP allocation operations happening simultaneously.

I COULD split this code out into full-fledged vRO workflows and then use vRO's internal LockSystem methods to ensure only one IPAM operation runs at a time across the entire vRO instance, but that would require that we port the entire solution into workflows/actions instead.  Everything else we do in vRO is JavaScript, while the IPAM plugin is written in Python.  vRO 8.x does support Python in Actions/Workflows, but if we ported this code over it would be completely different from the rest of the code base, which isn't ideal.  There are also dependencies (both internal and external) pulled in by the IPAM implementation, so I am not sure how that would work in vRO at all.

It would be nice if there was a system-wide locking mechanism available in ABX which I could leverage just like I can with LockSystem in vRO, but there isn't.  I can't remember the last time I saw this issue either, so while I would like a more bullet-proof solution for preventing simultaneous IP allocations than what we have today, it's not worth completely rewriting the entire IPAM implementation in vRO.

...and while I was writing this, I just had an idea that I could have our IP allocation code in the IPAM plugin hit the vRO API for the sole purpose of running an action to check/wait for a Lock to become available in vRO's LockSystem.  This is probably the best I can hope for since it appears there is no way to do any of this in Infoblox (which would've been nice).

[#6]

So you are using python code in your actions?  Of course you might run into the 3200 character limit for that which is a fustration in its own.

So when you create the subscription based on event topic to run your python action, select "blocking"

[#7]

We are not using actions at all.  We have a custom IPAM plugin, which is written in Python.  And as I mentioned, IPAM plugin code is actually run inside ABX behind the scenes.  IPAM code running in ABX isn't something we did - this is by design from VMware.

There are no subscriptions for any of this, and even if there were, setting a subscription to blocking is a deployment-level setting, not system-wide.  So, this would not prevent two IP allocation operations from happening at the same time and getting the same address.

[#8]

Sounds like you need to go back to the creator of the plugin and request they update the plugin with blocking enabled. 

Or if you have access to the source.  That or write your own. 

[#9]

You can download the sample IPAM plugin from https://developer.vmware.com/web/sdk/vmware-vrealize-automation-third-party-ipam-sdk/.  This is what we used as a starting point for our implementation.

If you download the official Infoblox IPAM plugin and extract it, you'll see that its actually based on the same framework as the sample above.

[#10]

We did write our own.  I've been very clear about that.  We cannot build in blocking as you continue to suggest, because IPAM plugins don't use event subscriptions to begin with, and as I have mentioned a couple times now, blocking events are not system-wide anyway, so they would not solve the original problem of multiple IP allocations running in parallel.

[#11]

I don't think I read this suggestion in the prior comments, but if it's there forgive me.

Also, assuming you're using Orchestrator... You could use orchestrator's locking mechanism and wrap your IP request code in a request lock -> get new IP -> release lock sequence.

Effectively, wrap calls to get IPs from Infoblox with locking which will be system wide.

Something like this:

LockingSystem.lock(lockid, owner)
LockingSystem.lockAndWait(lockid, owner);

// Go get your IP from Infoblox

LockingSystem.unlock(lockid, owner);
LockingSystem.unlockAll();

Of course, you'll want to add more robustness to it such as a try .... finally block to ensure the lock is removed on failiures, etc.

Hope this helps.

[#12]

Thanks for the response. I did mention the LockSystem, and we do use that for other things, however IPAM provider plugins run in ABX, not vRO so this is not available to them.