Leverage Idempotent, Declarative Profiles with the NSX-ALB (Avi) REST API

Idempotence and Declarative Methods - not just buzzwords

Idempotence

Coined by Benjamin Peirce, this term indicates that a mathematical operation will produce a consistent result, even with repetition.

Idempotence is much more complicated subject in mathematics and computer science. IT and DevOps use a simplified version of this concept, commonly leveraging flow logic instead of Masters-level Algebra.

Typically, an idempotent function in DevOps-land adds a few other requirements to the mix:

• If a change is introduced, convergence (the act of making an object match what the consumer asked for) should be non-invasive and safe
• It's the responsibility of the consumer to adequately test this
• Provide a "What If?" function of some kind, indicating how far off from desired state a system is
• It's the responsibility of the consumer to adequately test this. Idempotent systems should provide a method for indicating what will change, but won't always provide a statement of impact

Ansible's modules are a good example of idempotent functions, but Ansible doesn't require that everything be idempotent. Some good examples exist of methods that cannot be idempotent, re-defining it to add the "do no harm" requirement:

• Restarting a service
• Deleting and re-adding a file

As a result, many contributed modules are not pressured to be idempotent when they should be. It's the responsibility of the consumer (probably you) to verify things don't cause harmful change.

Declarative Methods

Lori MacVittie (F5 Networks) provides an excellent detailed explanation of Declarative Models here:

https://www.f5.com/company/blog/why-is-a-declarative-model-important-for-netops-automation

Declarative Methods provide a system interface that can be leveraged by a non-Expert, by allowing a consumer to specify what the consumer wants instead of how to build it (an Imperative method).

This is a huge issue in the IT industry in general, because we (incorrectly) conflate rote memorization of individual imperative methods with capability. In the future, the IT industry will be forced to transform away from this highly negative cultural pattern.

We as professionals need to solve two major problems to assist in this transition:

• Find a way to somehow teach fundamental concepts without imperative methods
• Teach others to value the ability to effectively define what they desire in a complete and comprehensive way

If you've ever been frustrated by an IT support ticket that has some specific steps and a completely vague definition of success, declarative methods are for you. The single most important aspect of declarative methods is that  the user/consumer's intent is captured in a complete and comprehensive way. If a user fails to define their intent in modern systems like Kubernetes, the service will fail to build. In my experience, problem #1 feeds into problem #2, and some people just think they're being helpful by requesting imperative things.

Obviously the IT industry won't accept that a computer system is allowed to deny them if they failed to define everything they need to. This is where expertise comes in.

How we can use it in DevOps

Here's the good news - designing and providing systems to provide idempotent, declarative methods of cyclical convergence isn't really an enterprise engineer's responsibility. Network Equipment Providers (NEP) and systems vendors like VMware are on the hook for that part. We can interact with provided functions leveraging some relatively simple flow logic:

Well-designed APIs (NSX ALB and NSX-T Data Center are good examples) provide a declarative method, ideally versioned (minor gripe with NSX ALB here, the message body contains the version and may be vestigial), and all we have to do is execute and test.

In a previous post, I covered that implementing reliability is the consumer's responsibility, transforming a systems engineer's role into one of testing, ensuring quality and alignment of vision as opposed to taking on all of the complex methods ourselves

TL;DR Example Time, managing Application Profiles as Code (IaC)

Let's start by preparing NSX ALB(Avi) for API access. The REST client I'm using uses HTTP Basic Authentication, so it must be enabled - the following setting is under System -> Settings -> Access Settings:

Note: In a production deployment other methods like JWT ought to be used.

The best place to begin here with a given target is to consult the API documentation, provided here: https://avinetworks.com/docs/21.1/api-guide/ApplicationProfile/index.html

When reviewing the documentation VMware provides, declarative CRUD methods are all provided (GET, PUT, PATCH, DELETE) for an individual application profile. Let's leverage the workflow above as code (Python 3)

 1# Recursively converge application profiles  
 2def converge_app_profile(app_profile_dict):  
 3    # First, grab a copy of the existing application profile  
 4    before_app_profile = json.loads(  
 5        cogitation_interface.namshub(  
 6            "get_app_profile", namshub_variables={"id": app_profile_dict["uuid"]}  
 7        )  
 8    )  
 9  
10    # Fastest and cheapest compare operation first  
11    if not app_profile_dict["profile"] == before_app_profile:  
12        # Build a deep difference of the two dictionaries, removing attributes that are not part of the profile, but the API generates  
13        diff_app_profile = deepdiff.DeepDiff(  
14            before_app_profile,  
15            app_profile_dict["profile"],  
16            exclude_paths=[  
17                "root['uuid']",  
18                "root['url']",  
19                "root['uuid']",  
20                "root['_last_modified']",  
21                "root['tenant_ref']",  
22            ],  
23        )  
24  
25        # If there are differences, try to fix them at least 3 times  
26        if len(diff_app_profile) > 0 and app_profile_dict["retries"] < 3:  
27            print("Difference between dictionaries found: " + str(diff_app_profile))  
28            print(  
29                "Converging "  
30                + app_profile_dict["profile"]["name"]  
31                + " attempt # "  
32                + str(app_profile_dict["retries"] + 1)  
33            )  
34            # Increment retry counter  
35            app_profile_dict["retries"] += 1  
36            # Then perform Update verb on profile  
37            cogitation_interface.namshub(  
38                "update_app_profile",  
39                namshub_payload=app_profile_dict["profile"],  
40                namshub_variables={"id": app_profile_dict["uuid"]},  
41            )  
42            # Perform recursion  
43            converge_app_profile(app_profile_dict)  
44        else:  
45            return before_app_profile  

Idempotency is easy to achieve, we leverage the deepdiff library to process data handled by a READ action, and then execute a re-apply action if it doesn't match. This method will allow me to just mash the execute key until I feel good with the results. I've included a retry counter as well to prevent looping.

That's actually all there is to it - this method can be combined with Semantically Versioned Profiles. I have provided public examples on how to execute that in the source code: https://github.com/ngschmidt/python-restify/tree/main/nsx-alb