Why Automate? Reliability Approaches with the VMware NSX-T API

Nov 24, 2021 · 11 min read · Home Lab Continuous Delivery JSON NSX-T Network Automation ·

Why should an infrastructure engineer leverage REST APIs?

I'm sure most IT workers have at least heard of REST APIs, or heard a sales pitch where a vendor insists that while a requested functionality doesn't exist, you could build it yourself by "using the API".

Or, participate in discussions where people seemed to try and offer you a copy of The DevOps Handbook or The Unicorn Project.

They're right, but software development and deployment methods have completely different guiding values than infrastructure management. Speed of delivery is almost completely worthless with infrastructure, where downtime is typically the only metric that infrastructure is evaluated on.

We need to transform the industry.

The industry has proven that "value-added infrastructure" is a thing that people want, otherwise, services like Amazon AWS, Azure, Lumen would not be profitable. Our biggest barrier to success right now is the perceptions around reliability because there clearly is demand for what we'd call abstraction of infrastructure. We can't move as slow as we used to, but we can't make mistakes either.

Stuck between a rock and a hard place?

I have some good news - everybody's just figuring this out as they go, and you don't have to start by replacing all of your day-to-day tasks with Ansible playbooks. Let's use automation tools to ensure Quality First, Speed Second. Machines excel at comparison operators, allowing an infrastructure administrator to test every possible aspect of infrastructure when executing a change. Here are some examples where I've personally seen a need for automation:

Large-scale routing changes: if 1,000 routes successfully migrate, and a handful of routes fail, manual checks tend to depend overly (unfairly) on the operator to eyeball the entire lot
Check: Before and after routes, export a difference
Check: All dynamic routing peers, export a difference
Reverse the process if anything fails
Certificate renewals
Check: If certificate exists
Check: If the certificate was uploaded
Check: If the certificate has a valid CA chain
Check: If the certificate was successfully installed
Reverse the process if anything fails
Adding a new VLAN or VNI to a fabric
Check: VLAN Spanning-Tree topology, export a difference
Check: EVPN AFI Peers, export a difference
Check: MAC Address Table, export a difference
Reverse the process if anything fails

The neat thing about this capability is the configuration reversal - API calls are incredibly easy to process in common programming languages (particularly compared to expect) and take fractions of a second to run - so if a tested process (it's easy to test, too!) does fail, reversion is straightforward. Let's cover the REST methods before exploring the deeper stuff like gNMI or YANG.

Anatomy of a REST call

When implementing a REST API call, a client request will have several key components:

Headers: Important meta-data about your request go here, the server should adhere to any specification provided in HTTP headers. If you're building API code or otherwise, I'd recommend just setting up a standard when reviewing the list of supported fields. Examples:
Authentication Attributes
{'content-type': 'application/xml'}
{'content-type': 'application/json'}
{'Accept-Encoding': 'application/json'}
{'Cache-Control': 'no-cache'}
Resource: This is specified by the Uniform Resource Indicator, the URL component after the system is specified. A resource is the "what" of a RESTful interaction.
Body: Free-form optional text, this component provides a payload for the API call. It's important to make sure that the server actually wants it!
Web Application Firewalls (WAF) can inspect header, verb, and body to determine if an API call is safe and proper.

When implementing a REST API call, a server response will have several key components:

Headers: Identical use case, but keep in mind that headers from server to client will be following a different list.
Response Code: This should provide detail on the status of the API call.
In network automation, I strongly discourage simply trusting the response code as a means of testing for changes. It's better to make multiple GET requests to verify that the change was executed and provided the intended effects.
If implementing API-specific code, vendors will provide what each error code means specifically to them. Python supports constructing a dictionary with numeric indexes, a useful mechanism for mapping the vendor list, ex:

1httperrors = {  
2    1: ('Unknown Command', 'The specific config or operational command is not recognized.'),  
3    2: ('Internal Error', 'Check with technical support when seeing these errors.'),  
4    3: ('Internal Error', 'Check with technical support when seeing these errors.'),  
5    4: ('Internal Error', 'Check with technical support when seeing these errors.'),  
6    5: ('Internal Error', 'Check with technical support when seeing these errors.')  
7    }

Body: Ideally used for any additional detail on why the response provided executed with the status provided, but not mandatory.

Verb

In a REST API, it's important to specify the TYPE of change you intend to make prior to actually invoking it. F5 Administrators will be familiar with this, with actions like tmsh create. We have 4 major REST verbs:

Create
Read
Modify/Update
Delete

When you use a particular transport, you need to implement these verbs in a method native to that transport. This is significant when using other remote command methods like SSH (tmsh does this) or NetCONF or RESTCONF, all of which need a different method to implement.

Fortunately for us, HTTP 1.1 seems like it's been made for this! HTTP has plenty of verbs that match the above, here's a brief decoder ring.

GET: READ-only request, typically does not include a message body.
This will normally use a URI to specify what details you want to grab.
Since you're "getting" information here, typically you'd want to JSON pretty-print the output
POST: CREATE request, if you're making a new object on a remote system a message body is typically required and POST conveniently supports that.
POST should not overwrite existing data, but REST implementations vary!
POST: READ request, occasionally used when a query requires a message body.
URIs don't always cut it when it comes to remote filtered requests or complex multi-tier queries.
Cisco NX-API avoids GET as a READ verb, and primarily uses POST instead with the REST verbs in the body
PUT: UPDATE request, is idempotent. Generally does not contain a lot of change safety, as it will implement or fully replace an object.
Situations definitely exist that you want to be idempotent, and this is the verb for that.
Doesn't require a body
PATCH: MODIFY request, will only modify an existing object.
This will take considerably more work to structure, as PATCH can optionally be safely executed, but the responsibility for assembling requests safely in this manner is on the developer.
Most API implementations simply use POST instead and implement change safety in the back-end.
DELETE: DELETE request, does exactly what it sounds like, it makes a resource disappear.

Nota Bene: None of this is a mandatory convention, so vendors may implement deviations from the REST spec. For example, Palo Alto will use XML and 0-100 series HTTP codes.

Executing a REST Call

Once the rules are set, the execution of a REST call is extremely easy, here's an example:

 1curl -k -u admin https://nsx.lab.engyak.net/api/v1/alarms  
 2Enter host password for user 'admin':  
 3{  
 4"results" : \[ {  
 5"id" : "3e79618a-c89e-477b-8872-f4c87120585b",  
 6"feature\_name" : "certificates",  
 7"event\_type" : "certificate\_expiration\_approaching",  
 8"feature\_display\_name" : "Certificates",  
 9"event\_type\_display\_name" : "Certificate Expiration Approaching",  
10"summary" : "A certificate is approaching expiration.",  
11"description" : "Certificate 5c9565d8-2cfa-4a28-86cc-e095acba5ba2 is approaching expiration.",  
12"recommended\_action" : "Ensure services that are currently using the certificate are updated to use a new, non-expiring certificate. For example, to apply a new certificate to the HTTP service, invoke the  
13following NSX API POST /api/v1/node/services/http?action=apply\_certificate&certificate\_id=<cert-id> where <cert-id> is the ID of a valid certificate reported by the GET /api/v1/trust-management/certificates NS  
14X API. Once the expiring certificate is no longer in use, it should be deleted by invoking the DELETE /api/v1/trust-management/certificates/5c9565d8-2cfa-4a28-86cc-e095acba5ba2 NSX API.",  
15"node\_id" : "37e90542-f8b8-136e-59bc-5dd3b79b122b",  
16"node\_resource\_type" : "ClusterNodeConfig",  
17"entity\_id" : "5c9565d8-2cfa-4a28-86cc-e095acba5ba2",  
18"last\_reported\_time" : 1637510695463,  
19"status" : "OPEN",  
20"severity" : "MEDIUM",  
21"node\_display\_name" : "nsx",  
22"node\_ip\_addresses" : \[ "10.66.0.204" \],  
23"reoccurrences\_while\_suppressed" : 0,  
24"entity\_resource\_type" : "certificate\_self\_signed",  
25"alarm\_source\_type" : "ENTITY\_ID",  
26"alarm\_source" : \[ "5c9565d8-2cfa-4a28-86cc-e095acba5ba2" \],  
27"resource\_type" : "Alarm",  
28"display\_name" : "3e79618a-c89e-477b-8872-f4c87120585b",  
29"\_create\_user" : "system",  
30"\_create\_time" : 1635035211215,  
31"\_last\_modified\_user" : "system",  
32"\_last\_modified\_time" : 1637510695464,  
33"\_system\_owned" : false,  
34"\_protection" : "NOT\_PROTECTED",  
35"\_revision" : 353  
36}

Now - saving the cURL commands can be very administratively intensive - So I recommend some form of method to save and automate custom API calls. Quite a few more complex calls will require JSON payloads, variables, stuff like that.

Executing a Procedure

Planning the Procedure

Here we'll use the API to resolve the following alarm. I'm going to use my own REST client, found here, because it's familiar. Let's write the desired result in pseudo-code first to develop a plan:

GET current cluster certificate ID
GET certificate store
PUT a replacement certificate with a new name
GET certificate store (validate PUT)
GET certificate ID (to further validate PUT). For idempotency, multiple runs should be supported.
POST update cluster certificate
GET current cluster certificate ID

This process seems tedious, but computers don't ever get bored, and the objective here is to be more thorough than is reasonably feasible with manual execution! If you're thinking, "Gee, this is an awful lot of work!" trick rocks into doing it for you.

Let's Trick Those Rocks

Some general guidelines when scripting API calls:

Use a familiar language. An infrastructure engineer's goal with automation is reliability. Hiring trends, hipster cred, don't matter here. If you do best with a slide rule, use that.
Use libraries. An infrastructure engineer's goal with automation is reliability. Leverage libraries with publicly available testing results.
Log and Report: An infrastructure engineer's goal with automation is reliability. Report every little thing your code does to your infrastructure, and test code thoroughly.

In this case, I published a wrapper for Python requests that allows me to save API settings here, and built a script on that library. Install it first:

1python3 \-m pip install restify-ENGYAK

From here, it's important to research the API calls required for this procedure (good thing we have the steps!). For NSX-T, the API Documentation is available here: https://developer.vmware.com/apis/1163/nsx-t

NSX-T's Certificate management API also has a couple of quirks, where the Web UI and the API leverage different certificates. It's outlined here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-50C36862-A29D-48FA-8CE7-697E64E10E37.html

Since we're writing code for reliability

I'd like to outline a rough idea of where my time investment was for this procedure. I hope it helps because the focus really isn't on writing code.

50%: Testing and planning testing. I used Jenkins CI for this, and I'm not the most capable with it. This effort reduces over time, but does not reduce importance! Write your test cases before everything!
30%: Research. Consulting the VMware API docs and official documentation was worth every yoctosecond - avoiding potential problems with planned work is critical (and there were some major caveats with the API implementation).
10%: Updating the parent library, setting up the python environment. Most of this work is 100% re-usable.
5%: Managing source code, Git branching, basically generating a bread-crumb trail for the implementation for when I don't remember it.
5%: Actually writing code!

**I'm saving useful API examples in my public repository:https://github.com/ngschmidt/python-restify
**

The Code

 1# JSON Parsing tool  
 2import json  
 3  
 4# Import Restify Library  
 5from restify.RuminatingCogitation import Reliquary  
 6  
 7# Import OS - let's use this for passwords and usernames  
 8# APIUSER = Username  
 9# APIPASS = Password  
10import os  
11  
12api_user = os.getenv("APIUSER")  
13api_pass = os.getenv("APIPASS")  
14  
15# Set the interface - apply from variables no matter what  
16cogitation_interface = Reliquary(  
17    "settings.json", input_user=api_user, input_pass=api_pass  
18)  
19  
20# Build Results Dictionary  
21stack = {  
22    "old_cluster_certificate_id": False,  
23    "old_certificate_list": [],  
24    "upload_result": False,  
25    "new_certificate_id": False,  
26    "new_certificate_list": [],  
27    "new_cluster_certificate_id": False,  
28}  
29  
30# GET current cluster certificate ID  
31stack["old_cluster_certificate_id"] = json.loads(  
32    cogitation_interface.namshub("get_cluster_certificate_id")  
33)["certificate_id"]  
34  
35# GET certificate store  
36for i in json.loads(cogitation_interface.namshub("get_cluster_certificates"))[  
37    "results"  
38]:  
39    stack["old_certificate_list"].append(i["id"])  
40# We need to compare lists, so let's sort it first  
41stack["old_certificate_list"].sort()  
42  
43# PUT a replacement certificate with a new name  
44print(cogitation_interface.namshub("put_certificate", namshub_variables="cert.json"))  
45  
46# GET certificate store (validate PUT)  
47for i in json.loads(cogitation_interface.namshub("get_cluster_certificates"))[  
48    "results"  
49]:  
50    stack["new_certificate_list"].append(i["id"])  
51# We need to compare lists, so let's sort it first, then make it the difference between new and old  
52stack["old_certificate_list"].sort()  
53stack["new_certificate_list"] = list(  
54    set(stack["new_certificate_list"]) - set(stack["old_certificate_list"])  
55)  
56  
57# Be Idempotent - this may be run multiple times, and should handle it accordingly.  
58if len(stack["new_certificate_list"]) == 0:  
59    stack["new_certificate_id"] = input(  
60        "Change not detected! Please select a certificate to replace with: "  
61    )  
62else:  
63    stack["new_certificate_id"] = stack["new_certificate_list"][0]  
64  
65# GET certificate ID (to further validate PUT)  
66print(  
67    cogitation_interface.namshub(  
68        "get_cluster_certificate",  
69        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
70    )  
71)  
72# POST update cluster certificate  
73print(  
74    cogitation_interface.namshub(  
75        "post_cluster_certificate",  
76        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
77    )  
78)  
79print(  
80    cogitation_interface.namshub(  
81        "post_webui_certificate",  
82        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
83    )  
84)  
85  
86# GET current cluster certificate ID  
87stack["new_cluster_certificate_id"] = json.loads(  
88    cogitation_interface.namshub("get_cluster_certificate_id")  
89)["certificate_id"]  
90  
91# Show the results  
92print(json.dumps(stack, indent=4))