ENGYAK.co Logo
ENGYAK.co

Why Automate Reliability Approaches with the VMware NSX-T API

Why should an infrastructure engineer leverage REST APIs?

I’m sure most IT workers have at least heard of REST APIs, or heard a sales pitch where a vendor insists that while a requested functionality doesn’t exist, you could build it yourself by “using the API”.

Or, participate in discussions where people seemed to try and offer you a copy of The DevOps Handbook or The Unicorn Project.

They’re right, but software development and deployment methods have completely different guiding values than infrastructure management. Speed of delivery is almost completely worthless with infrastructure, where downtime is typically the only metric that infrastructure is evaluated on.

We need to transform the industry.

The industry has proven that “value-added infrastructure” is a thing that people want, otherwise, services like Amazon AWS, Azure, Lumen would not be profitable. Our biggest barrier to success right now is the perceptions around reliability because there clearly is demand for what we’d call abstraction of infrastructure. We can’t move as slow as we used to, but we can’t make mistakes either.

Stuck between a rock and a hard place?

I have some good news - everybody’s just figuring this out as they go, and you don’t have to start by replacing all of your day-to-day tasks with Ansible playbooks. Let’s use automation tools to ensure Quality First, Speed Second. Machines excel at comparison operators, allowing an infrastructure administrator to test every possible aspect of infrastructure when executing a change. Here are some examples where I’ve personally seen a need for automation:

The neat thing about this capability is the configuration reversal - API calls are incredibly easy to process in common programming languages (particularly compared to expect) and take fractions of a second to run - so if a tested process (it’s easy to test, too!) does fail, reversion is straightforward. Let’s cover the REST methods before exploring the deeper stuff like gNMI or YANG.

Anatomy of a REST call

When implementing a REST API call, a client request will have several key components:

When implementing a REST API call, a server response will have several key components:

httperrors = {  
    1: ('Unknown Command', 'The specific config or operational command is not recognized.'),  
    2: ('Internal Error', 'Check with technical support when seeing these errors.'),  
    3: ('Internal Error', 'Check with technical support when seeing these errors.'),  
    4: ('Internal Error', 'Check with technical support when seeing these errors.'),  
    5: ('Internal Error', 'Check with technical support when seeing these errors.')  
    }

Verb

In a REST API, it’s important to specify the TYPE of change you intend to make prior to actually invoking it. F5 Administrators will be familiar with this, with actions like tmsh create. We have 4 major REST verbs:

When you use a particular transport, you need to implement these verbs in a method native to that transport. This is significant when using other remote command methods like SSH (tmsh does this) or NetCONF or RESTCONF, all of which need a different method to implement.

Fortunately for us, HTTP 1.1 seems like it’s been made for this! HTTP has plenty of verbs that match the above, here’s a brief decoder ring.

Nota Bene: None of this is a mandatory convention, so vendors may implement deviations from the REST spec. For example, Palo Alto will use XML and 0-100 series HTTP codes.

Executing a REST Call

Once the rules are set, the execution of a REST call is extremely easy, here’s an example:

curl -k -u admin https://nsx.lab.engyak.net/api/v1/alarms  
Enter host password for user 'admin':  
{  
"results" : \[ {  
"id" : "3e79618a-c89e-477b-8872-f4c87120585b",  
"feature\_name" : "certificates",  
"event\_type" : "certificate\_expiration\_approaching",  
"feature\_display\_name" : "Certificates",  
"event\_type\_display\_name" : "Certificate Expiration Approaching",  
"summary" : "A certificate is approaching expiration.",  
"description" : "Certificate 5c9565d8-2cfa-4a28-86cc-e095acba5ba2 is approaching expiration.",  
"recommended\_action" : "Ensure services that are currently using the certificate are updated to use a new, non-expiring certificate. For example, to apply a new certificate to the HTTP service, invoke the  
following NSX API POST /api/v1/node/services/http?action=apply\_certificate&certificate\_id=<cert-id> where <cert-id> is the ID of a valid certificate reported by the GET /api/v1/trust-management/certificates NS  
X API. Once the expiring certificate is no longer in use, it should be deleted by invoking the DELETE /api/v1/trust-management/certificates/5c9565d8-2cfa-4a28-86cc-e095acba5ba2 NSX API.",  
"node\_id" : "37e90542-f8b8-136e-59bc-5dd3b79b122b",  
"node\_resource\_type" : "ClusterNodeConfig",  
"entity\_id" : "5c9565d8-2cfa-4a28-86cc-e095acba5ba2",  
"last\_reported\_time" : 1637510695463,  
"status" : "OPEN",  
"severity" : "MEDIUM",  
"node\_display\_name" : "nsx",  
"node\_ip\_addresses" : \[ "10.66.0.204" \],  
"reoccurrences\_while\_suppressed" : 0,  
"entity\_resource\_type" : "certificate\_self\_signed",  
"alarm\_source\_type" : "ENTITY\_ID",  
"alarm\_source" : \[ "5c9565d8-2cfa-4a28-86cc-e095acba5ba2" \],  
"resource\_type" : "Alarm",  
"display\_name" : "3e79618a-c89e-477b-8872-f4c87120585b",  
"\_create\_user" : "system",  
"\_create\_time" : 1635035211215,  
"\_last\_modified\_user" : "system",  
"\_last\_modified\_time" : 1637510695464,  
"\_system\_owned" : false,  
"\_protection" : "NOT\_PROTECTED",  
"\_revision" : 353  
}

Now - saving the cURL commands can be very administratively intensive - So I recommend some form of method to save and automate custom API calls. Quite a few more complex calls will require JSON payloads, variables, stuff like that.

Executing a Procedure

Planning the Procedure

Here we’ll use the API to resolve the following alarm. I’m going to use my own REST client, found here, because it’s familiar. Let’s write the desired result in pseudo-code first to develop a plan:

This process seems tedious, but computers don’t ever get bored, and the objective here is to be more thorough than is reasonably feasible with manual execution! If you’re thinking, “Gee, this is an awful lot of work!” trick rocks into doing it for you.

Let’s Trick Those Rocks

Some general guidelines when scripting API calls:

In this case, I published a wrapper for Python requests that allows me to save API settings here, and built a script on that library. Install it first:

python3 \-m pip install restify-ENGYAK

From here, it’s important to research the API calls required for this procedure (good thing we have the steps!). For NSX-T, the API Documentation is available here: https://developer.vmware.com/apis/1163/nsx-t

NSX-T’s Certificate management API also has a couple of quirks, where the Web UI and the API leverage different certificates. It’s outlined here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-50C36862-A29D-48FA-8CE7-697E64E10E37.html

Since we’re writing code for reliability

I’d like to outline a rough idea of where my time investment was for this procedure. I hope it helps because the focus really isn’t on writing code.

**I’m saving useful API examples in my public repository:https://github.com/ngschmidt/python-restify
**

The Code

# JSON Parsing tool  
import json  
  
# Import Restify Library  
from restify.RuminatingCogitation import Reliquary  
  
# Import OS - let's use this for passwords and usernames  
# APIUSER = Username  
# APIPASS = Password  
import os  
  
api_user = os.getenv("APIUSER")  
api_pass = os.getenv("APIPASS")  
  
# Set the interface - apply from variables no matter what  
cogitation_interface = Reliquary(  
    "settings.json", input_user=api_user, input_pass=api_pass  
)  
  
# Build Results Dictionary  
stack = {  
    "old_cluster_certificate_id": False,  
    "old_certificate_list": [],  
    "upload_result": False,  
    "new_certificate_id": False,  
    "new_certificate_list": [],  
    "new_cluster_certificate_id": False,  
}  
  
# GET current cluster certificate ID  
stack["old_cluster_certificate_id"] = json.loads(  
    cogitation_interface.namshub("get_cluster_certificate_id")  
)["certificate_id"]  
  
# GET certificate store  
for i in json.loads(cogitation_interface.namshub("get_cluster_certificates"))[  
    "results"  
]:  
    stack["old_certificate_list"].append(i["id"])  
# We need to compare lists, so let's sort it first  
stack["old_certificate_list"].sort()  
  
# PUT a replacement certificate with a new name  
print(cogitation_interface.namshub("put_certificate", namshub_variables="cert.json"))  
  
# GET certificate store (validate PUT)  
for i in json.loads(cogitation_interface.namshub("get_cluster_certificates"))[  
    "results"  
]:  
    stack["new_certificate_list"].append(i["id"])  
# We need to compare lists, so let's sort it first, then make it the difference between new and old  
stack["old_certificate_list"].sort()  
stack["new_certificate_list"] = list(  
    set(stack["new_certificate_list"]) - set(stack["old_certificate_list"])  
)  
  
# Be Idempotent - this may be run multiple times, and should handle it accordingly.  
if len(stack["new_certificate_list"]) == 0:  
    stack["new_certificate_id"] = input(  
        "Change not detected! Please select a certificate to replace with: "  
    )  
else:  
    stack["new_certificate_id"] = stack["new_certificate_list"][0]  
  
# GET certificate ID (to further validate PUT)  
print(  
    cogitation_interface.namshub(  
        "get_cluster_certificate",  
        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
    )  
)  
# POST update cluster certificate  
print(  
    cogitation_interface.namshub(  
        "post_cluster_certificate",  
        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
    )  
)  
print(  
    cogitation_interface.namshub(  
        "post_webui_certificate",  
        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
    )  
)  
  
# GET current cluster certificate ID  
stack["new_cluster_certificate_id"] = json.loads(  
    cogitation_interface.namshub("get_cluster_certificate_id")  
)["certificate_id"]  
  
# Show the results  
print(json.dumps(stack, indent=4))