engyak.co

Idempotently Manage Ubiquiti Unifi resources with Ansible

Fri, 17 Apr 2026 00:00:00 -0900

Ubiquiti is beginning to roll out inbound APIs for their current product line (documentation here) as part of a broader effort to enable programmability. There are a number of useful features here, so I'll provide my summary assessment of the state of things first:

Ubiquiti is focused on monitoring first, so any Site Manager or Network endpoints in active development will first pick up telemetry and statistics
Idempotency isn't achieved by the API itself, so practice due care when applying things to an existing site
There are probably more API endpoints than there are documented. I'm going to assume that if an endpoint isn't documented, it may not be 100% ready

Like with all new API implementations, we do see some typical late-comer benefits here. The Ubiquiti developer portal provides code generators for common API consumption tasks, which provides both an easy way to get started and a rough idea of how the API integrations should work (Read: any idiosyncrasies).

Building an Action

As always, I try to keep all deployment automation encapsulated into a CI/CD pipeline. For this example, I'm also going to use 1Password's Devops tooling for password management. This GitHub Action also includes leveraging a Python 3 venv to ensure no pre-existing gunk is carried over from the OS, and it will install Ubiquiti's latest Ansible collection from scratch on every execution.

Note: This will require a requirements.txt file with all Python 3 dependencies, as it doesn't use the system's packages. Examples below.

Action

 1---
 2name: 'On-Demand: Build Unifi'
 3
 4on:
 5  workflow_dispatch:
 6
 7permissions:
 8  contents: read
 9
10jobs:
11  build:
12    name: 'Build Configurations (Unifi)'
13    runs-on: self-hosted
14    steps:
15      - uses: actions/checkout@v6
16      - name: Configure 1Password
17        uses: 1password/load-secrets-action/configure@v4
18        with:
19          service-account-token: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }}
20      - name: Load secret
21        uses: 1password/load-secrets-action@v4
22        with:
23          # Export loaded secrets as environment variables
24          export-env: true
25        env:
26          UNIFI_API: 'op://devops/unifi_api/hostname'
27          UNIFI_API_KEY: 'op://devops/unifi_api/credential'
28          UNIFI_SITE: 'op://devops/unifi_api/text'
29      - name: 'Build Unifi'
30        run: |
31          python3 -m venv .
32          source bin/activate
33          python3 -m pip install --upgrade pip
34          python3 -m pip install -r requirements.txt
35          curl -L -o ubiquiti-unifi_api-latest.tar.gz https://apidoc-cdn.ui.com/ansible-module/ubiquiti-unifi_api-latest.tar.gz
36          ansible-galaxy collection install ubiquiti-unifi_api-latest.tar.gz
37          ansible-playbook build_unifi.yml
38        working-directory: roles/unifi

`requirements.txt`

The most important one to include here is httpx. The software package provided by Ubiquiti doesn't provide it.

1###### Requirements without Version Specifiers ######
2jinja2
3requests
4urllib3
5httpx
6
7###### Requirements with Version Specifiers ######
8ansible >= 8.4.0              # Mostly just don't use old Ansible (e.g. v2, v3)

Building a Playbook

Unifi's Ansible offering all appears to leverage the same API wrapper module, mostly just for some quality of life enhancements like attaching API keys to HTTP headers, and module_defaults that prevent repetitive code.

Like with most network product playbooks, these executions are not designed to execute on a target node, and will require some tweaks. Don't become, don't gather_facts, and force controller execution.

Before we start applying configurations, we need to be aware of the controller's site options. The example here is a multi-site configuration, which would be the same as their SaaS controller. The following playbook will be my start point; it will leverage the Unifi Site Manager API and use selectaddr to return the first entry matching the assigned name (from the environment variable):

 1---
 2- name: 'Build Unifi Configs'
 3  hosts: localhost
 4  gather_facts: false
 5  # Before executing ensure that the prerequisites are installed
 6  # We start with a pre-check playbook, if it fails, we don't want to
 7  # make changes
 8  any_errors_fatal: true
 9  module_defaults:
10    group/ubiquiti.unifi_api.common:
11      base_url: "{{ lookup('env', 'UNIFI_API') }}"
12      token: "{{ lookup('env', 'UNIFI_API_KEY') }}"
13  vars:
14    unifi_site_name: "{{ lookup('env', 'UNIFI_SITE') }}"
15  tasks:
16    - name: 'Get Sites List'
17      ubiquiti.unifi_api.network:
18        path: '/v1/sites'
19        method: 'GET'
20      register: get_sites_result
21    - name: 'Select site based on name'
22      ansible.builtin.set_fact:
23        unifi_site_data: "{{ get_sites_result.data.data | selectattr('name', 'equalto', unifi_site_name) | first }}"
24    - name: 'Print Site data'
25      ansible.builtin.debug:
26        msg: 'Site data: {{ unifi_site_data }}'

Now - about Ubiquiti's Ansible module. It's not idemopotent, meaning that repeated runs of the same playbook are not "safe"; it will blindly apply the same change over itself without determining if any change is necessary. Repeated runs of a POST return an error:

1fatal: [localhost]: FAILED! => {"changed": false, "data": {"code": "api.network.validation.vlan-id-conflict", "message": "VLAN ID 22 is already in use by network: test_net", "requestId": "66afeb7c-aa6b-4532-9217-23734f386809", "requestPath": "/integration/v1/sites/{{site }}/networks", "statusCode": 400, "statusName": "BAD_REQUEST", "timestamp": "2026-04-12T18:11:39.682017935Z"}, "msg": "API call failed", "status": 400

While this is better than, say, destroying and recreating a VLAN with devices in it, idempotency is developer's responsibility. We need to modify the following play:

1    - name: 'Create Network'
2      ubiquiti.unifi_api.network:
3        path: '/v1/sites/{{ unifi_site_data.id }}/networks'
4        method: 'POST'
5        body:
6          management: 'UNMANAGED'
7          name: 'test_net'
8          enabled: true
9          vlanId: 22

To be more clever. The ideal method here would be to write our own Ansible module, but that negates the benefits of using a vendor-provided module (simplicity). Here's an improvised way to handle things - first, we must gather the filtered list of VLANs from the Unifi API:

1    - name: 'Get Networks'
2      ubiquiti.unifi_api.network:
3        path: '/v1/sites/{{ unifi_site_data.id }}/networks'
4        method: 'GET'
5        query:
6          filter: "or(vlanId.eq({{ item.vlanId }}), name.eq('{{ item.name }}'))"
7      loop: '{{ vlans }}'
8      register: get_vlans_result

This will submit an API request for each VLAN we intend to "manage" with Ansible, filtered per Ubiquiti's documentation here.

Ansible then formats the results from each run specially when we use a loop. It'll produce a list with the key results, which provides plenty of metadata from the run! Both the item.item (the variables we provided) and the item.data.data[0] variables are useful, allowing us to create a decision tree:

If item.data.data size is 0, then it's a new VLAN. Create from item.item (POST)
If item.data.data and item.item don't match, update (PUT)
If item.data.data and item.item do match, don't do anything.

Here's an example output below. It's generic to Ansible loop, so it can be re-used for just about anything:

 1{
 2   "changed":false,
 3   "msg":"All items completed",
 4   "results":[
 5      {
 6         "ansible_loop_var":"item",
 7         "changed":false,
 8         "failed":false,
 9         "item":{
10            "ansible_loop_var":"item",
11            "changed":false,
12            "data":{
13               "count":1,
14               "data":[
15                  {
16                     "default":false,
17                     "enabled":true,
18                     "id":"uuid",
19                     "management":"UNMANAGED",
20                     "metadata":{
21                        "origin":"USER_DEFINED"
22                     },
23                     "name":"test_net",
24                     "vlanId":22
25                  }
26               ],
27               "limit":25,
28               "offset":0,
29               "totalCount":1
30            },
31            "failed":false,
32            "invocation":{
33               "module_args":{
34                  "api_key_header":"X-API-KEY",
35                  "base_url":"***",
36                  "body":null,
37                  "ca_path":null,
38                  "client_cert":null,
39                  "client_key":null,
40                  "console_id":null,
41                  "files":null,
42                  "headers":{
43                     
44                  },
45                  "method":"GET",
46                  "path":"/v1/sites/uuid/networks",
47                  "query":{
48                     "filter":"or(vlanId.eq(22), name.eq('TestVlan'))"
49                  },
50                  "token":"VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
51                  "validate_certs":false
52               }
53            },
54            "item":{
55               "enabled":true,
56               "management":"UNMANAGED",
57               "name":"TestVlan",
58               "vlanId":22
59            },
60            "status":200
61         },
62         "msg":"{'name': 'TestVlan', 'vlanId': 22, 'enabled': True, 'management': 'UNMANAGED'} was found as a potential match for {'management': 'UNMANAGED', 'id': 'uuid', 'name': 'test_net', 'enabled': True, 'vlanId': 22, 'metadata': {'origin': 'USER_DEFINED'}, 'default': False}"
63      }
64   ],
65   "skipped":false
66}

Here's an example for the Create/Update plays, with the conditional logic:

 1    - name: 'Update pre-existing VLANs'
 2      ubiquiti.unifi_api.network:
 3        path: '/v1/sites/{{ unifi_site_data.id }}/networks/{{ item.data.data[0].id }}'
 4        method: 'PUT'
 5        body:
 6          management: '{{ item.item.management }}'
 7          name: '{{ item.item.name }}'
 8          enabled: '{{ item.item.enabled }}'
 9          vlanId: '{{ item.item.vlanId }}'
10      loop: '{{ get_vlans_result.results }}'
11      when:
12        - item.item.name != item.data.data[0].name
13    - name: 'Create VLANs'
14      ubiquiti.unifi_api.network:
15        path: '/v1/sites/{{ unifi_site_data.id }}/networks'
16        method: 'POST'
17        body:
18          management: '{{ item.item.management }}'
19          name: '{{ item.item.name }}'
20          enabled: '{{ item.item.enabled }}'
21          vlanId: '{{ item.item.vlanId }}'
22      loop: '{{ get_vlans_result.results }}'
23      when:
24        - item.data.data | length == 0

And that's a (slightly verbose) method to implement idempotency with an Ansible module that doesn't provide it natively, without code. The when directive for the update method takes a list, each field you want to test for a match will have to be explicitly defined as a test. This can get pretty top-heavy pretty quickly, which is why most mature API providers just process a client request and implement idempotency on the backend.

The only downside to waiting for a provider to do that is that you'll be waiting a while.

Starting an IaC Repository with GitHub and Terraform

Sat, 13 Dec 2025 00:00:00 -0900

There are only two hard things in Computer Science: cache invalidation, naming things, and off by one errors.

Loosely attributed to Phil Karlton

Let's start off with a bit of a hot take - Terraform isn't particularly hard to learn. It does use unique configuration languages, but most people don't struggle with learning the code.

Infrastructure-as-Code (IaC) isn't about the programming language - it's about establishing a body of discipline around managing infrastructure. Tools like Ansible and Terraform simply facilitate the practice.

Instead of focusing on some programmatically elegant tricks here, let's try to focus on how to build a "starter kit" of sorts to build upon this practice. The managed resources in this example will be intentionally simple to shift focus to the structure, naming, and release management aspects of Infrastructure-as-Code.

Repositories (Structure and Naming)

Start a GitHub repository with some basic documentation before contributing code:

README.md should describe what the project is for, describe the project structure: how the software works.
USAGE.md should describe how to consume resources within the project, how release management works.
CONTRIBUTING.md should describe how to contribute to the codebase: the branch and merge workflows and rules of conduct go here.
CHANGELOG.md should be created based on the Keep a Changelog standards
.gitignore should make sure that any temporary files created by tools, like pycache, Terraform locks don't accidentally get committed to the repository
markdownlint.json and any other linting rules - automated code QC is a good thing
img/ should be created to contain rendered images for documentation. Use illustrations to make the repository easy to understand!
dwg/ should be created to contain unrendered diagrams, e.g. svg, d2
doc/ may be created for any automatically rendered documentation, e.g. ReadTheDocs

Once these are created, start mapping out what loose structures should be included in the repository. Here are some examples:

conf.d/ for any flat file configurations that may get deployed
- Make subdirectories for any machine targets
roles/ for any Ansible roles. Since this is IaC, breaking this down into roles instead of one giant pile will be simpler
- Within each role:
  - templates/ should contain any Jinja2 templates. Ansible will auto-detect this folder by name, and it simplifies structure quite a bit.
  - requirements.txt should contain any software prerequisites for the Ansible playbooks. This facilitates CI/CD tooling with virtual environments, in addition to better documenting software dependencies.
  - Playbooks and truth files, of course
terraform/ for any Terraform code
- modules/ for any Terraform re-usable modules
- accounts/ for any Terraform tenants, e.g. AWS Accounts, CloudFlare accounts, or other unrelated resources to keep them separate and organized
python/ for any Python code
js/ for any JavaScript
...and so on.

Now that the raw structure is somewhat laid out, we can shift focus to the Terraform account's subdirectory (in /terraform/accounts/{{ account_type }}_{{ account_id }}_{{account_name}}) structure. Here's what I've seen lead to a maintainable code base:

/terraform/accounts/cloudflare_12345_engyak_co
- templates/ for any gotmpl templates
- provider.tf should declare any Terraform pre-requisites, e.g. the Cloudflare provider minimum version
- vars.tf should declare any input variables. In my experience, this is a good place for module inputs, but not as useful for actual infrastructure declarations
- locals.tf should declare any Don't Repeat Yourself (DRY) variables. I typically use them for consistent resource names and IDs. There are a lot of opinions about vars versus locals, but there are a few key differences:
  - vars should actually be variable (non-static multiples of a resource)
  - locals can render and iterate on an input, e.g. with for_each loops
- backend.tf should indicate where terraform.tfstate is placed, any file locking. Normally, this points to an S3 bucket and provides authorization for it
- data.tf should have any external data resources. This example doesn't need any, but AWS IAM policy documents and S3 bucket policies fit this category. Any resource prefixed with data instead of resource goes here, essentially

Now that all that's out of the way, we're able to actually create resources. Things can be a lot more free-form here, because the definition of related resources can vary greatly based on who's doing the work.

My personal preference is to maintain small, easily readable files that function independently wherever possible. In this example, we'll use one file for each DNS zone. Here's /terraform/accounts/cloudflare_youwish_engyak_co/engyak.co.tf:

 1resource "cloudflare_record" "engyak_co_blog" {
 2  content = "blog-engyak-co.pages.dev"
 3  name    = "blog"
 4  proxied = false
 5  ttl     = 1
 6  type    = "CNAME"
 7  zone_id = "redacted"
 8}
 9
10resource "cloudflare_record" "engyak_co_root" {
11  content = "blog-engyak-co.pages.dev"
12  name    = "engyak.co"
13  proxied = true
14  ttl     = 1
15  type    = "CNAME"
16  zone_id = "redacted"
17}
18
19resource "cloudflare_record" "engyak_co_uri_blog" {
20  name     = "engyak.co"
21  priority = 1
22  proxied  = false
23  ttl      = 1
24  type     = "URI"
25  zone_id  = "redacted"
26  data {
27    target = "blog.engyak.co"
28    weight = 1
29  }
30}

These resources are built according to the provider in provider.tf:

 1terraform {
 2  required_providers {
 3    cloudflare = {
 4      source  = "cloudflare/cloudflare"
 5      version = "~> 4"
 6    }
 7  }
 8}
 9
10provider "cloudflare" {
11}

Always consult the provider's documentation on how to use their resources.

Actions (Release Management)

The biggest advantage a Git repository has for Infrastructure-as-Code is its versioning capability, but the ability to control the release of changes can really take things to the next level.

First, I'd recommend starting out with a branch management plan. It can start simple, like:

Don't allow any commits directly to main (GitHub branch protection rules, plus general threads in CONTRIBUTING.md)
Only allow code to be pushed to main via a successful pull request (GitHub branch protection rules do this as well)
- At least 1 approving peer review
- All testing must PASS (more on this later)
All prospective changes must start as a diverging branch (or fork, but forking is much more advanced) that is up-to-date with main
Outline appropriate change windows, if applicable

At this point, the rules are in place, but none of it actually controls release. GitHub doesn't have credentials to release changes; ideally no users should either. The objective here is to prevent all direct changes to infrastructure. This can be achieved with AWS IAM roles, Cloudflare RBAC, or an equivalent. Take away the keys!

GitHub Actions provides a (usually free or cheap) amnesic container service to run ephemeral code from source control. This is going to be the foundation for this example moving forward, but other providers like GitLab and Atlassian have equivalents as well. If the source control provider doesn't have a built-in service, plenty of other CI tools exist to fill that gap, like Jenkins and Concourse.

For a Terraform pipeline, there should be two Actions per account:

terraform plan: This will test your code for validity, and also explain any potential impacts the change might have
terraform apply: This will implement tested changes. This Action should be restricted to the main branch!

Here's an example plan Action. I named it based on `{{ event trigger }}: {{ provider }} {{ action }} to keep things organized.

 1---
 2name: 'On-Commit: Cloudflare Terraform Plan'
 3
 4on:
 5  push:
 6
 7permissions:
 8  contents: read
 9
10jobs:
11  plan:
12    name: 'Terraform Plan'
13    env:
14      CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
15    runs-on: ubuntu-latest
16    steps:
17      - uses: actions/checkout@v4
18      - name: 'Terraform Setup'
19        uses: hashicorp/setup-terraform@v3
20        with:
21          terraform_version: '>= 1.10.5'
22      - name: 'Terraform Plan'
23        run: |
24          terraform init
25          terraform validate
26          terraform plan -input=false
27        working-directory: terraform/accounts/cloudflare_youwish_engyak_co/

Here's a rundown on how the testing works:

We use the env directive to expose CLOUDFLARE_API_TOKEN (specified in the cloudflare provider as the way to pass credentials)
We use actions/checkout@v4 (or latest version) to load a copy of main into the Actions runner
We use hashicorp/setup-terraform@v3. Previous Actions runners shipped with Terraform, but the base image didn't update this package frequently enough. Now it doesn't ship with the image - but this tool lets us restrict and control software versions as part of the pipeline. This lets us slow releases if breaking changes occur with terraform without having to monkey around with internals - it's a much better system.
The Terraform Plan step is where most of the work gets done. We initialize Terraform in non-interactive mode (-input=false) using our workspace with the working-directory key.

This will now run every time code is committed to the repository, and it'll display any expected changes every time code is contributed. If it fails, it will produce an error and (ideally) notify engineers/developers on where to fix it.

Note: terraform validate and terraform plan do not catch all problems, just test for config validity. Resource conflicts, API idiosyncrasies will pass this step and only reveal things on apply!

Now, we can finally start releasing changes:

 1---
 2name: 'Cron-Demand: Cloudflare Terraform Apply'
 3
 4on:
 5  workflow_dispatch:
 6    branches: ['main']
 7  schedule:
 8    - cron: "15 4,5 * * *"
 9
10permissions:
11  contents: read
12
13jobs:
14  plan:
15    name: 'Terraform Plan'
16    env:
17      CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
18    runs-on: ubuntu-latest
19    steps:
20      - uses: actions/checkout@v4
21      - name: 'Terraform Setup'
22        uses: hashicorp/setup-terraform@v3
23        with:
24          terraform_version: '>= 1.10.5'
25      - name: 'Terraform Plan'
26        id: tf_plan
27        run: |
28          terraform init
29          terraform validate
30          terraform plan -input=false --detailed-exitcode
31        continue-on-error: true
32        working-directory: terraform/accounts/cloudflare_youwish_engyak_co/
33      - name: 'Terraform Apply'
34        run: |
35          terraform apply
36        working-directory: terraform/accounts/cloudflare_youwish_engyak_co/
37        if: github.ref != 'refs/heads/main' && needs.tf_plan.outputs.exit-code == 2

This Action will either run daily at 0415-0515 UTC or if executed manually. We've established a "change window", and there are quite a few more complexities added to this workflow to implemet change safety:

detailed-exitcode and id: tf_plan allow us to "catch" the results of terraform plan. A return code of 0 means no changes required, and 2 means changes are required.
if: conditionals restrict the dangerous parts of the workflow to only execute when the branch is main and plan is valid and expects changes.

Terraform Starter Kit

This template should act as a foundational "starter kit" for establishing an effective, robust, mature Infrastructure-as-Code practice. I've found that it's easier to modify and improve an existing process than to start anew - the objective here is to get engineers past that "writer's block."

Happy coding!

Visualize and Report Ansible with OpenTelemetry and Syslog

Sun, 23 Nov 2025 00:00:00 -0900

Ansible is a fantastic tool to manage fleets of machines, but it's difficult to provide effective reporting when the fleet massively scales. Imagine hundreds of lines like this; try to find the one that failed (and why):

1PLAY RECAP *********************************************************************
2dev.lab.engyak.net         : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

...it's not difficult to read, but it doesn't decide what might deserve individual attention. It's possible to create Jinja reports that will be more executive-friendly, but they're focused on individual executions as well.

Ansible callback plugins provide us a framework to aggregate and analyze information about playbook execution without compromising idempotency.

Types of Callback Plugins

`aggregate`

aggregate callback plugins modify the summary at the end of a task's output. They don't appear to impact recap, and don't have many useful examples.

Aggregate Plugin list

`stdout`

stdout callback plugins modify the continual output presented as Ansible completes work:

1TASK [Update Apt!] *************************************************************
2ok: [dev.lab.engyak.net]

This is where the fun begins! Note that only one plugin for stdout can be selected for a given playbook.

Using `stdout` callbacks

The process for enabling callback plugins in ansible.cfg. Since this is executed from an environment (GitHub Actions), I prefer leveraging environment injection.

ANSIBLE_CALLBACK_RESULT_FORMAT controls how data is printed out from individual tasks on the screen, this is up to preference. I prefer yaml, and recommend playing with this setting to see what works best for you.
ANSIBLE_PYTHON_INTERPRETER silences any chatter about the discovered Python interpreter. Since this is a consistent environment without any tight coupling to specific releases, I don't feel the need to pin one, and I don't want to see the messages.
DEFAULT_STDOUT_CALLBACK will let you set the stdout callback plugin

In GitHub Actions, you can use the env key to manipulate outputs without having to change any code. I'm also integrating Netbox into this pipeline.

 1jobs:
 2  build:
 3    name: 'Manage Lab Configurations'
 4    runs-on: self-hosted
 5    env:
 6      ANSIBLE_PYTHON_INTERPRETER: 'auto_silent'
 7      ANSIBLE_STDOUT_CALLBACK: 'default'
 8      ANSIBLE_CALLBACK_RESULT_FORMAT: 'yaml'
 9      NETBOX_TOKEN: ${{ secrets.NETBOX_TOKEN }}
10      NETBOX_API: ${{ vars.NETBOX_URL }}
11    steps:
12      - uses: actions/checkout@v4
13      - name: Execute Ansible Management Playbook
14        run: |
15          python3 -m venv .
16          source bin/activate
17          python3 -m pip install --upgrade pip
18          python3 -m pip install -r requirements.txt
19          ansible-inventory -i local.netbox.netbox.nb_inventory.yml --graph
20          ansible-playbook -i local.netbox.netbox.nb_inventory.yml lab-management.yml

For reference purposes, I've added all compatible fields here. The yaml results format is considerably more compact given the character limit per line.

dense seems to be a popular callback, and it uses colorization to generate play output, and tries to place things all on one line:

1task 1.task 1: ns2.lab.engyak.nettask 1: ns2.lab.engyak.net ns.lab.engyak.nettask 2.task 2: ns2.lab.engyak.nettask 2: ns2.lab.engyak.net ns.lab.engyak.nettask 3.task 3: ns2.lab.engyak.nettask 3: ns2.lab.engyak.net ns.lab.engyak.nettask 4.task 4: ns.lab.engyak.nettask 4: ns.lab.engyak.net ns2.lab.engyak.nettask 5.task 5: ns2.lab.engyak.nettask 5: ns2.lab.engyak.net ns.lab.engyak.nettask 6.task 6: ns2.lab.engyak.nettask 6: ns2.lab.engyak.nettask 6: ns2.lab.engyak.nettask 6: ns2.lab.engyak.net ns.lab.engyak.nettask 6: ns2.lab.engyak.net ns.lab.engyak.nettask 6: ns2.lab.engyak.net ns.lab.engyak.nettask 7.task 7: ns.lab.engyak.nettask 7: ns.lab.engyak.net ns2.lab.engyak.nettask 7: ns.lab.engyak.net ns2.lab.engyak.nettask 7: ns.lab.engyak.net ns2.lab.engyak.nettask 7: ns.lab.engyak.net ns2.lab.engyak.nettask 7: ns.lab.engyak.net ns2.lab.engyak.nettask 8.task 8: ns2.lab.engyak.nettask 8: ns2.lab.engyak.net ns.lab.engyak.nettask 9.task 9: ns2.lab.engyak.nettask 9: ns2.lab.engyak.net ns.lab.engyak.nettask 10.task 10: ns2.lab.engyak.nettask 10: ns2.lab.engyak.net ns.lab.engyak.net

It's definitely compact, but not super readable. oneline is probably the best non-default plugin of the bunch, but it's much more verbose than the default one. It also displays a lot of system-specific information, so no snippet here.

`notification`

This is where things get really good for those of us with execution environments! notification callback plugins send data to external systems when a play finishes.

Directing results to OpenTelemetry

OpenTelemetry is a truly neat open standard for exchanging "trace information" between systems.

This is incredibly useful, but also difficult to explain in a way that's clear without providing concrete examples. Essentially, OpenTelemetry-based traces allow debugging systems that do not all exist in the same software package, and it offers a timeline for each step. As it happens, Ansible's callback plugin is well-architected and a good example of the value that a trace can have, even from an application perspective.

First, we'll need to assemble an OpenTelemetry-compliant platform to stream Ansible results to. I've selected Jaeger for this purpose. It has an all-in-one quickstart function:

1docker run --rm --name jaeger \
2  -p 16686:16686 \
3  -p 4317:4317 \
4  -p 4318:4318 \
5  -p 5778:5778 \
6  -p 9411:9411 \
7  cr.jaegertracing.io/jaegertracing/jaeger:2.12.0

Once it's running, we need to instruct Ansible to forward data. This is achievable exclusively with environment variables:

1  env:
2    ANSIBLE_CALLBACKS_ENABLED: 'community.general.opentelemetry'
3    ANSIBLE_OPENTELEMETRY_ENABLE_FROM_ENVIRONMENT: 'ANSIBLE_OPENTELEMETRY_ENABLED'
4    ANSIBLE_OPENTELEMETRY_ENABLED: 'true'
5    OTEL_EXPORTER_OTLP_ENDPOINT: 'http://jaeger.lab.engyak.net:4317'
6    OTEL_EXPORTER_INSECURE: 'true'
7    OTEL_SERVICE_NAME: 'ansible'

In addition to these variables, the module requires the following additions to requirements.txt:

1opentelemetry-sdk
2opentelemetry-exporter-otlp-proto-grpc
3opentelemetry-exporter-otlp-proto-http

Once these changes get applied, with no other required changes to the Ansible code itself, all subsequent runs submit OTLP traces to Jaeger It looks like this:

This provides a comprehensive "drill down" for every step taken by Ansible, and I've honestly never seen this level of detail before. Every single programmatic step is logged with a timestamp, allowing an engineer to find out:

Which node took too long
Which step slowed things down the most
Whether that matches the baseline for other nodes

For a transactional application this has to be even more useful.

Directing Results to Syslog

Now, for something quite a bit more boring (but equally important). If OpenTelemetry is a microscope, Syslog is the 10,000 foot view. This can also be set up by CI, and should run in parallel with OpenTelemetry:

1  env:
2    ANSIBLE_CALLBACKS_ENABLED: 'community.general.opentelemetry,community.general.syslog_json'
3    ANSIBLE_OPENTELEMETRY_ENABLE_FROM_ENVIRONMENT: 'ANSIBLE_OPENTELEMETRY_ENABLED'
4    ANSIBLE_OPENTELEMETRY_ENABLED: 'true'
5    OTEL_EXPORTER_OTLP_ENDPOINT: 'http://jaeger.lab.engyak.net:4317'
6    OTEL_EXPORTER_INSECURE: 'true'
7    OTEL_SERVICE_NAME: 'ansible'
8    SYSLOG_PORT: '54514'
9    SYSLOG_SERVER: '127.0.0.1'

Each of these callback plugins serves a different purpose. Syslog callbacks provide a shorter summary as JSON, which can easily be dashboarded:

1<14>1 2025-11-30T07:41:00-09:00 10.66.1.143 gh-runner2 - - - ansible-command: task execution OK; host: ns.lab.engyak.net; message: {"changed": false, "checksum": "a46e7011b00c560dddcc193ef16f01fd2d05970e", "dest": "/etc/unbound/unbound.conf", "gid": 0, "group": "root", "mode": "0640", "owner": "root", "path": "/etc/unbound/unbound.conf", "size": 4531, "state": "file", "uid": 0}

Some example conditions:

"changed": true would indicate how many modifications were made per hostname (identified by host: ns.lab.engyak.net)
!= 'task execution OK' would search for job failures

Modernizing the Monitoring Stack

Ansible, despite being an infrastructure tool, provides a good example of the different types of modern monitoring. Thematically, these concepts should be applied to actual applications.

Traces are an excellent tool to identify software process bottlenecks. Any tool that has long-running jobs can benefit from tracing. They're computationally costly, so they should be saved for any tool where performance degradation truly matters.
Syslog is the "swiss army knife" of monitoring. It's the best tool for simple events, and can be the foundation for event-driven programming.
Metrics allow infrastructure engineers to "just send the important bits" via tools like protobuf, sort of like SNMP but better. In the network realm, this is where Model-Driven Telemetry reigns supreme, and in the application stack Prometheus is a popular option.

One thing I did find interesting - Grafana + Alloy allowed the unification of all of these data types. Here's a preview of what Jaeger in Grafana looks like:

Automate DNS Zone Generation and Deployment with Ansible and Netbox

Sun, 10 Nov 2024 00:00:00 -0900

In a previous post, I covered a method to automatically generate DNS zones from an embedded YAML list.

This wasn't the most useful on its own, only ensuring that forward and reverse DNS entries match each other (you'll be shocked by how many places it isn't!) - and we need a good way to simplify DNS administration with tooling less expensive that, say, Infoblox.

This isn't to say that Infoblox is bad, but a fully loaded Infoblox license is a little pricy for home labs.

The Pattern

First, let's illustrate a potential design:

The Code

In order to do this, we're going to need to find a good way to pull pre-filtered data for ansible to work with, and Netbox has a GraphQL API (/graphql/) that's perfect for this:

 1{
 2  ip_address_list(filters: {dns_name: {i_contains: "example.net"}, family: 4}) {
 3    dns_name
 4    address
 5  }
 6}
 7{
 8  ip_address_list(filters: {dns_name: {i_contains: "example.net"}, family: 6}) {
 9    dns_name
10    address
11  }
12}

This will give us a separate sheet for IPv4 and IPv6 addresses attached to a given zone - and we can assemble it without any postprocessing in Ansible.

Netbox's GraphQL sandbox produces the following data output:

 1{
 2  "data": {
 3    "ip_address_list": [
 4      {
 5        "dns_name": "ns.example.net",
 6        "address": "1.1.1.1/32"
 7      }
 8    ]
 9  }
10}

Now, this is received via a graphical interface, which means we can't consume it programmatically. In order to do that, we'll need to package the GraphQL payload in JSON. Here's an Ansible task that does just that:

 1    - name: "Try Fetching `lab.engyak.net` IPv4 GraphQL!"
 2      ansible.builtin.uri:
 3        url: "https://netbox/graphql/"
 4        method: POST
 5        body:
 6          query: "query { ip_address_list(filters: {dns_name: {i_contains: \"example.com\"}, family: 4}) { dns_name address }}"
 7        body_format: "json"
 8        headers:
 9          Authorization: "Token {{ lookup('ansible.builtin.env', 'NETBOX_TOKEN') }}"
10          Content-Type: "application/json"
11          Accept: "application/json"
12        validate_certs: false
13      register: result_example_net_v4

There aren't any pynetbox based modules that automate this into Ansible, so here we're using the ansible.builtin.uri module (also known as the Jack of All Trades module) to pull JSON data. It also uses the environment variable NETBOX_TOKEN, which must be exposed by secrets management / CI processes.

In this case, I'm pulling IPv4 and IPv6 records separately. Jinja doesn't know the difference between types of record, so I cheat on postprocessing and let GraphQL do all the heavy lifting. IPv6 is the same, but with family: 6/result_example_net_v6.

The next step is to build Jinja templates to define the zonefiles. I created them in a previous post, but will include all of them in a Gist at the end of this post. They need to be modified to process output from GraphQL, because we don't control any of the field names with it.

The Jinja templates used in this example are unique to ansible - the custom filter ansible.utils.ipaddr is amazing, converting Netbox's {{ address }}/{{ cidr }} notation is compact and efficient, but it doesn't work as an A record target. Invocations like |ansible.utils.ipaddr('address') or |ansible.utils.ipaddr('revdns') are particularly useful here.

Finally, it's good to test the resulting zonefiles for sanity. It's included in the Gist.

Retrospective

Netbox's GraphQL API is a really effective tool for aggregating pre-filtered data and driving automation processes. I was quite impressed that I could just ask an API endpoint for this nice and tidy report, already pre-formatted for me!

Lack of field and format control is an issue with GraphQL (you're stuck with whatever data structure the application architect has in store for you) - but Ansible and Jinja2 empower you to present the back-end data in any front-end manner you prefer (in my case, as DNS data loaded into an Unbound instance).

Nearly any business reporting process can be driven from Netbox in this fashion, as long as the resulting format can be Jinjafied. Here are some ideas on how this can be used further:

Report on Circuits per Region
Report on IT-Managed assets in a given Site
Report on how many Sites have IPv6 coverage

The Gist

As promised, here's the raw code I created to automate DNS zonefile management from Netbox:

VM Deployment Pipelines with Proxmox

Sat, 31 Aug 2024 00:00:00 -0900

Decoupled approaches to deployment of IaaS workloads are the way of the future.

Here, we'll try to construct a VM deployment pipeline leveraging GitHub Actions and Ansible's community modules.

Proxmox Setup

Not featured here: Loading a VM ISO is particular to the Proxmox deployment, but it's necessary for future steps.

Let's create a VM named deb12.6-template:

I set a separate VM ID range for templates to simplify visual automatic sorting.

Note: Paravirtualized hardware is still the optimal choice, like with vSphere - but in this case, VirtIO is the code supplier.

Note: SSD Emulation and qemu-agent are required for virtual disk reclamation with QEMU. This is particularly important in my lab.

In this installation, I'm using paravirtualized network adapters and have separated my management(vmbr0) and data plane(vmbr1)

Debian Linux Setup

I'll skip the Linux installer parts for brevity, Debian's installer is excellent and easy to use.

At a high level, we'll want to do some preparatory steps before declaring this a usable base image:

Create users
- Recommended approach: Create a bootstrap user, then shred it
  - Leave the bootstrap user with an SSH key on the base image
  - After creation, build a takeover playbook that installs the latest and greatest username table, sssd, SSH keys, APM, anything with confidential cryptographic material that should not be left unencrypted on the hypervisor
  - This won't slow the VM deployment speed by as much as you think
Install packages
- This is just a list of some basics that I prefer to add to each machine. It's more network-centric; anything more comprehensive should be part of a build playbook specific to whatever's being deployed.
- Note: This is an Ansible playbook, and therefore, it needs Ansible to run (apt install ansible)

 1---
 2- name: "Debian machine prep"
 3  hosts: localhost
 4  tasks:
 5  - name: "Install standard packages"
 6    ansible.builtin.apt:
 7      pkg:
 8        - 'curl'
 9        - 'dnsutils'
10        - 'diffutils'
11        - 'ethtool'
12        - 'git'
13        - 'mtr'
14        - 'net-tools'
15        - 'netcat-traditional'
16        - 'python3-requests'
17        - 'python3-jinja2'
18        - 'tcpdump'
19        - 'telnet'
20        - 'traceroute'
21        - 'qemu-guest-agent'
22        - 'vim'
23        - 'wget'

Clean up the disk. This will make our base image more compact - each clone will inherit any wasted space, so consider it a 10,20x savings in disk usage. I leave this as a file on the base image and name it reset_vm.sh:

 1#!/bin/bash
 2
 3# Clean Apt
 4apt clean
 5
 6# Cleaning logs.
 7if [ -f /var/log/audit/audit.log ]; then
 8  cat /dev/null > /var/log/audit/audit.log
 9fi
10if [ -f /var/log/wtmp ]; then
11  cat /dev/null > /var/log/wtmp
12fi
13if [ -f /var/log/lastlog ]; then
14  cat /dev/null > /var/log/lastlog
15fi
16
17# Cleaning udev rules.
18if [ -f /etc/udev/rules.d/70-persistent-net.rules ]; then
19  rm /etc/udev/rules.d/70-persistent-net.rules
20fi
21
22# Cleaning the /tmp directories
23rm -rf /tmp/*
24rm -rf /var/tmp/*
25
26# Cleaning the SSH host keys
27rm -f /etc/ssh/ssh_host_*
28
29# Cleaning the machine-id
30truncate -s 0 /etc/machine-id
31rm /var/lib/dbus/machine-id
32ln -s /etc/machine-id /var/lib/dbus/machine-id
33
34# Cleaning the shell history
35unset HISTFILE
36history -cw
37echo > ~/.bash_history
38rm -fr /root/.bash_history
39
40# Truncating hostname, hosts, resolv.conf and setting hostname to localhost
41truncate -s 0 /etc/{hostname,hosts,resolv.conf}
42hostnamectl set-hostname localhost
43
44# Clean cloud-init - deprecated because cloud-init isn't currently used
45# cloud-init clean -s -l
46
47# Force a filesystem sync
48sync

Shutdown the Virtual Machine. I prefer to start it back up and shut it down from the hypervisor to ensure that qemu-guest-agent is working properly.

Deployment Pipeline

First, we will want to create an API token under "Datacenter -> Permissions -> API Tokens":

There are some oddities with the Ansible proxmoxer based module and Ansible to keep in mind:

api_user is needed and used by the API client, formatted as {{ user }}@domain
api_token_id is not the same as the output from the command, it's what you put into the "Token ID" field.
- {{ api_user}}!{{ api_token_id }} should form the combined credential presented to the API, and match the created token.

If you attempt to use the output from the API creation screen under api_user or api_token_id, it'll return a 401 Invalid user without much explanation as to what might be the issue.

Here's the pipeline. Github's primary job is to set up the Python/Ansible environment, and translate the workflow inputs into something that Ansible can properly digest.

I also added some cat steps - this allows us to use the GitHub Actions log to store intent until Netbox registration completes.

 1---
 2name: "On-Demand: Build VM on Proxmox"
 3
 4on:
 5  workflow_dispatch:
 6    inputs:
 7      machine_name:
 8        description: "Machine Name"
 9        required: true
10        default: "examplename"
11      machine_id:
12        description: "VM ID (can't re-use)"
13        required: true
14      template:
15        description: "VM Template Name"
16        required: true
17        type: choice
18        options:
19          - deb12.6-template
20        default: "deb12.6-template"
21      hardware_cpus:
22        description: "VM vCPU Count"
23        required: true
24        default: "1"
25      hardware_memory:
26        description: "VM Memory Allocation (in MB)"
27        required: true
28        default: "512"
29
30permissions:
31  contents: read
32
33jobs:
34  build:
35    runs-on: self-hosted
36    steps:
37      - uses: actions/checkout@v4
38      - name: Create Variable YAML File
39        run: |
40          cat <<EOF > roles/proxmox_kvm/parameters.yaml
41          ---
42            vm_data:
43              name: "${{ github.event.inputs.machine_name }}"
44              id: ${{ github.event.inputs.machine_id }}
45              template: "${{ github.event.inputs.template }}"
46              node: node
47              hardware:
48                cpus: ${{ github.event.inputs.hardware_cpus }}
49                memory: ${{ github.event.inputs.hardware_memory }}
50                storage: ssd-tier
51                format: qcow2
52          EOF
53      - name: Build VM
54        run: |
55          cd roles/proxmox_kvm/
56          cat parameters.yaml
57          python3 -m venv .
58          source bin/activate
59          python3 -m pip install --upgrade pip
60          python3 -m pip install -r requirements.txt
61          python3 --version
62          ansible --version
63          
64          export PAPIUSER="${{ secrets.PAPIUSER }}"
65          export PAPI_TOKEN="${{ secrets.PAPI_TOKEN }}"
66          export PAPI_SECRET="${{ secrets.PAPI_SECRET }}"
67          export PHOSTNAME="${{ secrets.PHOSTNAME }}"
68          export NETBOX_TOKEN="${{ secrets.NETBOX_TOKEN }}"
69          export NETBOX_URL="${{ secrets.NETBOX_URL }}"
70          export NETBOX_CLUSTER="${{ secrets.NETBOX_CLUSTER_PROX }}"
71          ansible-playbook build_vm_prox.yml

In addition, a requirements.txt is required by GitHub to set up the venv, and belongs in the role folder (roles/proxmox_kvm as above):

 1###### Requirements without Version Specifiers ######
 2pytz
 3netaddr
 4django
 5jinja2
 6requests
 7pynetbox
 8
 9###### Requirements with Version Specifiers ######
10ansible >= 8.4.0              # Mostly just don't use old Ansible (e.g. v2, v3)
11proxmoxer >= 2.0.0

This Ansible playbook also integrates Netbox, as my vSphere workflow did, and uses a common schema to simplify code re-use. There are a few quirks with the Proxmox playbooks:

There's no module to grab VM Guest network information, but the API provides it, so I can get it with uri
Proxmox has a nasty habit of breaking Ansible with JSON keys that include -. The best way to fix it is with a debug action: {{ prox_network_result.json.data | replace('-','_') }}
Proxmox's VM copy needs a timeout configured, and announces it's done before the VM is ready for actions. I added an ansible.builtin.pause step before starting the VM, and after (to allow it to boot)

  1---
  2- name: "Build VM on Proxmox"
  3  hosts: localhost
  4  gather_facts: true
  5  # Before executing ensure that the prerequisites are installed
  6  # `ansible-galaxy collection install netbox.netbox`
  7  # `python3 -m pip install aiohttp pynetbox`
  8  # We start with a pre-check playbook, if it fails, we don't want to
  9  # make changes
 10  any_errors_fatal: true
 11  vars_files:
 12    - "parameters.yaml"
 13
 14  tasks:
 15    - name: "Debug"
 16      ansible.builtin.debug:
 17        msg: '{{ vm_data }}'
 18    - name: "Test connectivity and authentication"
 19      community.general.proxmox_node_info:
 20        api_host: '{{ lookup("env", "PHOSTNAME") }}'
 21        api_user: '{{ lookup("env", "PAPIUSER") }}'
 22        api_token_id: '{{ lookup("env", "PAPI_TOKEN") }}'
 23        api_token_secret: '{{ lookup("env", "PAPI_SECRET") }}'
 24      register: prox_node_result
 25    - name: "Display Node Data"
 26      ansible.builtin.debug:
 27        msg: '{{ prox_node_result }}'
 28    - name: "Build the VM"
 29      community.general.proxmox_kvm:
 30        api_host: '{{ lookup("env", "PHOSTNAME") }}'
 31        api_user: '{{ lookup("env", "PAPIUSER") }}'
 32        api_token_id: '{{ lookup("env", "PAPI_TOKEN") }}'
 33        api_token_secret: '{{ lookup("env", "PAPI_SECRET") }}'
 34        name: '{{ vm_data.name }}'
 35        node: '{{ vm_data.node }}'
 36        storage: '{{ vm_data.hardware.storage }}'
 37        newid: '{{ vm_data.id }}'
 38        clone: '{{ vm_data.template }}'
 39        format: '{{ vm_data.hardware.format }}'
 40        timeout: 500
 41        state: present
 42    - name: "Wait for the VM to fully register"
 43      ansible.builtin.pause:
 44        seconds: 15
 45    - name: "Start the VM"
 46      community.general.proxmox_kvm:
 47        api_host: '{{ lookup("env", "PHOSTNAME") }}'
 48        api_user: '{{ lookup("env", "PAPIUSER") }}'
 49        api_token_id: '{{ lookup("env", "PAPI_TOKEN") }}'
 50        api_token_secret: '{{ lookup("env", "PAPI_SECRET") }}'
 51        name: '{{ vm_data.name }}'
 52        state: started
 53    - name: "Wait for the VM to fully boot"
 54      ansible.builtin.pause:
 55        seconds: 45
 56    - name: "Get VM information"
 57      community.general.proxmox_vm_info:
 58        api_host: '{{ lookup("env", "PHOSTNAME") }}'
 59        api_user: '{{ lookup("env", "PAPIUSER") }}'
 60        api_token_id: '{{ lookup("env", "PAPI_TOKEN") }}'
 61        api_token_secret: '{{ lookup("env", "PAPI_SECRET") }}'
 62        vmid: '{{ vm_data.id }}'
 63      register: prox_vm_result
 64    - name: "Report the VM!"
 65      ansible.builtin.debug:
 66        var: prox_vm_result
 67    - name: "Fetch VM Networking information"
 68      ansible.builtin.uri:
 69        url: 'https://{{ lookup("env", "PHOSTNAME") }}:8006/api2/json/nodes/{{ vm_data.node }}/qemu/{{ vm_data.id }}/agent/network-get-interfaces'
 70        method: 'GET'
 71        headers:
 72          Content-Type: 'application/json'
 73          Authorization: 'PVEAPIToken={{ lookup("env", "PAPIUSER") }}!{{ lookup("env", "PAPI_TOKEN") }}={{ lookup("env", "PAPI_SECRET") }}'
 74        validate_certs: false
 75      register: prox_network_result
 76    - name: "Refactor Network Information"
 77      ansible.builtin.debug:
 78        msg: "{{ prox_network_result.json.data | replace('-','_') }}"
 79      register: prox_network_result_modified
 80    - name: "Register the VM in Netbox!"
 81      netbox.netbox.netbox_virtual_machine:
 82        netbox_token: '{{ lookup("env", "NETBOX_TOKEN") }}'
 83        netbox_url: '{{ lookup("env", "NETBOX_URL") }}'
 84        validate_certs: false
 85        data:
 86          cluster: '{{ lookup("env", "NETBOX_CLUSTER") }}'
 87          name: '{{ vm_data.name }}'
 88          description: 'Built by the GH Actions Pipeline!'
 89          local_context_data: '{{ prox_vm_result }}'
 90          memory: '{{ vm_data.hardware.memory }}'
 91          vcpus: '{{ vm_data.hardware.cpus }}'
 92    - name: "Configure VM Interface in Netbox!"
 93      netbox.netbox.netbox_vm_interface:
 94        netbox_token: '{{ lookup("env", "NETBOX_TOKEN") }}'
 95        netbox_url: '{{ lookup("env", "NETBOX_URL") }}'
 96        validate_certs: false
 97        data:
 98          name: '{{ vm_data.name }}_intf_{{ item.hardware_address | replace(":", "") | safe }}'
 99          virtual_machine: '{{ vm_data.name }}'
100          vrf: 'Campus'
101          mac_address: '{{ item.hardware_address }}'
102      with_items: '{{ prox_network_result_modified.msg.result }}'
103      when: item.hardware_address != '00:00:00:00:00:00'
104    - name: "Reserve IP"
105      netbox.netbox.netbox_ip_address:
106        netbox_token: '{{ lookup("env", "NETBOX_TOKEN") }}'
107        netbox_url: '{{ lookup("env", "NETBOX_URL") }}'
108        validate_certs: false
109        data:
110          address: '{{ item.ip_addresses[0].ip_address }}/{{ item.ip_addresses[0].prefix }}'
111          vrf: 'Campus'
112          assigned_object:
113            virtual_machine: '{{ vm_data.name }}'
114        state: present
115      with_items: '{{ prox_network_result_modified.msg.result }}'
116      when: item.hardware_address != '00:00:00:00:00:00'
117    - name: "Finalize the VM in Netbox!"
118      netbox.netbox.netbox_virtual_machine:
119        netbox_token: '{{ lookup("env", "NETBOX_TOKEN") }}'
120        netbox_url: '{{ lookup("env", "NETBOX_URL") }}'
121        validate_certs: false
122        data:
123          cluster: '{{ lookup("env", "NETBOX_CLUSTER") }}'
124          tags: 
125            - 'lab_debian_machines'
126            - 'lab_linux_machines'
127            - 'lab_apt_updates'
128          name: '{{ vm_data.name }}'
129          primary_ip4:
130            address: '{{ item.ip_addresses[0].ip_address }}/{{ item.ip_addresses[0].prefix }}'
131            vrf: "Campus"
132      with_items: '{{ prox_network_result_modified.msg.result }}'
133      when: item.hardware_address != '00:00:00:00:00:00'

Conclusion

Overall, the Proxmox API/playbooks are quite a bit simpler to use than the VMware ones. The proxmoxer based modules are relatively feature complete compared to vmware_rest, but the largest exception I found (examples not in this post) was that I could always fall back to Ansible's comprehensive Linux foundation to fill any gaps I needed to. It's a refreshing change.

Starting from scratch with Netbox IPAM

Sat, 11 May 2024 00:00:00 -0900

Spreadsheets are not an adequate method to manage IP addressing

Different IP design strategies

IPv4

Bogons, and the basics

There are a number of valid and invalid prefixes for use internally within an enterprise. Here's a list of invalid prefixes in the global routing table; of those, the RFC 1918 prefixes are available for use:

Prefix	RFC	Usable Internally?
0.0.0.0	1122	🤪 Everybody more or less agreed not to use it
10.0.0.0/8	1918	✅ Use this block for large prefix allocations
100.64.0.0/10	6598	🙄 CG-NAT, can technically be used, but will break in random cloud applications
127.0.0.0/8	1122	❌ loopback
169.254.0.0/16	3927	✅, but don't allocate it (APIPA)
172.16.0.0/12	1918	✅ Use this block for medium prefix allocations
192.0.0.0/24	5736	❌ IETF skunkworks
192.0.2.0/24	5737	❌ Carrier test networks
192.88.99.0/24	3068	❌ 6to4 relays
192.168.0.0/16	1918	✅ Avoid this block for enterprises, it'll collide with home networks when people use VPN
198.18.0.0/15	2544	❌ device benchmarking
198.51.100.0/24	5737	❌ Carrier test networks
203.0.113.0/24	5737	❌ Carrier test networks
224.0.0.0/4	3171	❌ multicast
240.0.0.0/4	1122	🤯 madlad play, might work, might not. Linux seems to live in this space just fine

All of these prefixes must be dropped at any network perimeter, e.g. firewalls, extranet routers, to prevent internal traffic or misconfigured NATs from leaking. It also prevents protocol abuse, which is a cheap and easy way to improve security.

In multi-site networks, dropping all of these prefixes would be wise - an ethernet loop + APIPA can turn a switching issue into a network-wide outage pretty easily. Longest Prefix Match can ensure that any allocated networks remain reachable.

Each of these prefixes should be created in Netbox, so you can use it as a reference later. I'd recommend tagging them with some form of hint to indicate usability, e.g.:

IP:Usable
IP:Unusable

As you get more familiar with the API/search, tag-based filters become incredibly handy.

IPv6

IPv6 is quite easy. All valid routable addresses fall under one allocated prefix:

2000::/3

This means that you can implement a "default route" that won't accidentally leak bogons like in IPv4, but with a much simpler approach. Instead of implementing ::0/0 for your default route, use 2000::/3.

If you insist on using private addressing, I'd encourage a thorough review of why - but this is the prefix available:

fc00::/7

Link-local addressing, or addressing that is "always on" regardless of prefix allocation, is also allocated a specific prefix. This prevents the need of a bunch of little helper protocols that simply don't need to exist, or become standardized. Traffic like Router Advertisements(RA), Routing Protocols, First Hop Redundancy Protocols have a distinct source address that can be pinged even before a network is online.

It's also incredibly handy when bootstrapping new devices! All that's required is some form of helper on the default gateway to act as an SSH proxy and some neighbor discovery, and you suddenly have always-on remote management.

fe80::/10

Multicast also has its own prefix:

ff80::/8

This is much simpler, but where to get IPv6 addressing can be more complex. If it's in a lab environment and doesn't need internet access, fc00::/7 is just fine to use.

The recommended method for acquiring an IPv6 prefix is to request it with DHCP-PD or to request it through a tunnel broker.

There's one more "gotcha" to keep in mind with IPv6 - weird stuff breaks if you go with a longer prefix than /64. I'd strongly encourage avoiding cutesy CIDR block allocations like /120 or /65; that's an IPv4 solution to a problem IPv6 doesn't have. Just request enough IP addressing for your site instead.

Constructing an IP hierarchy

For the purposes of this post, we're going to use the following language to describe a network address:

1{{ prefix }}{{ subprefix }}{{ host bits }}/{{ prefix length }}
2   10.99.         100.           0                /24

The major first step here is to decide how to break down your addressing. There are two major paths to follow:

Location based addressing is used when prefix scale is a concern. If you need to summarize routes on routers due to RIB limits, this is the way to go. This can be for a few reasons:
- "I have a lot of routes / lot of sites and am worried about RIB capacity in my hardware"
  - Most enterprise equipment can handle 16-64k routes; if this is not enough, follow this approach
  - ISPs will follow this path
  - Cloud providers will follow this path
Purpose based addressing is used when perimeter security is a concern. Easy summarization to a common prefix per "network role" allows for straightforward firewall policy creation, including a number of microsegmentation tools that may have laughably low table capacities.
- "I want to keep my workloads separated from each other"
  - Financial services will follow this path
  - Healthcare will follow this path

Committing to one or the other before allocating blocks will simplify your life later.

Note: With IPv6, only the largest of organizations (ones that need more than 2^8 or 2^16 networks per site) will need to allocate their own top-level prefix. It's easier to just run DHCP-PD and ask for a /56 or /48.

Guidance on prefix allocation

To assess the proper 1918/bogon prefix for use, first assess the number of prefixes you would need as a ceiling:

1num_sites*num_network_roles

Attempt to select a site that will fit this prefix count with a minumum of 80% buffer (leaving a reserve for point-to-point connects, etc.)

I would highly encourage not getting creative with CIDR prefix lengths in IPv4-land. If possible, try and stick to /24 for a subprefix. IPv6 does not support prefix lengths longer than /64 particularly well (with specific exceptions for point-to-point, /126 or /127 depending on hardware), and using prefixes like /65 for access segments will lead to trouble with end devices like Android.

It's much simpler to translate the /24 in question linearly to a /64 and using that calculation to estimate what IPv6 prefix size you want. It's also much simpler to troubleshoot and maintain if you don't build a pile of weird stuff, even if it makes you feel smart!

As a starting point, it's good to set up a set of standard t-shirt sizes for networks in `{{ IPv4 }}/{{ IPv6 }} format. Here's an example:

Large Site/Role: /16//56
Medium Site/Role: /18//60
Small Site/Role: /22//62
"Normal" subprefix: /24//64
"Small" subprefix: /26//64
Point-to-point: /31//127

Note: Service providers don't always support weird DHCP-PD sizes, so options may be limited to the above.

Note: Service providers are typically pretty generous with prefix allocations, and keep in mind a /56 is roughly equivalent to a /16. I'd recommend allocating a /56 per site in production or in the lab whenever permitted.

Automating it

Once you have sizes set, it's actually pretty easy to let go of your artisanal, hand-crafted prefixes and automate aggressively. With Netbox and Ansible, it's incredibly easy to leverage the netbox.netbox.netbox_prefix module. The following example will grab a /24 from 10.99.0.0/16:

 1- name: "Example Ansible Playbook"
 2  connection: local
 3  hosts: localhost
 4  gather_facts: False
 5  tasks:
 6    - name: "Get next available prefix"
 7      netbox.netbox.netbox_prefix:
 8        netbox_url: "{{ netbox_url }}"
 9        netbox_token: "{{ netbox_token }}"
10        data:
11          parent: "10.99.0.0/16"
12          prefix_length: 24
13        state: present
14        first_available: yes

It's extremely rewarding to design, deploy, and automate an IP design in this manner - and you'll find that automation is considerably easier if what to automate is well-defined.

Manage Linux patching with Ansible and Netbox!

Sun, 07 Apr 2024 00:00:00 -0900

Patching all of my random experiments took too much of my free time, so I automated it

This is a pretty cheesy thing to do, but over the years it became more and more time-consuming to maintain all the different deployed workloads and infrastructure.

Requirements

With all system design, it's best to consider all relevant needs ahead of time. Given that this is a home lab, I decided to adopt an intentionally aggressive, but theoretically viable in production approach:

Nightly patching
Nightly reboots
No exempt packages
Distribution-agnostic, it should patch multiple distributions at once
This workflow should execute consistently from-code

Iteration 1: Ansible with Jenkins

The earliest implementation I built here had the least refinement by far. Here I tied Jenkins to an internal repository:

To leverage this, I started out with an INI inventory, but it quickly became problematic. I wanted a hierarchy, with each distribution potentially fitting multiple categories. This became pretty messy pretty quickly, so I moved to a YAML Inventory:

 1debian_machines:
 2  hosts:
 3    hostname_1:
 4      ansible_host: "1.1.1.1"
 5    hostname_2:
 6      ansible_host: "2.2.2.2"
 7ubuntu_machines:
 8  hosts:
 9    hostname_3:
10      ansible_host: "3.3.3.3"
11apt_updates:
12  children:
13    debian_machines:
14    ubuntu_machines:
15nameservers:
16  children:
17    hostname_2:
18    hostname_3:

This allowed me to simplify my playbooks and inventory by making "groups of groups", and avoid crazy stuff like taking down all nodes for an application at once. We'll use nameservers: as an example here:

 1---
 2- name: "Reboot APT Machines, except DNS"
 3  hosts: apt_updates,!nameservers
 4  tasks:
 5    - name: "Ansible Self-Test!"
 6      ansible.builtin.ping:
 7    - name: "Reboot Apt Machines!"
 8      ansible.builtin.reboot:
 9- name: "Reboot nameservers"
10  hosts: nameservers
11  serial: 1
12  tasks:
13    - name: "Ansible Self-Test!"
14      ansible.builtin.ping:
15    - name: "Reboot nameservers serially!"
16      ansible.builtin.reboot:

The serial: 1 key instructs the Ansible controller to only execute this playbook on one machine at a time, so DNS continuity is preserved.

Retrospective

I had several issues with this approach, but to my surprise, Linux patching and actual Ansible issues haven't cropped up at all. With most mainstream distributions, the QC must be good enough to patch nightly like this.

I did have issues with inventory management, however. To update the Ansible inventory, I could deploy as-code, which was nice, but it was still clunky. If I deployed 5 Alpine images in a day, I want them to automatically be added to my inventory for maximum laziness.

I also quickly discovered that maintaining Jenkins was labor-intensive. It's a truly powerful engine, and great if you need all the extra features, but there aren't many low-friction ways to automate all the required maintenance, particularly around plugins. I was able to update Jenkins itself with a package manager, but it seems like every few days I had to patch plugins (manually).

Iteration 2: Ansible, Netbox, GitHub Actions

I'll be up-front - for parameterized builds, GitHub Actions is less capable.

It has some pretty big upsides, however:

You don't have to maintain the GUI at all
Logging is excellent
Itegration with GitHub is excellent
Pipelines are YAML defined in their own repository
Status badges in Markdown (we don't need some stinkin' badges!)

This workflow has been much smoother to operate. Since the deployment workflow already updates Netbox, all machines are added to the "maintenance loop after first boot.

I was really surprised at how little work was required to convert these CI pipelines. This was naive of me - ease of conversion is the entire point of CI pipelines, but it's still mind-boggling to realize how effective it is at times.

To make this work, I first needed to create a CI process in .github/workflows on my Lab repository:

 1name: "Nightly: @0100 Update Linux Machines"
 2
 3on:
 4  schedule:
 5    - cron: "0 9 * * *"
 6
 7permissions:
 8  contents: read
 9
10jobs:
11  build:
12    runs-on: self-hosted
13    steps:
14    - uses: actions/checkout@v3
15    - name: Execute Ansible Nightly Job
16      run: |
17        python3 -m venv .
18        source bin/activate
19        python3 -m pip install --upgrade pip
20        python3 -m pip install -r requirements.txt
21        python3 --version
22        ansible --version
23        export NETBOX_TOKEN=${{ secrets.NETBOX_TOKEN }}
24        export NETBOX_API=${{ vars.NETBOX_URL }}
25        ansible-galaxy collection install netbox.netbox
26        ansible-inventory -i local.netbox.netbox.nb_inventory.yml --graph
27        ansible-playbook -i local.netbox.netbox.nb_inventory.yml lab-nightly.yml

This executes on a GitHub Self-Hosted runner in my lab with a Python Virtual Environment. The workflow will run a clean build, every time - by wiping out the workspace prior to each execution. No configuration artifacts are left behind.

With GitHub Actions, all processes are listed alphabetically, you can't do folders and trees to keep it more organized. I developed a naming convention:

1{{ workflow_type }}: {{ description}}

To keep things sane.

From there, we need a way to point to Netbox as an inventory source. This requires a few files:

requirements.txt is the Python 3 Pip inventory - since things are running in a virtual environment, it will only use python packages in this list.

 1###### Requirements without Version Specifiers ######
 2pytz
 3netaddr
 4django
 5jinja2
 6requests
 7pynetbox
 8
 9###### Requirements with Version Specifiers ######
10ansible >= 8.4.0              # Mostly just don't use old Ansible (e.g. v2, v3)

The next step is to build an inventory file. This has to be named specifically for the plugin to work - local.netbox.netbox.nb_inventory.yml:

1plugin: netbox.netbox.nb_inventory
2validate_certs: False
3config_context: True
4group_by:
5  - tags
6device_query_filters:
7  - has_primary_ip: 'true'

Not featured here - The API Endpoint and API Token directives are handled by GitHub Actions Secrets, and therefore don't need to be in this file.

This file is pretty straightforward. It indicates that we should use Netbox tags to develop our inventory, and we can assign multiple tags in the netbox application to each Virtual Machine. I also added the has_primary_ip directive - if a machine doesn't get an IP address for some reason, it won't try to reach that VM and patch it, causing late night failures.

Here's a preview of the Netbox application with these tags:

Refactoring the Ansible playbooks was hilariously easy. The Netbox inventory plugin prepends the group_by field onto the group, so all I had to do in each playbook was prepend tags_ to each name. Here's an example:

 1---
 2- name: "Apt Machines"
 3  hosts: tags_lab_apt_updates
 4  tasks:
 5    - name: "Ansible Self-Test!"
 6      ansible.builtin.ping:
 7    - name: "Update Apt!"
 8      ansible.builtin.apt:
 9        name: "*"
10        state: latest
11        update_cache: true
12- name: "Apk Machines"
13  hosts: tags_lab_apk_updates
14  tasks:
15    - name: "Ansible Self-Test!"
16      ansible.builtin.ping:
17    - name: "Update Apk!"
18      ansible.builtin.apt:
19        available: true
20        upgrade: true
21        update_cache: true

After that, the CI tooling just takes care of it all for me!

Retrospective

I'm going to stick with this method for a while. Netbox tagging makes inventory management much more intuitive, and I can develop tag "pre-sets" in my deployment pipeline to correctly categorize all the stuff I deploy. Since it's effectively documentation, I have an easy place to put data I'll need to find later for those "what was I thinking?" moments.

I'll be honest - behind that, I haven't really given it much thought. This approach requires zero attention to continue, and it happens while I sleep. I haven't gotten any problems from it, and it allows me to focus my free time on things that are more important.

10/10 would recommend.

Abstracting DNS Record Management with Ansible and Jinja 2

Sat, 06 Jan 2024 00:00:00 -0900

Synchronizing properly implemented DNS zones is, to put it lightly, a real chore:

Creating forward DNS entries, e.g. A, AAAA, CNAME. These names are used to resolve to resources.
Creating reverse DNS entries, e.g. PTR.
Creating DNS entries that define the zone, e.g. SOA, NS

For a system to behave properly, your forward and reverse entries need to be identical, but software like BIND/Unbound rely on zonefiles that don't connect the two. Many information systems / DNS zones exist with improperly implemented reverse DNS, or partially implemented forward DNS asymptomatically for a time. Certain events (e.g. CA validation, discovery, implementing IPv6) can bring things to the forefront if ordinary network management practice doesn't.

For this post, we'll first work on abstracting the DNS zonefile - ensuring that a user can deploy zonefiles conformant to a standard - and then we'll illustrate how that can be used with Netbox to automatically populate DNS entries from Netbox.

Abstracting the zonefile here will achieve a few goals - but the file size is guaranteed to be longer than if you simply managed the zone files from source. Here are some advantages:

This pipeline ABSOLUTELY MUST establish forward and reverse records from the same data!
This pipeline must test zonefiles, and avoid installing them if they aren't good (prevents outages)
This pipeline must establish documentation standards for a DNS zone (abstract the standard)
This pipeline must scale to support large quantities of DNS zones / records
This pipeline must be easy to use, even with inexperienced DNS administrators (we can't have it all be on the shoulders of that one guy who can safely make DNS changes)

To achieve this, we'll first establish a YAML schema and Jinja2 template to structure the data. Here's the YAML schema:

 1zones:
 2    - name: filename
 3      zonename:
 4      soa:
 5      settings:
 6        ttl:
 7        serial:
 8        refresh:
 9        retry:
10        expiry:
11      nameservers: []
12      reverse_zones:
13        ip4:
14        ip6:
15      records: [{ "name": "", "type": "", "addr": ""}]

There are also some subtle differences between IPv4 and IPv6 reverse zones, so in this case, we're going to use three Jinja2 templates (in the Gist below).

It also assumes that there's a dedicated classful prefix for each DNS zone. This isn't always true for more complex deployments, but they can also do stuff like buy Infoblox.

I have also included a GitHub Action in the gist, because it provides a good place to demostrate best practices (e.g. using venv) in one compact place. If you want to install generated zone files on-premises, you can run this on a self-hosted runner with an Ansible inventory group (e.g. nameservers).

It's still a little clunky, the next step should help with that (harvesting DDI information from Netbox IPAM data).

Build and Consume Alpine Linux vSphere Images

Sun, 24 Dec 2023 00:00:00 -0900

Deploying Linux for the impatient

If you've ever wanted to just "test something out really quick" in a live environment, Linux distributions have always been generally lightweight, but that's not the only implicit requirement for experimentation.

A Linux IaaS distribution should be:

Reasonably secure (basic hardening applied, fewer packages == fewer vulnerabilities)
Light on disk usage (shortening deployment times)
Light on system resources, e.g. CPU/Memory
Flexible (supports a package manager with a wide ecosystem of packages)

Package flexibility is usually the compromise made here - but when you're deploying programmable code, container images and virtual environments like Python's venv should be able to bridge some gaps.

Alpine Linux focuses on these goals - but doesn't compromise on automation. Combined with a dynamic inventory bootstrapping process, it's relatively straightforward to bring Alpine's APK ansible module into play to build any extra software on a new machine.

Customizing and building the ISO

First, let's upload the ISO to a datastore:

Let's create a new Virtual Machine. We'll attach the ISO to it

Target any storage and compute as preferred. SSD datastores will be faster, of course.

vSphere 8.0 Update 2 doesn't have a preset for Alpine Linux, and the guest OS options are important - it defines what paravirtualized hardware is available:

Ensure that VMXNET 3 and PVSCSI are both available. The "New Network" will become the default port-group assigned to the template.

CPU/Memory are mostly irrelevant, as the deployment pipeline can customize afterwards - and this OS doesn't need much in terms of resources:

Select the Alpine "Datastore ISO" and enable "Connect and Power On" for the assigned CD/DVD drive:

Start the machine - it'll boot to a command prompt very quickly. Log in as root:

The installation guide indicates to use the setup-alpine script, and follow the prompts:

The majority of setup here is extremely simple - because it's not installing a bunch of software. GUIs are also possible after the installation is complete - but it does defeat the point.

Instead of rebooting as instructed, shut the virtual machine down and delete the disk drive:

Note: The shutdown process isn't installed with Alpine, and the following command does execute a graceful shutdown!

1shutdown now

Start the machine up - we'll want to add some quality of life improvements to this machine like guest tools:

1{{ insert favorite editor here }} /etc/apk/repositories

Remove the # from the line ending in /alpine/v{{ version }}/community and save.

Per Alpine's guide, install and enable open-vm-tools:

1apk add open-vm-tools open-vm-tools-guestinfo open-vm-tools-deploypkg
2rc-service open-vm-tools start
3rc-update add open-vm-tools boot

Once running, ensure that guest power actions are available:

Personally, I like testing, so instead of powering off the VM, I use the guest action to ensure everything is working. Either way, shut the VM down.

Hit "Actions" on the VM, or right click it, and select "Clone → Clone as Template to Library":

Select whatever storage backing and content libraries are preferable at this point. It won't take long to clone in. Delete the old VM whenever it makes sense, I usually do so after testing a deployment:

Note: Set "Power on VM after creation" - this will clone _extremely quickly and boot even faster.

Modifying the deployment pipeline

The deployment pipeline code itself is available here. I've made some modifications from previous versions:

GitHub Actions now supports the choice type, which means we can select UUIDs. There isn't a way to build a "friendly name" mapping. We achieve this by creating a "lookup dictionary" with the friendly name as a key and a UUID as the value. This list will need to be populated via data collection (featured below).

First, we'll need to find out what the UUID of the template and cluster are. Here's an example to collect the required information. The UUIDs of system resources (and this template) are only available via the API. Use this information to form the parameters.yml file created in the GitHub Action workflow, e.g. datastore, cluster, folder

Adjusting and running this workflow will allow an engineer to populate the previous workflow and expose vSphere assets to further deployment automation!

For reference, this machine deployed in about 3 seconds on a shared SSD (iSCSI):

The GitHub workflow takes >2 minutes to complete, but the workflow it's attached to has manual wait step:

Apollo 13 "Failure is not an option", and how non-engineers misinterpret it

Sat, 25 Nov 2023 00:00:00 -0900

Failure is not an option!

It might surprise you to know that this quote wasn't real - it feels legendary, but was never said by Gene Kranz. It was written up for the film.

The aerospace engineering discipline isn't really something everybody gets to experience, so it makes sense that "spicing things up" for the movie would be generally accepted as reality.

When you create a program (or release a new capability), it makes perfect sense to get all excited and release it as soon as you feel it's "done" - but this is just an example of how IT/Computer Science is relatively young compared to other engineering disciplines.

With more traditional engineering disciplines, testing is a key aspect to deployment and design. Everything is tested for safety. Concrete is thoroughly tested before integration in bridges and structures. Most pickup trucks are tested to their listed tow capacity.

This isn't a perfect ideal world, however. Bridges still fail, and in this case companies didn't follow the SAE J2807 standard until forced (Toyota: 2011, General Motors: 2015, Ford: 2015, Dodge: 2015).

Industry-wide changes take time

Here's why: It's expensive to re-tool in the physical world. NASA just straight up didn't have the option, so they compensated by accounting for as many potential scenarios as possible, at the expense of cost. That's what "Failure is not an option" was intended to reflect. Everything is tested and planned ahead of time, and the mission systems didn't try anything truly new.

Engineering is the practice of taking learned experiences and codifying them, ensuring that the same mistake doesn't happen twice. The safety codes and engineering artifacts we use in the physical world are "written in blood" - many structural engineering practices were learnt from a loss of life, it's why they're so important.

I don't think anybody has died due to an email not getting through, but I'd counter that the same practices are much easier to execute in IT and therefore should be followed. IT is a relatively young engineering-adjacent discipline, and the standards for performance are relatively low, albeit always increasing.

Here's a rough estimate of each engineering discipline's age:

Chemical Engineering (~1800s AD)
Civil Engineering (BC, formalized in the 1700s AD)
Electrical Engineering (1700s AD, formalized in the 1800s AD)
Mechanical Engineering (BC, formalized in the 1800s AD)
Software Engineering (1960s AD)

More recent engineering disciplines fit in these families, and one could argue (correctly) that while they are younger, they benefit from the preceding disciplines and the broader body of knowledge. This is particularly true in the field of aerospace.

Systems Engineering practitioners have collected a number of practices together to integrate new technologies and disciplines in the SEBoK - which essentially forms a "starter kit" of practices and protocols for developing new solutions. The SEBoK is an excellent (albeit overwhelming) place to procure methods for continuous improvement, either as a team or individually.

Don't fear failure, understand it

Across all of these disciplines, we see a common pattern around failure; the natural reaction to failure is to avoid it. Humans don't want to be associated with failure, and this reflex must be overridden to be a successful engineer.

I'd like to provide an example of good failure analysis instead of harping on past failures - my concern here is that any controversy may get in the way of the idea I want to convey - which deviates from the practice of failure analysis somewhat.

Washington State DOT's analysis of the Tacoma Narrows bridge failure is an example of well-executed failure analysis.

In this case, the structure was too rigid - "common sense" would tell us that if a bridge is extremely strong, it won't have any issues standing up to high winds.

Applying failure analysis to IT

It's important that we learn from these shortcomings and integrate solutions into future designs. Typically, this is where "system integration" comes into play - as a product is validated for release, all known tests are applied to it to ensure that failures don't recur. The NASA engineers supporting Apollo 13 didn't try anything new on the mission system (Apollo 13 itself). NASA tested all solutions thoroughly with the ground crew, astronauts, and QA engineers before rollout was ever considered an option.

The Apollo program was extremely expensive compared to most of our IT budgets, but we're almost always testing software. Failure Analysis practices are trivial with software debugging and mature unit testing, and eventually we're going to have to perform at the standards held by traditional engineering disciplines.

Example - a maintenance window backfired

We've all been here before - let's say that spanning tree did something unexpected during a maintenance window and caused unexpected downtime.

The first and most effective aspect of failure analysis (at least for our careers) is to provide a compelling narrative. We need to invert the human reflexive reaction to failure and encourage interest over punitive behaviors. Writing a complete and compelling narrative both ensures that people will react more positively to the occurrence and provide confidence that due diligence will be performed to ensure it doesn't happen again.

Sure, it'll always happen again with STP in some way, but other materials have common patterns and properties too. We didn't stop using aluminum because it isn't as strong as steel or as good of a conductor as copper; instead we learned its strengths and weaknesses, applying the solution judiciously. In this case, we need to prove that we will apply the solution more judiciously as well.

Second, gather all possible data on the time of the outage. Don't try to filter it yet, and don't react slowly. Anything that can record system data is valuable here (telemetry in particular) - so automatic gathering is extremely valuable.

Third, find ways to locate precursors and the failure itself. This part should be automated and attached to any CI pipelines for the future, "set it and forget it" is the best way. As this practice evolves, a solution develops incredible mass and manually executing every failure analysis unit test after every change will quickly become tedious and slow.

Why?

The pressure to follow this pattern is only going to grow in the future. The previous decade's reliability standards were hilariously low compared to the quality of technology and service today - just look at the standards people hold us to. Instead of fearing this trend, let's analyze it and find ways to improve. It'll give us a competitive edge in the future.

As with Apollo 13, our greatest failures drive our greatest successes.

Internet Load Balancing with pfSense

Sun, 08 Oct 2023 00:00:00 -0900

With full-time remote work, internet outages transform from a nuisance to a real problem

Prior to the pandemic, "working hours" were typically considered fair game by internet service providers to schedule necessary system maintenance. It's unrealistic to expect perfect uptime from any service provider - as the saying goes:

Schedule maintenance on your equipment before your equipment schedules it for you!

ISPs are terrible about this, mostly because "old and stable" means customers receive reliable service. Eventually that trusty Toyota Corolla dies, though, causing severe customer impact.

I'd suggest taking matters into your own hands here. The technologies involved in internet load balancing are fairly complex, but if you follow a known formula it's doable for most tech-savvy users.

Internet Load Balancing

Load balancing network traffic is traditionally a separate domain from routing and firewalling, with most of the general industry focus centering around Server Load Balancing (SLB). An Internet Load Balancer (IPv4) needs to provide the following functions reliably:

Monitor each available path for viability with some form of end-to-end test.
Evenly (or with a ratio) balance new flows between each available path.
Track related sessions and place "affinity" to a specific path, ensuring that protocols like RTP + RTCP work
NAT Outbound traffic for its relevant link (IPv4)

To clarify, this doesn't cover SD-WAN and why it's more effective. Per-packet assessment and FEC lead to a much higher quality user experience and can achieve much cleaner ratios than what I provide below, but home users typically have high individual bandwidth with their internet services and like the concept of using them to their fullest. If the connectivity options at home are sufficiently mismatched or slow, it would be worthwhile to take SD-WAN solutions into consideration.

Let's establish an example topology, and cover the tunables that will provide a "good enough" WAN load balancing solution that centers around minimizing impact to remote work:

In this scenario, we'll just assume that one's wireline and one isn't to make things easy to explain. The transport doesn't really matter much, but it simplifies any documentation from here on out.

First, let's assign a new interface for the second internet link, and configure it for DHCP. This menu can be found under Interfaces ⇾ Assignments:

Note: Ensure that "Block private networks and loopback addresses" and "Block bogon networks" are checked. This is a WAN link, after all.

When using DHCP, the secondary WAN link should automatically install a "gateway", but it won't load balance just yet. We need to create a Gateway Group to enforce load balancing policies, and then assign it as the default gateway for things to take effect:

Now, let's create monitoring IPs so pfSense can periodically test for packet loss or latency on that link. The following menu is available by editing the service provider gateway under System → Routing → Gateways:

I'd suggest using the Service Provider's DNS services or an anycast DNS provider you don't typically use for the monitor addresses. pfSense installs a static route via that WAN for the monitor address, which means that it'll go down with the WAN link.

Note: Duplicate this IP and create a DNS monitor with Uptime Kuma if you want to monitor per-provider reliably. It's quick and easy!

This is all that's required, assuming that you want to get the most even load balancing possible. Here are a few tunables that may apply to more specific scenarios:

pfSense won't load balance asymmetric link speeds by default. If the interface speeds are different, you will need to create a policy-based routing rule (Firewall → Rules → LAN → New rule), and modify the Advanced Option Gateway:
While editing the gateway (System → Routing → Gateways), look for an Advanced setting labeled Weight. This will allow you to set a ratio between gateway groups, e.g. 2:1.
pfSense provides a simplified persistence mechanism that will pin each client to a specific WAN link. This is important, particularly if your remote work situation requires comprehensive use of voice and video services like Zoom or Teams. Please note that this feature will impact load balancing evenness to a great degree!
pfSense provides gateway status under Status → Gateways, but I haven't found a way to externally track those statistics via SNMP.

The Internet and its scaling issues

We've created a problem with global routing that is just plain fascinating.

Network Address Translation allows us to "spoof" our internal private networks with multiple public prefixes. This both solves and creates problems - as an upside, we're able to leverage WAN redundancy with service-provider public IPv4 addressing somewhat easily. This matters, because the public internet routing table currently can't support a designated prefix for every home network, and we're already experiencing internet availability issues due to route propagation:

In August 2014 we celebrated "512k Day" by enjoying a number of network outages related to TCAM capacity worldwide: link
The Internet Service Providers (ISPs) at the time managed to postpone this issue by "carving" TCAM, re-allocating ternary memory from other purposes to postpone the doomsday clock. This provided a capacity of 256,000 routes, but the clock was ticking. This bought ~ 5 years of time, and this was generally enough time to bump up capacity and lifecycle hardware.

Now, we have a new problem.

IPv4 routes consume 64 bits (8 bytes) of memory each assuming that no hardware optimization is used (you can store the number 1-32 as a 5-bit integer, but route lookups would be a multi-pass operation / require a lookup table), resulting in an internet routing table size of 4 Megabytes on 512k day, or 6 Megabytes on 768k day. It doesn't sound like much, but TCAM is designed for fast lookup and is somewhat limited in capacity.

IPv6 requires 256 bits (32 bytes) of storage per prefix, but more cleanly summarizes. Apples-to apples at a million routes would be 8 Megabytes (IPv4) + 32 Megabytes (IPv6), or 40 MB of TCAM.

Most of the absolute latest networking hardware is up to the task, but this is also with decades of hacks and best-practice engineering optimizing it. If I, as a household, establish my own /64, it's not that much of a problem, but every other network doing so would result in a table exponentially larger than hardware today can handle. This generally violates the design principle of "prefix summarization is hiding useful information," but it's driven by hardware limitations (as it always has).

The Future (IPv6) Solution

Interestingly enough, IPv6 is well-suited to this solution, and simple. Endpoints typically have tons of compute resources available for simple tasks like internet load balancing - but the client software isn't quite up to snuff. A dynamic IPv6 network leverages Router Advertisements and DHCPv6 to configure host devices with DNS and IP addresses, and there is nothing restricting multiple routers from advertising multiple prefixes over the same network:

Well, nothing except our own internal limitations, and client software. This would require a client device to automatically test each "path" and decide which one to use for a given application. We're not quite there yet, but the key elements are in place to guarantee a much higher service quality than our core and home routers can execute.

Retrospectives

While researching this topic, I discovered a few things that might be good for budget-conscious or hands-on users:

You don't need the biggest internet plan from each service provider. While 2 500 Megabit plans are definitely not going to be equal to a gigabit service, a typical household only uses a few megabits at a time. Right-sizing your internet services will save some serious cash, and may be cheaper than the single provider plan!
Capped services can be reduced to a lower ratio, or shut off entirely when the cap is reached. This approach is particularly appealing if services in your area are capped, because bandwidth caps are a tiny fraction of your link speed, and pfSense will average out your ratio rather effectively with more diverse usage.
Purchase an appliance with at least four ethernet ports! If a second service provider makes sense, it's entirely possible that a third may become an option.
If your ISP provides notice of maintenance, it's trivial to disable a gateway temporarily(System → Routing → Gateways → Edit):
Site-to-Site VPNs will need to be pinned to a specific WAN link via static routes, or by using dynamic tunnel IDs (not IP address identities!)
- Transport within a service provider will typically have much higher available bandwidth and lower latency than transport crossing multiple ISPs. If a site is important, try to match the service providers on both sides and run a tunnel per service provider for best results.

Handoff to Day-N Automation with vSphere Content Libraries and Netbox

Sat, 30 Sep 2023 00:00:00 -0900

The challenge with build automation is too much convenience

Think about it. If it's easy to compose and deploy workloads, it's also easy to develop sprawl, and a good system designer would have methods in place to mitigate that.

In a previous post I covered how to deploy vSphere VMs with Ansible and the Automation Value Proposition that comes with it:

Providing this capability to a company as-is is hazardous. Ask the following questions, in rough order of priority:

How do we track decommissions/unused machines?
How do we track who owns / uses what?
How do we track what OS images are end-of-life?
How do we track resource consumption (e.g. IP usage) and avoid re-using addresses?
How do we track certificates?

VMs also don't do much good without customization, unless you're comfortable handing those root credentials to whomever wants them.

System Integration

Linux heads live for this type of work - we return to the Unix design principles where a system or subsystem should excel at a single task instead of solving all possible issues at the expense of quality.

Let's explore a multi-system integration:

For this example, we'll re-implement the previous VM build process, but orchestrate it with GitHub Actions. I'll provide a gist at the end of this post.

I don't keep my vCenter exposed to the internet, so there will be some preparation required for this Action to function. We're using several prerequisites, install them first:

1python3 -m pip install aiohttp pynetbox
2ansible-galaxy collection install vmware.vmware_rest netbox.netbox

This Action leverages parameterization heavily, with Ansible relying on variables injected from GitHub to the virtual environment (venv). It provides a little "quiz" that will let consumers define attributes about the deployed machine, e.g. vCPU count and memory. Any input sanitization should be done by Ansible in this context.

Once a VM deployed, the vmware_rest module returns the virtual machine's Managed Object ID (MOID). We can use that to get operational data about the VM via VMware Tools.

Ansible keeps all of this data as registered variables for future utilization. Now, we have to put the data somewhere persistent. Netbox is a valuable tool for documenting information assets, but it can also be used as an Inventory. We can dump all the information about the VM into netbox rather easily, and pave the way for further customization seamlessly.

Note: I excluded the Guest Customization play in this version of the deployment script. It hasn't been particularly stable across 8.x releases with my automated testing, either failing completely with a Service Unavailable or crashing vCenter. It is possible, however, to change IP addresses, install packages, copy artifacts with Ansible after the fact. Customization via Ansible might even be a better approach in more complex deployments.

Circumventing Coders block and starting a new project

Sat, 26 Aug 2023 00:00:00 -0900

It's difficult to start a new software project

Documentation

Depending on how a software project starts, it can either be the easiest or the hardest aspect of a new project.

Documentation suffers from a similar issue, so a good place to get things moving would be to simplify the basics of repository management. Here's a number of things that should be in a Git repository:

1.gitignore
2CHANGELOG.md
3README.md

Each of these things can be simplified in some way, and will make your future life easier. Let's start with the easy stuff:

.gitignore is easily templated by your source control provider. At this point, it's smart to include one of their templates, this feature has really grown over the years.
- GitHub provides templates for common development setups, and integrates it into their new repository wizard.
- GitLab does too.
CHANGELOG.md takes very little effort to start, but is difficult to apply to an existing project.
- Keep a changelog provides an excellent template and guidance on how to effectively write changelogs.

README.md is the core of your project's documentation, and deserves separate mention because it's a powerful tool to organize how a project should function. Spend plenty of time on this part!

Since README.md is the page that renders by default in SCM, the objective should be to provide everything a user needs to consume your software. I personally prefer to outline how the software should function and use it as a reference when writing the actual code. Here's a decent starting point:

 1# {{ Name the Project }}
 2
 3## Goal(s)
 4
 5{{ Write what your project should do }}
 6
 7## Overview
 8
 9### Validation
10
11{{ Write how functional software should be evaluated at an end-to-end level }}
12
13### Unit Testing
14
15{{ Write how functional software should be evaluated at a component level }}
16
17## HOWTO
18
19{{ Indicate how the software should be used. Provide examples, fix it later when the functional code revises your intricate plans }}
20
21## Software Dependencies
22
23{{ Include what the software will need to run. Update as you `include` new libraries}}
24
25## Contributors
26
27{{ Set up a place for software contributors to put their names. It might encourage participation }}

CI

Continuous Integration is not easy to establish, but pays off over time as it catches small issues that appear throughout development. Since the testing strategy is written in plain language, CI tool setup is ideally started soon after the documentation. For a given CI tool, e.g. GitHub Actions or Jenkins, common practices can be templated for re-use.

...Then start writing code

From here, it's going to be easier to begin authoring software from an outline. Start by writing out your plan ("pseudo-code") in comments, defining class structures if applicable.

Infrastructure automation is typically a play-by-play implementation of an operating procedure, which won't necessarily need object-oriented coding. In this case, the same approach still works - transpose the operating procedure as comments, and implement it as code.

Hindsight

After some practice, it quickly becomes easier to start a new software repository. Structuring a software project quickly becomes important after only a few hundred lines of code - so if you're stuck, put some work in on the structure, it'll get the process moving, and the effort spent multiplies itself as the project grows.

Most Git providers also support repoository templates - use this feature to create a form of "starter kit", and copy it whenever a new project is created to safe time.

Why wait? Eventual Consistency and Reliability

Sun, 16 Jul 2023 00:00:00 -0900

Patience is tough when deploying automated code; Here's why it is important

Reliability-centric infrastructure engineers need to focus on careful, procedural, validated workflows; the systems we're responsible are simply too important to casually "toss" infrastructure requests at a common API gateway.

We can't really avoid automation either. Here's an example workflow:

Easy, right? There are a few issues with simply coding against this workflow:

We should define a standard format for BGP Peers to make the process re-usable
We should define what information we should send to the BGP Peer
We should define what we expect the change to be
We should test to ensure the expected change occurred (ideally before rolling out to production)
We should revert the change if it doesn't produce the exact result we want

Infrastructure-as-Code

Now - the modified workflow may seem complex and require some level of acclimation. Computers don't mind processing thousands of rows of data, or even millions - humans are a little more error-prone at that scale. Let's offload some of those tasks to the computer like this:

Implementing Infrastructure-as-Code in this case achieves several benefits at once:

Engineers end up with a "spec" defining a router or device without having to compile it from the configuration
New implementations of a given standard can expose "build against the spec" interfaces to make revising infrastructure trivial
The implementation process for a given change can be standardized across engineers and continually improved upon
Everything is carefully logged by the CI tool automatically

The Downsides

For the purposes of this post, let's ignore the debate about abstraction and obfuscation, and examine why eventual consistency matters to achieve this goal.

Infrastructure Engineering

Infrastructure engineers have wildly different values from a typical developer. In a nutshell:

Measure twice, cut once doesn't work as a principle (didn't check enough times)
Slow is safe, and safe is fast
There are permissible and impermissible times to perform infrastructure work
- This imposes time limits on work, which violates principles #1 and #2

We have some problems here. Company leaders want to minimize downtime, and enforce aggressive maturation cycles. Once the gear stops falling apart, the biggest danger to availability quickly becomes the infrastructure engineers themselves. This leads to shortening of maintenance windows, which leads to rushed work, which then leads to more pain.

I'd like to propose a different workflow.

Network Assurance

Let's try a new workflow:

In this world, we shift focus from the pressures of change execution to the change itself. The procedure itself should exist as-code (and ideally automated); we want to leverage a common concept in trades, cognitive loading.

Picture your mind as a physical workspace. All people are less efficient with a cluttered workspace. Instead of the past year's unfinished projects, the cognitive loading originates from stressors within the environment:

Am I going too fast?
Am I missing anything?
Do I have enough time to finish?
What if I missed something?

Early in the IT industry's maturation cycle, IT leadership pushed for the implementation of Standard Operating Procedures to act as a guide while executing a change, dramatically improving reliability outcomes.

Complexity is factorial in nature, and our human brains (a mental workspace) do handle this problem well, up to a point. Once we overwhelm our engineers, that's when mistakes happen - we need to leverage our computers to help with that. This is why we implement the procedure itself as code - the engineers construct the programmatic instructions themselves and continually improve on it with source control and peer review tooling (pull requests).

Demanding that we do things fast detracts from this, engineers should focus on the procedure and sequence of events when planning changes. This shifts their mental workspace to focus on delivering reliability.

It's not about the code, it's just another example of using a computer to better engineer solutions.

Python Virtual Environment Setup in Jenkins

Tue, 04 Jul 2023 00:00:00 -0900

Python's Virtual Environments Feature presents a unique opportunity with CI/CD tooling - where users congregate to publish and execute code of their own devising. Developers might use different versions of the same package (creating dependency conflicts), or they might simply need a newer version of a package than is on the system by default.

Or, with security-centric implementations, developers don't get root access, which prevents package installation outside of userspace.

The Virtual Environments feature allows users to set up a container of sorts (albeit very limited) that permits installing of ephemeral packages (and specific package versions) to prevent dependency conflicts.

Prerequisites

Set Jenkins Global Parameters

Under Jenkins -> Manage Jenkins -> System, configure the Shell executable to /bin/bash. It's blank by default (Bourne Shell):

Install Packages (example from Debian 12)

1apt install python3-venv python3-virtualenv

Consuming Virtual Environments in a Jenkins Job

Consuming a virtual env (venv) in Jenkins is now relatively straightforward. Here's an example from my Certificate Checker

1python3 -m venv checkcerts_env
2source checkcerts_env/bin/activate
3python3 -m pip install requests ruamel.YAML pyOpenSSL fqdn
4python3 check-certificates.py -f certs_list.json

With this example, I create a virtual environment (fresh every time) named checkcerts_env. It's not as necessary to create a venv with a unique name inside the Jenkins workspace if it's cleaned on run, but I prefer the extra safety.

The source command changes python3's environment to checkcerts_env. You can verify immediately after with:

1(checkcerts_env) jenkins@localhost:~/workspace/certificate-tracker$ which python
2/var/lib/jenkins/workspace/certificate-tracker/checkcerts_env/bin/python

From there, pip can install packages unique to this ephemeral workspace. Keep in mind this will slow execution down quite a bit, which is a fair trade for reliable outcomes.

Mellanox `nmlx5_core` driver `4.23` issues on ESXi 8.0 Update 1

Wed, 28 Jun 2023 00:00:00 -0900

Problem Inventory - Mellanox Driver Update on ESXi 8.0u1 causing network virtualization issues

After installing ESXi 8.0 Update 1, some issues start to appear with affected nmlx5_core adapters:

Delayed / Failed IP discovery on VLAN-backed segments, even within the same host. Once in the ARP cache, no issues persist
Delayed / Failed IP discovery, IP allocation failures on VLAN trunked port-groups, even within the same host. Issues persist even after IP discovery is established
Overlay encapsulation offload failures:
- ICMP with any payload size will function bidirectionally via Edge Transport Nodes / FRRLinux machines, but TCP and UDP will not
- All overlay traffic encapsulated by a vSphere host flows correctly between workloads on the sane NSX overlay segment
- All overlay traffic encapsulated by a vSphere host flows correctly between segments on the same NSX distributed router

These issues are seen on the following hardware models:

MCX4121A-ACAT firmware revisions 14.25 and 14.32

These issues are experienced with the upgrade to vSphere 8.0 Update 1, which includes the following updated driver:

nmlx5-core 4.23.0.36-8vmw.800.1.0.20513097

This driver from NVIDIA ships with support for both Bluefield SmartNIC and ConnectX Generation 5 network adapters as one package, and rolling back to a previous release of ESXi 8 with the previous driver (nmlx5-core 4.22) immediately resolves all overlay issues

Resolution

UPDATE: This problem has been resolved with ESXi 8.0 update 1c

API Conversations and Why They are Important

Sun, 25 Jun 2023 00:00:00 -0900

API Interactions are designed to be easy

Believe it or not, the IT infrastructure industry is trying to make things easier by building API access out.

Programmatic interfaces are a new mental model competing for brain-space with GUI and CLI implementations; we need to play to its strengths:

Parsing output from a computer automatically
Easy batch entry
Ensuring that a thing configured matches the thing requested
(Usually) easy pass/fail responses based on HTTP response codes

API Conversation Structure

RESTful, SOAP, and NetCONF interfaces all interleave the concept of a conversation with the Hypertext Transfer Protocol (HTTP) standards:

Identify Yourself
Ask for things
Get a Response

NB: Most API implementers (vendors) will implement API standards very loosely!

NB: RESTful interfaces will be used for the examples in this post, as it's quickly becoming the most common.

Authentication

Authentication is where most new users get stuck. It's complicated, but usually an API provider will also leverage an SDK to simplify the authentication process when you "graduate" to a programming language.

This only covers what a client has to do to perform API work - implementing an API (and authorization with it) is considerably more complex.

Basic Authentication

The title says it all, this authentication schema just uses Base64 encoding (ASCII-formatted binary) to place your username and password in an HTTP header:

1Authorization: Basic {{ Base64 String }}

If this approach makes you feel a little uncomfortable, it should. This is not a secure way to execute commands; it puts your credentials at risk. There are a few ways to mitigate the security risks:

Ensure that any endpoint you interact with uses strong cryptography and has a valid certificate
- Postman and cURL do half of that by default
- If it's an API you manage, issue it a client-recognizable certificate and tune cryptography upwards
Try using a "read-only" account unless a change is required
Use Basic Authentication as a method to establish a session token

Most new API sessions will at least start using Basic authentication, so these guidelines will apply unless client certificate authentication is used.

Here are some examples of client authentication use:

cURL

1curl -u {{ username }} -p {{ password }} https://{{ api_endpoint }}/get_stuff
2curl --header 'Authorization: Basic {{ string }}' https://{{ api_endpoint }}/get_stuff

Python 3

 1import requests
 2import sys
 3
 4try:
 5    # This example is to generate a bearer token with Cisco's Firepower Threat Defense
 6    do_api_url = "https://{{fmc_ip}}/api/fmc_platform/v1/auth/generatetoken"
 7    # The Requests library supports converting to Base64 from a tuple to keep things simple
 8    do_api_request = requests.request(
 9        "POST",
10        url= do_api_url,
11        auth=(username, password)
12    )
13    do_api_request.raise_for_status()
14except requests.Timeout:
15    sys.exit("TCP Timeout!")
16except HTTPError as e:
17    sys.exit("HTTP Error Found! " + do_api_request.status_code + " " + str(e))

Token Authentication

Most API Providers will require you to use safer authentication for continued requests. This is a good thing - but it does add of work. Usually, Basic or Certificate authentication is used to establish a timeboxed token for future authentication.

These tokens have several forms:

Bearer Token: Simple. They'll use the Authorization header per the HTTP standard:
- `Authorization: Bearer '{{ token }}'
JSON Web Token: Standardized and much more robust. Base64 encodes a JSON payload. It supports signing of all requests and provides context in all future requests, which makes things easier to parse on the server side
I'll just call this third category weird stuff. Vendors loosely follow standards, so they'll typically have headers that are specific to their platform. The mechanics remain the same, but the header name won't be Authorization

Here are some examples:

cURL

 1# GitHub follows the Bearer standard
 2curl --request GET \
 3--url "https://api.github.com/octocat" \
 4--header "Authorization: Bearer YOUR-TOKEN" \
 5--header "X-GitHub-Api-Version: 2022-11-28"
 6# VMware vSphere uses `vmware-api-session-id` as an API key
 7curl --location --globoff 'https://{{vsphere_vcenter}}/api/content/library/{{vsphere_base_images_library}}' \
 8--header 'vmware-api-session-id: {{vsphere_key}}'
 9# Cisco FirePower uses `X-auth-access-token` as a custom header
10curl --location --globoff 'https://{{fmc_ip}}/api/fmc_config/v1/domain/{{domain_uuid}}/devices/devicerecords' \
11--header 'X-auth-access-token: {{auth_token}}'

Python 3

 1# Cisco DNA Center uses `X-Auth-Token` as a custom header
 2import requests
 3import json
 4import sys
 5
 6url = "https://sandboxdnac2.cisco.com/dna/intent/api/v1/site"
 7
 8payload = {}
 9headers = {
10  'Content-Type': 'application/json',
11  'X-Auth-Token': '{{ token }}'
12}
13
14try:
15    response = requests.request("GET", url, headers=headers, data=payload)
16    response.raise_for_status()
17    print(response.text)
18except requests.Timeout:
19    sys.exit("TCP Timeout!")
20except HTTPError as e:
21    sys.exit("HTTP Error Found! " + do_api_request.status_code + " " + str(e))

Verbs

In API Terminology, clients (and servers) should identify what type of operation they intend to execute with a matching HTTP Method:

GET: Read only actions, typically without a submitted payload.
POST: Frequently (mis)used. Can be read only, can be a non-idemopotent change, or it could be an idempotent change. By standard, POST methods create an object in a tree or change the state machine - but nearly every vendor defaults to this method.
PUT: Meant to be a "safe" method that either updates an existing record, or creates a new object. Usually, if an API provider uses this verb, they're using it properly.
PATCH: PUT is idempotent, which is safe from an API perspective, but not from a production service perspective. It will wipe and recreate a service - the PATCH verb indicates a messier but production safe action.
DELETE: Idempotent, but it does delete a resource, so I'd be careful considering it production-safe.

Why?

Infrastructure operators benefit from programmatic interface usage, but the pattern differs.

Researching secure, well-managed authentication methods leaves a lot of room for improvement with the commodity resources available today, and a security engineer with even some basic API knowledge can quickly and easily secure resources with these rules. It's easy to imagine how firewalling an API service can quickly become secure:

1allow user any GET under /api/v3/healthcheck
2allow user admin PUT under /*
3deny user off-net any under /*
4deny user any DELETE under /*

Essentially, any load balancer can become a highly granular firewall for an API.

APIs are easy to parse due to their dictionary formats. Prior to their inception, network engineers had to write screen scrapers like Scrapli and "guess" what value in a given table has what meaning. YAML and JSON formats allow language-native association between values not normally present in a tab-separated table like a routing table.

The most important advantage to API automation, though, is for change safety. No matter what discipline you follow, invasive IT work is completed at night and with a strict schedule - one that does not promote thoroughness. Leverage APIs to check if your system is healthy - only you as the engineer know how to do that - and make change windows less stressful.

Some Tips

jq is a command-line utility that parses JSON - and lets you navigate or prune it to relevant data. Think of it like grep for JSON.
If you do complex or scaled work with Python, check out tqdm to build progress bars for your work.

Escape Platos Cave to build better IT infrastructure

Sun, 09 Apr 2023 00:00:00 -0900

Let's be honest, most IT engineers are autodidacts out of necessity. Our industry isn't mature compared to most of the professions that exist today. Computer Science was first offered as a degree in 1962 in the US, and Network Science is flatly ancient, but generalized IT infrastructure support only became a schooled profession much more recently than that.

Early IT engineers were typically self-reliant trade professionals that entered the field with a pioneer mindset, bringing their prior professional experiences. Technology was in its infancy, and there was no shortage of mountains to climb. Resiliency is a given for these folks, because there was no "Googling" before Google, and everything was new.

We saw another wave of IT engineers enter the field (myself included) in the mid-'00s/'10s. Industry education like Cisco Networking Academy became widely available in this time period, resulting in more well-rounded individuals without sacrificing a sense of self-reliance. It's our culture to teach ourselves, but we're OK with learning in a classroom too.

Now, we see new entrants on the field. IT degrees are available in many universities, and the quality of education is good. Will we all be replaced by the "smartphone generation"?

Why does this matter?

The values of stoicism and self-education are embedded deeply in the IT professions, but it's important to understand its weaknesses. Autodidacts don't receive the benefits of "periodic curriculum review" because there's no curriculum; focus is experiential.

Stoicism can be harmful, too. IT engineers specialize in finding flaws in technology systems, anything created by an engineer will be evaluated ruthlessly for problems.

When we introduce IT automation, we invite a toxic level of self-criticism with any code or solutions we create. The IT industry grew in parallel with computer science, but we need to learn a few lessons the easy way.

Plato's Cave

Let's talk about Plato's Allegory of the Cave. Plato attempts to produce an example of synthesizing Form by isolating a real object from its projection.

Plato created a hypothetical scenario where some individuals are imprisoned from childhood in a cave, unable to see the outside world. The details on execution don't matter much in this case, because it's not a guidebook on how to implement a real experiment - that would be inhumane. They can't see the outside world for whatever reason, but light sources are external, and have their own natural phenomena. Plato goes further and states that puppeteers can create shadows and create new stimuli for the prisoners.

Humans are amazing and adaptable, but this works to our disadvantage here. The prisoners optimize to their surroundings, developing complex narratives about the lights and shadows they perceive - but they cannot observe the true phenomena. These complex narratives become ingrained in our sociocultural foundation, and for all purposes, it becomes reality.

What do you think happens when the prisoners are released? Would they be overjoyed to experience sunlight, astronomy, other Greek wonders?

Socrates doesn't think so. Excessive stimuli introduced too quickly will force an individual back to the comfort of their own beliefs, resulting in an immediate rerouting back to that cave. A prisoner in this hypothetical scenario is forced to leave and cannot return until they've explored the world at least a little.

When this individual returns to the cave to tell others about their experience, they notice that he's blind (simply unaccustomed to cave darkness) and overwhelmed with new information. Long story short, they all agree to kill anyone who forces them out of their cave of psychological comfort.

Caves at Work

We're self-critical autodidacts, isolated by the limits of our own experiences. This doesn't reflect on capability at all, and the resilience that comes with the profession has a great deal of value.

How do we see our own value and capability, then? Most IT infrastructure work doesn't have a direct result, it enables others to create opportunity and value. It's not directly observable in day-to-day work, but we will see shadows of it in our cave of comfort.

We need to get out of that cave. IT Automation is just the most current example because it merges the CS and IT professions - which we thought were incompatible.

Don't kill the messenger

The first step to improvement is to remove that pact to wipe out any individual who is introducing new ideas. New stimuli are good, and capable IT engineers should have no problem interfacing with new ideas or technologies - it's what we live for. There's no reason to fear new ideas and technology in a field borne of pioneering and exploration. Don't be the firefighter in Fahrenheit 451.

Don't hold ideas too tightly either - we live in an industry where the technology cycles at a dizzying rate. What was right yesterday may not be right tomorrow, or it might be right forever. Learn to feel comfortable evaluating ideas for validity and usefulness. Be about as attached to them as the ol' company laptop - if it's a good one, keep it, if it doesn't, turn it in after a few years.

Remember that everyone's uncomfortable

Instead of wiping out threats to your team, I'd suggest being supportive of others instead. Avoid personal attacks, and use language to shift focus towards refining a technical solution when you peer review code.

Embrace new Qualia

Here are two completely different forms of qualia:

We as humans bring a unique combination of skills and knowledge with us everywhere we go.

Learn to enjoy experiencing new things, it's the entire point of our industry, and nobody can truly predict what an individual can create when new things are added to a collective experience.

This isn't exclusive to IT, and shouldn't be - while further suggestions are IT-specific, I'm also specifically saying to seek qualia outside the IT field as well.

Peer Review / `pull request`

Fortunately for us, git already exists and is used heavily both by IT engineers and software developers alike due to the "DevOps" movements.

This post is already too long to cover creation of a branching strategy for a software project, which is its own subject.

Instead, I'd like to suggest something. If you think your code is bad, start peer reviewing others. Enterprise Source Control Management (SCM) platforms include the concept of a Pull Request (Examples) as a method to invite others to take a look at their code.

Use peer reviewing opportunities as a way to pick up new qualia and improve your coding style! I'd recommend opening up a distribution list (or posting somewhere, anything works really) with a list of individuals interesting in participating in peer review. Ask questions! Encourage those who opened their ego to punishment by sharing what's essentially their own thought processes for everyone to see - they probably aren't comfortable with it either.

Your mission, as an enthusiastic peer reviewer, should be twofold:

First, strive to create an environment where individuals want to present their code for peer review. This doesn't have to be prescriptive, if it takes donuts to make it fun, then use donuts. Avoid any language or patterns that might make it painful or difficult to receive feedback. Remind others that their contribution is valuable as are they.

Second, try to learn from others. Branching strategies, coding styles, all that you participate in with a pull request is valuable. It doesn't need to be rushed, because you're not studying for a certification - you're participating in collective, experiential learning.

Remember that we as humans bring a unique combination of skills and knowledge with us everywhere we go.

Automate Cisco IOS/IOS-XE documentation with Ansible (with examples)

Sun, 26 Mar 2023 00:00:00 -0900

Note: This post integrates several automation tools at once. My objective is to provide some well-documented, concrete examples of executable Ansible Playbooks, D2 Diagrams, and best practice to illustrate ways to make good documentation easy.

Note: All code examples in this blog post are structured in a format to be complete and executable. They don't necessarily represent best practice (e.g. including passwords), but are intended as a functional starting point for Ansible beginners

Ansible Setup

Welcome to the messy world of network automation! If you're familiar with Ansible itself, there are a few things you'll need to understand to effectively use the tool:

Validate Idempotency: First and foremost, Ansible network automation differs the most from main-line systems automation here. The responsibility for making a module idempotent, e.g. safe for re-execution requires a substantial amount of effort, so you'll often see restrictions on module use. Testing any modules used in an enterprise should be a matter of practice, and it applies doubly here.
Authentication is weird: Credential storage is more complex with Ansible network modules, with many vendors implementing things their "own way". It's being reined in, and the examples used here are IOS-XE, which follows the standard.
Secrets: In production, use Vault or something similar to store credentials. Most Network OSes don't support SSH key authentication, which is the typical method for Ansible.

Ansible Inventory

Things get opinionated on how to store an inventory. Ansible's documentation is going to be the most up-to-date and best, but it boils down to a few choices:

ini vs yaml format. It's your file, use what's easiest. I consistently choose yaml for the easy object hierarchies
Organization: Build a document plan to decide what hosts will be installed where, and how to name them. Whiteboard-grade is the best

In this example, we'll take a new yaml inventory and apply it. There's more to the inventory, but the top-level hierarchy is the same. yaml and json files start with a top-level dictionary, and yaml prefers a start-of-file line (---):

 1---
 2cml_ios_xe_machines:
 3  hosts:
 4    AnsLabIOSXEv-1:
 5      ansible_host: "10.7.3.10"
 6  vars:
 7    ansible_network_os: "ios"
 8    ansible_user: "admin"
 9    ansible_password: "cisco"
10    ansible_become: "yes"
11    ansible_become_method: "enable"
12    ansible_connection: "ansible.netcommon.network_cli"

Let's cover what each of these fields does, and how it serves us:

cml_ios_xe_machines serves as a root dictionary key. Use this name under the hosts: directive in an Ansible playbook to automatically pick up that inventory object.
hosts: defines the inventory group members.
- AnsLabIOSXEv-1: defines an individual node as part of this inventory group. It'll use this name, and if it's resolvable via DNS, you don't need to specify anything else.
  - ansible_host: enables a user to set a specific IP address for a given node. Since this particular box is in CML and is frequently wiped/replaced, I just keep the records here.
- vars: enables an Ansible administrator to automagically set variables available on every instance invoking this inventory. They don't have to be required by the module, we can set foo: 'bar' here if we want.
  - ansible_network_os: instructs Ansible on which NOS the node runs, instead of relying on automatic detection.
  - ansible_user: needs to be set somewhere, as authentication via SSH is required, even on CML nodes.
  - ansible_password: Don't do this in real life, full stop. There are methods for acquiring secret input in an Ansible playbook that are better, but they aren't self-contained.
  - ansible_become: instructs Ansible that an additional command will be required to achieve escalated privileges.
  - ansible_become_method: instructs Ansible on what command achieves escalated privileges.
  - ansible_connection: instructs Ansible on what driver to use for CLI interaction, in this case paramiko

Configuring Cisco IOS with Ansible

After firing up the CML nodes required for this lab, I was quickly reminded of how frustrating the old Cisco IOS CLI really is - let's build a tool that will configure interfaces worth diagramming.

We're going to run into issues here - there isn't a cisco.ios module for Layer 3 802.1q subinterfaces. This is resolvable with a Jinja2 template, but is no longer idempotent. To use this template, simply place it in the same directory as the Ansible playbook:

1{% for i in ios_interfaces %}
2interface {{ i.name }}
3  encapsulation dot1q {{ i.tag }}
4{% endfor %}

This is a simple example of Jinja2 looping - the {% for i in ios_interfaces %} stanza receives input from the Ansible playbook as part of vars. The iterator (i) in this example will contain whatever is stored in vars (a dictionary), and can be invoked without dictionary traversal, e.g. {{ i.name }}.

Let's try a playbook to configure some interfaces:

 1---
 2- name: "IOS Gather Facts"
 3  hosts: "AnsLabIOSXEv-1"
 4  connection: network_cli
 5  gather_facts: yes
 6  vars:
 7    ios_interfaces:
 8    - name: GigabitEthernet4.100
 9      tag: 100
10      ipv4:
11        address: 10.10.100.1/24
12    - name: GigabitEthernet4.101
13      tag: 101
14      ipv4:
15        address: 10.10.101.1/24
16    - name: GigabitEthernet4.102
17      tag: 102
18      ipv4:
19        address: 10.10.102.1/24
20    - name: GigabitEthernet4.103
21      tag: 103
22      ipv4:
23        address: 10.10.103.1/24
24    - name: GigabitEthernet4.104
25      tag: 104
26      ipv4:
27        address: 10.10.104.1/24
28  tasks:
29    - name: "Set Interface Config Sheet"
30      template:
31        src: ios_subinterfaces.j2
32        dest: '{{ inventory_hostname }}.conf'
33    - name: "Apply Layer 2 Configuration"
34      cisco.ios.ios_config:
35        src: '{{ inventory_hostname }}.conf'
36    - name: "debug"
37      debug:
38        msg: '{{ item }}'
39      with_items: '{{ ios_interfaces }}'
40    - name: "Set Interface IPs!"
41      cisco.ios.ios_l3_interfaces:
42        config:
43        - name: '{{ item.name }}'
44          ipv4:
45            - address: '{{ item.ipv4.address }}'
46      with_items: '{{ ios_interfaces }}'

With this playbook, I invoked the specific node AnsLabIOSXEv-1, because the playbook itself includes unique data. Ansible also supports injecting variables from a separate file, e.g. ansible-playbook {{ playbook_name }} --extra-vars "@file.json".

The vars structure is doing most of the heavy lifting here - defining each interface for configuration in a concise, readable format. This may follow stricter formatting, making it the "Model" portion of the Model-View-Controller architecture.

If you have troubles coming up with a structure for your data, or constant revising causes issues, check out Openconfig for vendor-neutral, well-organized models.

tasks is where the actual work happens:

template is Ansible's Jinja2 driver, and will take vars and combine them with any file template chosen by the user.
cisco.ios.ios_config is the "fall-back" method for device configuration, used here due to a lack of subinterface support. It's not idempotent.
debug allows an engineer to print any internal variables provided either by automatic data collection or by the user directly. I included it here to provide an example of "debugging" tools, and because I like Ansible playbooks that "repeat back" to me prior to executing.
cisco.ios.ios_l3_interfaces enables idempotent configuration of Layer 3 interfaces, with some restrictions on name (shortened names are not idempotent!)

Running the playbook is fairly straightforward (truncated):

1ansible-playbook set_interfaces_anslabiosxev-1.yml
2TASK [Set Interface IPs!] ******************************************************
3ok: [AnsLabIOSXEv-1] => (item={'name': 'GigabitEthernet4.100', 'tag': 100, 'ipv4': {'address': '10.10.100.1/24'}})
4ok: [AnsLabIOSXEv-1] => (item={'name': 'GigabitEthernet4.101', 'tag': 101, 'ipv4': {'address': '10.10.101.1/24'}})
5ok: [AnsLabIOSXEv-1] => (item={'name': 'GigabitEthernet4.102', 'tag': 102, 'ipv4': {'address': '10.10.102.1/24'}})
6ok: [AnsLabIOSXEv-1] => (item={'name': 'GigabitEthernet4.103', 'tag': 103, 'ipv4': {'address': '10.10.103.1/24'}})
7ok: [AnsLabIOSXEv-1] => (item={'name': 'GigabitEthernet4.104', 'tag': 104, 'ipv4': {'address': '10.10.104.1/24'}})
8PLAY RECAP *********************************************************************
9AnsLabIOSXEv-1             : ok=5    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The above command is a re-run, but note how there are 2 changed tasks. This is an indicator that a change is not idempotent.

Generating IOS Documentation with Ansible

Now, we finally have a router to automatically document.

As part of the playbook process, Ansible will try and gather a number of details about the system it intends to change. These facts aid idempotency but also provides important context in a way that's easy to tap for automation engineers. Our previous instructions in the inventory provided Ansible important context, so most of the data required to auto-document will already be there.

This playbook will collect all possible supported information about a node and print it without logging in to the node. Handy, isn't it?

 1---
 2- name: "IOS Gather Facts"
 3  hosts: "cml_ios_xe_machines"
 4  connection: network_cli
 5  gather_facts: yes
 6  tasks:
 7    - name: "Collect Data"
 8      cisco.ios.ios_facts:
 9        gather_subset: 'all'
10      register: 'ios_deadbeef'
11    - name: "Print Data"
12      debug:
13        msg: '{{ ios_deadbeef }}'

Note that the Jinja2 escaping ('{{}}') is required to print a variable with the debug -> msg combination.

I'd rather not hand off Ansible playbook logs to other engineers, clients, and executives as network documentation, though. Let's try to make something pretty with Jinja2 and D2, starting with a definitely not confusing .j2 file:

 1'{{ inventory_hostname }}': {
 2  icon: 'router.png'
 3  interfaces: |md
 4{% for i in ansible_facts.net_interfaces|dict2items %}
 5{% if i.value.ipv4|length %}
 6    * {{ i.key }}: {% for ii in i.value.ipv4 %}{{ ii.address }}/{{ ii.subnet }}{% endfor %}  ({{ i.value.type }})
 7{% endif %}
 8{% endfor %}
 9  |
10}

This Jinja2 template will be universal to any router it's executed on, and print all interfaces with ipv4 addresses. We're also using |dict2items because the key for i isn't visible otherwise. It formats the dictionary like so:

1{
2    "key": "name"
3    "value": {
4        "key": "stuff"
5    }
6}

Rendering a D2 document will require several stages:

Collect data from IOS node (implicitly collected by gather_facts: yes)
Template the D2 definition
Render the D2 definition

After all that work, this is all that's required to auto-document an IOS node:

 1---
 2- name: "IOS Gather Facts"
 3  hosts: "cml_ios_xe_machines"
 4  connection: network_cli
 5  gather_facts: yes
 6  tasks:
 7    - name: "Use facts to draw a node diagram"
 8      template:
 9        src: 'node_diagram.j2'
10        dest: '{{ inventory_hostname }}.d2'
11    - name: "Render node diagrams!"
12      ansible.builtin.shell: 'd2 "{{ item }}" "{{ item }}.png" --sketch'
13      with_fileglob: '*.d2'

With the tool provided, it'll generate a network diagram node (as always with the glorious Crayon Visio stencils):

Conclusions / Lessons Learned

Network Engineers tend to be pretty rigid about standards - which lends itself to this type of automation. The examples in this blog post are designed to scale - it'll generate hundreds of images if given hundreds of IOS nodes, saving unimaginable hours of time.

In the future, I'd expand the scope of templating far beyond a simple diagram. Auto-generating HTML, Markdown, Microsoft Word documents (if you must) are all well-supported by Jinja2 - it just needs to be available as text somehow. There's a distinct beauty to providing solution delivery complete with unique, use-case customized documentation every time.

I can see full operating manuals and nighttime procedure runbooks being delivered to IT consumers using the simple methods outlined here - it's a bright outlook for tomorrow's IT service quality.

Document Networks as Code with D2!

Sat, 18 Mar 2023 00:00:00 -0900

Let's do something a bit more stereotypical when we talk about IT diagramming - building Network Diagrams with D2.

D2's OSS layout engines (Dagre, ELK) are both centered around hierarchical layouts, which was a bit part of why we needed to hack D2 to get vSphere diagrams to draw. Here, it quickly becomes our best friend - Network Engineers will arrange just about anything into a tree hierarchy if permitted - and our network topologies reflect that.

First, let's load some stencils! We'll use Crayon Visio for this exercise. The Cisco SAFE Architecture Toolkit listed here is a bit more serious and visually appealing.

If you want to see why I did it, check out this lovely firewall icon:

This is a Visio stencil, so it'll take a bit of work to convert. The vssx extension for Microsoft Visio is a Zip-compressed file, and the images are listed under visio/media, and convert with transparency using GIMP (this is not fun). Drop the exported png files (it's a raster image, not a vector) in a folder, in this case dwg/.

Drawing Networks

Let's try it out!

 1direction: right
 2Firewall {
 3    icon: dwg/firewall.png
 4}
 5Outside Router {
 6    icon: dwg/router.png
 7}
 8Inside Router {
 9    icon: dwg/router.png
10}
11Inside Router -- Firewall -- Outside Router

Now, let's try a hierarchy (Layer 3 Clos Fabric):

 1direction: right
 2AS65533 {
 3  spine-1: "as65533-spine-1" {
 4    icon: "dwg/router.png"
 5  }
 6  spine-2: "as65533-spine-2" {
 7    icon: "dwg/router.png"
 8  }
 9  spine-3: "as65533-spine-3" {
10    icon: "dwg/router.png"
11  }
12  spine-4: "as65533-spine-4" {
13    icon: "dwg/router.png"
14  }
15  leaf-1: "as65533-leaf-1" {
16    icon: "dwg/switch.png"
17  }
18  leaf-2: "as65533-leaf-2" {
19    icon: "dwg/switch.png"
20  }
21  leaf-3: "as65533-leaf-3" {
22    icon: "dwg/switch.png"
23  }
24  leaf-4: "as65533-leaf-4" {
25    icon: "dwg/switch.png"
26  }
27  leaf-5: "as65533-leaf-5" {
28    icon: "dwg/switch.png"
29  }
30  leaf-6: "as65533-leaf-6" {
31    icon: "dwg/switch.png"
32  }
33  spine-1 -- leaf-1: "eth1/1<->eth1/49"
34  spine-1 -- leaf-2: "eth1/2<->eth1/49"
35  spine-1 -- leaf-3: "eth1/3<->eth1/49"
36  spine-1 -- leaf-4: "eth1/4<->eth1/49"
37  spine-1 -- leaf-5: "eth1/5<->eth1/49"
38  spine-1 -- leaf-6: "eth1/6<->eth1/49"
39  spine-2 -- leaf-1: "eth1/1<->eth1/50"
40  spine-2 -- leaf-2: "eth1/2<->eth1/50"
41  spine-2 -- leaf-3: "eth1/3<->eth1/50"
42  spine-2 -- leaf-4: "eth1/4<->eth1/50"
43  spine-2 -- leaf-5: "eth1/5<->eth1/50"
44  spine-2 -- leaf-6: "eth1/6<->eth1/50"
45  spine-3 -- leaf-1: "eth1/1<->eth1/51"
46  spine-3 -- leaf-2: "eth1/2<->eth1/51"
47  spine-3 -- leaf-3: "eth1/3<->eth1/51"
48  spine-3 -- leaf-4: "eth1/4<->eth1/51"
49  spine-3 -- leaf-5: "eth1/5<->eth1/51"
50  spine-3 -- leaf-6: "eth1/6<->eth1/51"
51  spine-4 -- leaf-1: "eth1/1<->eth1/52"
52  spine-4 -- leaf-2: "eth1/2<->eth1/52"
53  spine-4 -- leaf-3: "eth1/3<->eth1/52"
54  spine-4 -- leaf-4: "eth1/4<->eth1/52"
55  spine-4 -- leaf-5: "eth1/5<->eth1/52"
56  spine-4 -- leaf-6: "eth1/6<->eth1/52"
57}

Note that I used the direction: right directive to make this readable on blog and mobile. direction: down and some custom padding would be more appropriate when describing a solution in a Visio/Print-style media.

Let's make this fun, and add externalities:

 1direction: right
 2AS65534: {
 3    firewall-cluster-1: "AS65534 Firewall Cluster" {
 4        icon: "dwg/firewall.png"
 5    }
 6}
 7Storage: {
 8    nfs-array-1: "General-Use NFS" {
 9        icon: "dwg/storage.png"
10    }
11    nfs-array-2: "PCI NFS" {
12        icon: "dwg/storage.png"
13    }
14}
15Compute: {
16    vsphere-cluster-1: "General Use vSphere" {
17        icon: "dwg/server-cluster.png"
18    }
19    vsphere-cluster-2: "PCI vSphere" {
20        icon: "dwg/server-cluster.png"
21    }
22}
23AS65534.firewall-cluster-1<->AS65533.leaf-1
24AS65534.firewall-cluster-1<->AS65533.leaf-2
25Storage.nfs-array-1<->AS65533.leaf-3
26Storage.nfs-array-2<->AS65533.leaf-4
27Compute.vsphere-cluster-1<->AS65533.leaf-5
28Compute.vsphere-cluster-1<->AS65533.leaf-6
29Compute.vsphere-cluster-2<->AS65533.leaf-5
30Compute.vsphere-cluster-2<->AS65533.leaf-6

If you click on the image, it will direct you towards a ELK diagram format with direction: down.

Conclusion

The automatic layout settings for D2 require a certain amount of tweaking to get just right, and as a topology gets "full" curating the layout becomes more difficult.

That being said, changing between formats (blog/web to a stricter canvas) only takes a few seconds, as opposed to hours. D2 (as a solution) excels in situations where a topology or solution may be fluid or repeatable, and it doesn't necessarily need to be presentation grade.

Terrastruct's layout engine (TALA) appears to have the highest quality output of the bunch - several ELK layouts crashed go while executing these examples, and Dagre as a layout engine is unmaintained since 2018. TALA does have the potential to routinely deliver presentation-grade solutions, and D2 ships with a variety of "freebie" resources to make it a good end-to-end solution.

D2 also performed exceptionally well with ridiculous large topologies, it's difficult to consume a full second of CPU time with a diagram, and it can run server-side with an API - Kroki implements one. If automatic documentation is a regular part of business, I'd recommend publishing the stencils to an object store or HTTP server and make it accessible to a system like Kroki - this effectively lets you auto-diagram with no user intervention.

Document vSphere as Code with D2!

Sat, 11 Mar 2023 00:00:00 -0900

In a previous post we tested methods to illustrate documentation as-code with Terrastruct's D2 language.

Documentation is good for infrastructure engineers in a variety of scenarios practically speaking, but we often forget the value of building consumer confidence. This may vary based on culture! My experiences are almost exclusively in the United States! We should advance and protect the profession by providing a visible internal infrastructure, documented in a way that is visually appealing and easy to understand. If we provide this to stakeholders it will help them understand the value of what they're paying for.

Different infrastructure "silos" participate at different levels of documentation with this cause, with networkers focusing primarily on diagrams and avoiding the "why" questions, where systems designers tend to carefully document the use cases and user perspective over detail on functionality. These guiding values indicate where a design priority is made; automating documentation is a powerful tool to shift some of that cognitive load away from an author's strong points to their weak spots. The object of tools like D2 is to eliminate time spent on aligning and formatting diagrams, and allowing an engineer to document more.

The scope of this post will be to produce a code-driven diagram describing my vSphere Lab Infrastructure. In the future, we'll even be able to collect the required information via vSphere's RESTful API; we must lay down a solid foundation first.

vSphere Stencils

Note: VMware is no longer maintaining the design stencils for VVD. The stencils provided are good, but will no longer be maintained by VMware to assist VI Engineers in design and planning their infrastructure deployments.

The Visio Library provided by VMware for infrastructure design is located on TenThirtyAM's GitHub Page, but the version that includes the SVG-formatted icons is located here. D2 natively supports SVG icons, so that's what we want to use.

D2's icon: directive requires a URL to access stencils from - it can be either local or a remote server (HTTP/S) this example will leverage VMware's provided stencil library with some level of workspace setup. Let's extract the stencils provided in vmware-stencils-main.zip/svg into a new folder, e.g.:

1dwg/
2diagram.d2

Placing the stencils into a separate folder will help keep the clutter under control. There are quite a few icons, and the author was kind enough to label all of them properly for us.

Adding a VVD icon becomes easy, for example:

1Enterprise PKI: {
2  icon: dwg/certificate-server.svg
3}
4Application: {
5  icon: dwg/application.svg
6}
7Application -> Enterprise PKI: Submit Certificate Signing Request

Documenting a Cluster

Now, vSphere deployments are much more complex. D2 is much more effective with large diagrams, where cultivating careful connector lines has swallowed more time than I'm willing to admit. Here's a way to depict a vSphere cluster with D2 - take note of how the object hierarchy is used to control object placement:

 1vSphere Cluster: {
 2    icon: 'dwg/vmware-cloud.svg'
 3    Compute Nodes: {
 4        icon: 'dwg/site-container.svg'
 5        compute_host_1: {
 6            icon: 'dwg/vm-server.svg'
 7            Attributes: |yaml
 8            Hostnames:
 9                mgt: compute_host_1.abc.co
10                vmo: compute_host_1_vmo.abc.co
11                vsan: compute_host_1_vsan.abc.co
12            Interfaces:
13                vmhba0: wwn
14                vmk0: 10.101.0.2
15                vmk1: 10.101.1.2
16                vmk10: 10.201.0.2,3
17            |
18        }
19        compute_host_2: {
20            icon: 'dwg/vm-server.svg'
21            Attributes: |yaml
22            Hostnames:
23                mgt: compute_host_2.abc.co
24                vmo: compute_host_2_vmo.abc.co
25                vsan: compute_host_2_vsan.abc.co
26            Interfaces:
27                vmhba0: wwn
28                vmk0: 10.101.0.3
29                vmk1: 10.101.1.3
30                vmk10: 10.201.0.4,5
31            |
32        }
33        compute_host_3: {
34            icon: 'dwg/vm-server.svg'
35            Attributes: |yaml
36            Hostnames:
37                mgt: compute_host_3.abc.co
38                vmo: compute_host_3_vmo.abc.co
39                vsan: compute_host_3_vsan.abc.co
40            Interfaces:
41                vmhba0: wwn
42                vmk0: 10.101.0.4
43                vmk1: 10.101.1.4
44                vmk10: 10.201.0.6,7
45            |
46        }
47    }
48}

Use the --sketch switch to set the formatting depicted here!

In this example, I used a YAML text block to draw everything because it's simpler and more visually appealing. I placed a container in a container to illustrate the text with icons, either by clustering them together or using connectors, e.g.:

 1compute_host_1: {
 2    icon: 'dwg/vm-server.svg'
 3}
 4compute_host_1 -> compute_host_1_attributes
 5compute_host_1_attributes: |yaml
 6    Hostnames:
 7        mgt: compute_host_1.abc.co
 8        vmo: compute_host_1_vmo.abc.co
 9        vsan: compute_host_1_vsan.abc.co
10    Interfaces:
11        vmhba0: wwn
12        vmk0: 10.101.0.2
13        vmk1: 10.101.1.2
14        vmk10: 10.201.0.2,3
15    |

Connectors are useful with this platform when you want to control the layout of your diagram. Keep in mind that the goal here is to quickly diagram a solution, not necessarily to make it the absolute best.

This solution now describes the compute cluster effectively, but it's not really sufficient to describe, say, a VCF stack. Let's do that:

 1direction: right
 2VMware Cloud Foundation: "VMware Cloud Foundation\nSDDC:\n{{ site }}-{{ stack }}-sddc.{{ domain }}" {
 3  icon: 'dwg/vmware-cloud-foundation.svg'
 4  Management: {
 5    icon: 'dwg/inftrastructure.svg'
 6    vCenters: {
 7        vCenter_1: "vCenter:\n{{ site }}-{{ stack }}-{{ wld }}-vc.{{ domain}}" {
 8            icon: 'dwg/vcenter-server.svg'
 9            Attributes: |yaml
10                version: 8.0patch-b
11                ipv4: 10.101.200.2/24
12                ipv6: 4000:20::2/64
13                size: super-duper-x-large
14                compute: {{ clustername }}
15                cpus: 400
16                memory: 4,000 GB
17                disk: 10 PB
18            |
19        }
20    }
21    NSX Managers: {
22        NSX_1: "NSX Mgr:\n{{ site }}-{{ stack }}-{{ wld }}-nsx00.{{ domain }}" {
23            icon: 'dwg/vmware-nsx.svg'
24            Attributes: |yaml
25                version: 4.1.1
26                ipv4: 10.101.200.96/24
27                ipv6: 4000:20::96/64
28                size: super-duper-x-large
29                compute: {{ clustername }}
30                cpus: 400
31                memory: 4,000 GB
32                disk: 10 PB
33            |
34        }
35    }
36  }
37  Monitoring: {
38    icon: 'dwg/alert.svg'
39    VROPS: "vROPS:\n{{ site }}-{{ stack }}-{{ wld }}-vrops.{{ domain }}" {
40        icon: 'dwg/vrops.svg'
41            Attributes: |yaml
42                version: 8.0.0
43                ipv4: 10.101.200.192/24
44                ipv6: 4000:20::192/64
45                size: super-duper-x-large
46                compute: {{ clustername }}
47                cpus: 400
48                memory: 4,000 GB
49                disk: 10 PB
50            |
51    }
52    VRLI: "vRLI:\n{{ site }}-{{ stack }}-{{ wld }}-vrli.{{ domain }}" {
53        icon: 'dwg/vmware-vrealize-log-insight.svg'
54            Attributes: |yaml
55                version: 8.0.0
56                ipv4: 10.101.200.192/24
57                ipv6: 4000:20::192/64
58                size: super-duper-x-large
59                compute: {{ clustername }}
60                cpus: 400
61                memory: 4,000 GB
62                disk: 10 PB
63            |
64    }
65    VRNI: "vRNI:\n{{ site }}-{{ stack }}-{{ wld }}-vrni.{{ domain }}" {
66        icon: 'dwg/vmware-vrealize-network-insight.svg'
67            Attributes: |yaml
68                version: 8.0.0
69                ipv4: 10.101.200.192/24
70                ipv6: 4000:20::192/64
71                size: super-duper-x-large
72                compute: {{ clustername }}
73                cpus: 400
74                memory: 4,000 GB
75                disk: 10 PB
76            |
77    }
78  }
79  Consumption: {
80    icon: 'dwg/consumption-plane.svg'
81    VRA: "vRA:\n{{ site }}-{{ stack }}-{{ wld }}-vra.{{ domain }}" {
82        icon: 'dwg/vmware-cloud-assembly.svg'
83            Attributes: |yaml
84                how-to-reach: it's SaaS
85            |
86    }
87  }
88}

I formatted this drawing to be optimal for blogging and mobile use, which follows a different convention than for workstations. The directive direction: right instructs D2 to render downwards if we don't provide any connectors. This ensures that eastward expansion is possible as connectors are built. The biggest change here for most of us will be the fact that we must give up a lot of control over layout in exchange for a higher yield per effort. Think of it like Python's black module, any color you like, as long as it's black.

When compiling this diagram, we're also using iconography to help solidify understanding, which creates an issue with padding. To manipulate dagre (the drawing algorithm used here), we can add the switch --dagre-edgesep 40 to request 40 pixels of padding between objects and make it fit cleanly.

Now, let's combine the diagrams by simply putting them in the same file:

Conclusion

It's fascinating to see more opinionated development for automated documentation. In many cases, the primary detractor for maintaining documentation is time. D2 attempts to build something simple and visually appealing that minimizes the effort required to build documentation.

D2 is also really fast, and the generated diagrams are tiny with vector exports. Complex diagrams are drawn in under a second with minimal compute resources - something that can't be said of Visio or Inkscape.

The only major gripe I have with the platform (when I stop wrestling with the fact that I don't have much control) is that the layout engines provided (elk, dagre) don't perform as well as the costware one (tala). This is how Terrastruct is monetizing their product, which makes sense - and the company does provide a great deal of public support for their product without any cost.

My advice if you're using it, like how it looks, but become frustrated with the layout: D2 doesn't lock you into their software in any way. Open the svg in your preferred Graphic editor and tweak it to your liking. This is the big benefit of open source software!

Diagram as Code with D2!

Sat, 04 Mar 2023 00:00:00 -0900

Documentation is always important, but always takes too much time

Ever had an issue where a new installation is completed, but there's just no time to update the ol' Visio diagrams?

Manually composed diagrams always possess a certain art to them, but the hours per unit of documentation isn't always worthwhile (particularly when prototyping a solution).

Deploying Infrastructure-As-Code should involve deployment of a mature, production-ready implementation; subjecting a coded prototype to rigorous and comprehensive testing prior to release contributes directly to end-to-end reliability. Infrastructure consumers demand ever-increasing improvements to reliability and features - mostly because they are being subjected to higher standards as well - which will require some changes to how infrastructure is delivered.

Some of these changes are positive! The inevitable result of this shift is the elimination of artisanal, hand-crafted infrastructure customized to perfection for customers, trading for a consistent, predictable infrastructure.

Consistent infrastructure without documentation, however, will confuse and deter aspirant consumers - confidence is key. I'd propose that we ought to explore completely automated ways of generated documentation in IT, and network/infrastructure diagrams are no exception.

Let's test out a new diagramming tool - D2. Here's a summary of the features it provides:

Installable via the Go package manager
Supports server-side rendering
Configurable Color Theming
Object Grouping (hierarchical)
SVG Export (no more setting and calculating your own canvas size!)
Intuitive connectors
Preview plugin for Visual Studio Code
Language API support
Custom object types (icons must be accessible via HTTP from the diagram server)

Deploying the tool

Installation instructions for D2 are listed in their GitHub repository - this is probably the easiest way to execute the installation:

1go install oss.terrastruct.com/d2@latest

Go's built-in package manager is in charge of the installation at this point, and the platform installs cleanly on Windows and Linux.

Drawing Diagrams

Let's start with a simple diagram:

1hola! -> como estas!

Visual Studio Code can provide a preview of this diagram in either Markdown or as a .d2 file, but GitHub-flavored markdown won't render it. This is still super useful, and seems to handle special characters really well. Let's try to render it:

1echo 'hola! -> como estas!' > diagram.d2
2d2 diagram.d2

This will produce an SVG:

D2 is extremely fast at rendering vector graphics, but it can support PNG and PDF output as well. The CLI interpreter detects the file output settings from the extension:

1d2 diagram.d2 diagram.png

It's slower, and consumes more bandwidth to draw PNG - vector graphics are extremely efficient, particularly with diagrams. We can apply themes as well:

1d2 diagram.d2 --theme=101

Custom themes are also supported.

Adding Icons

D2 supports usage of icons, but they have to be accessible via HTTP. Terrastruct provides an Icons Library to help out - and custom stencils would have to be stored somewhere for rendering (either as SVG or PNG).

Let's try drawing a diagram with icons:

 1'User': {
 2    icon: https://icons.terrastruct.com/essentials%2F365-user.svg
 3    shape: circle
 4}
 5'Internet access': {
 6    icon: https://icons.terrastruct.com/essentials%2F214-worldwide.svg
 7    shape: hexagon
 8}
 9'User' -> 'Internet access': {
10    label: wants
11}

Here, we introduced several new constructs. D2's language is very intuitive, with clear inferences from JSON and YAML - and we use key-value pairs to identify what we should draw. D2 supports hierarchical nesting as well. On the third section, we added a "connector" by using the -> directive between the two named objects:

It's also possible to attach object attributes by misusing the class shape type:

1'Router01': {
2    icon: https://icons.terrastruct.com/essentials%2F092-network.svg
3    shape: hexagon
4}
5'Router01 Interfaces': {
6    shape: class
7    'Ethernet1/41': 101.101.101.1/24
8}
9'Router01' -> 'Router01 Interfaces'

This tool has tremendous illustrative potential when built with automation. Let's imagine a server-side workflow that could integrate rendering of an object as part of the CI workflow.

Actually, instead of imagining it, let's try diagramming it instead:

 1# Object Definitions
 2'Continuous Delivery Stack': {
 3    'CI Tool': {
 4        icon: https://www.jenkins.io/images/logos/pixelart/jenkins-pixelart-256.png
 5        shape: hexagon
 6    }
 7    'Ansible': {
 8        icon: https://upload.wikimedia.org/wikipedia/commons/2/24/Ansible_logo.svg
 9        shape: hexagon
10    }
11    GitHub: {
12        icon: https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png
13        shape: circle
14    }
15}
16'Rendering': {
17    'Jinja2': {
18        icon: https://jinja.palletsprojects.com/en/3.1.x/_images/jinja-logo.png
19        shape: hexagon
20    }
21    'D2': {
22        icon: https://icons.terrastruct.com/assets/icons/d2-logo.svg
23    }
24}
25'Node Definition': {
26    'Object': {
27        icon: https://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/Official_YAML_Logo.svg/64px-Official_YAML_Logo.svg.png
28    }
29    'Object Facts': {
30        shape: class
31        name: Router01
32        baseline: '2023001'
33        'Ethernet1/41': '101.101.101.1/24'
34    }
35    'Node Configuration': {
36        icon: https://icons.terrastruct.com/aws%2FNetworking%20&%20Content%20Delivery%2FAmazon-VPC_Router_light-bg.svg
37    }
38}
39# Connectors
40'Continuous Delivery Stack'.'CI Tool' -> 'Continuous Delivery Stack'.'GitHub': 'Fetch Source Code'
41'Continuous Delivery Stack'.'CI Tool' -> 'Continuous Delivery Stack'.'Ansible': 'Execute Playbooks'
42'Continuous Delivery Stack'.'Ansible' -> 'Node Definition'.'Object': 'Get Node Facts'
43'Node Definition'.'Object Facts' -> 'Node Definition'.'Object'
44'Continuous Delivery Stack'.'Ansible' -> 'Rendering'.'Jinja2': 'Render Configuration'
45'Continuous Delivery Stack'.'Ansible' -> 'Rendering'.'Jinja2': 'Render Diagram Templates'
46'Continuous Delivery Stack'.'Ansible' -> 'Rendering'.'D2': 'Execute D2 Draw'
47'Rendering'.'Jinja2' -> 'Rendering'.'D2'
48'Rendering'.'Jinja2' -> 'Rendering'.'Node Configuration'

D2 proves to be an excellent method of automatically generating diagrams. The diagram above is automatically positioned, so it won't have the "polish" that a manually generated diagram would have, but it also takes a fraction of the effort. This is a game-changing tool; engineers should use this tool to generate and render solution documentation on-every IaC update, and then rely on manual documentation for higher-value work like executive presentations or sales pitches.

For the rest of the use cases, let's save time by automating work where that artisanal, curated experience doesn't help us. In future posts we'll examine ways to document infrastructure.

Enable ToFU (Trust on First Use) with OpenSSH

Sat, 11 Feb 2023 08:58:00 -0900

Ansible is a fantastic tool for Linux-based and NOS-based automation. Have you seen this error before?

1"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed."

When you use Ansible with SSH transport (as opposed to an API), you rely on the SSH key trust store to validate that the end-host is authentic. It's stored in ~/.ssh/known_hosts on every user profile on a Linux machine.

Users building new machines will run afoul of this tool - because the machine doesn't exist yet and should generate its own key, it's not particularly easy to add to an orchestrator's SSH trust store.

Fortunately, the designers of OpenSSH have us covered. The following stanza can be added to either ~/.ssh/config or to /etc/ssh/ssh_config, depending on how elaborate the existing configuration is:

1Host *  
2  StrictHostKeyChecking accept-new

The configuration enables the ToFU (Trust on First Use) model with OpenSSH. It'll assume that the first key for a given IP address or hostname is valid, and continue to trust it without majorly compromising security by disabling trust.

ToFU is not better than a well-established Web of Trust or any hierarchical authorization (like PKI). ToFU seeks to strike a balance between administrative overhead and convenience - which is probably acceptable when building new virtual machines.

Try using ToFU to bootstrap the machine and apply more secure methods of administration after that as a solution to any security concerns - in theory, those should all be handled by Ansible Inventory Groups in a way that's scalable and easy.

With the build steps covered here and managing an Ansible inventory file as code, the only remaining step is to join any newly built VMs to the existing inventory file in your Git repository!

Deploy vSphere VMs with Ansible!

Sat, 04 Feb 2023 09:10:00 -0900

In a previous post, we covered how to create a virtual machine from a VM template in vSphere using Python and the REST API as an example of service-agnostic methods to invoke infrastructure resources.

VMware's PowerCLI is a fantastic tool, but it's not for me. Tweaking or porting functionality from PowerCLI to other languages beyond Windows and PowerShell isn't supported. Technology professionals with more than a few years of experience are leery of code portability issues; standard hardware architectures today are the results of consolidation efforts and caused some quite painful transition points. Change safety features like idempotency or context awareness are important to me; I prefer the approach of "check parameters, check destructiveness, execute, test."

Engineers develop linguistic preferences as part of a normal progression throughout their careers. Bash/Zsh/Python/Perl and JSON/YAML/XML appear more intuitive to me as universal formats to store artifacts and execution code. BASIC-style languages like PowerShell/CLI are not for everybody.

It's possible to use Ansible to manage and deploy Virtual Machines now - the previously covered Python code leverages the same RESTful API as the Ansible modules. The Ansible modules shift responsibility for code maintenance away from internal teams. Let's not forget the cost of writing and maintaining custom code:

Let's take a look. We need to install the Ansible "Collection" to leverage the REST API (this does require the Python package aiohttp:

 1ansible-galaxy collection install vmware.vmware_rest  
 2Starting galaxy collection install process  
 3Process install dependency map  
 4Starting collection install process  
 5Installing 'vmware.vmware_rest:2.2.0' to '/root/.ansible/collections/ansible_collections/vmware/vmware_rest'  
 6Downloading https://galaxy.ansible.com/download/vmware-vmware_rest-2.2.0.tar.gz to /root/.ansible/tmp/ansible-local-346454pqg9aobo/tmpi1o29dho  
 7vmware.vmware_rest (2.2.0) was installed successfully  
 8Installing 'cloud.common:2.1.2' to '/root/.ansible/collections/ansible_collections/cloud/common'  
 9Downloading https://galaxy.ansible.com/download/cloud-common-2.1.2.tar.gz to /root/.ansible/tmp/ansible-local-346454pqg9aobo/tmpi1o29dho  
10cloud.common (2.1.2) was installed successfully

First, let's plan where automation will deploy workloads. Good work starts with good data - Ansible combines with a popular template tool (Jinja), enabling automation engineers to compile "pretty" reports from variables we collect during the execution phase:

The playbook will help any planning efforts with consistent infrastructure. At work, I'd recommend rendering an HTML page with these artifacts (Jinja can do that as well) periodically with a CI tool and publishing it to a web server for other teams to reference.

Let's use that information to deploy a virtual machine:

Correctly setting the session_timeout parameter in the playbook is critical. VM Deployment times depend on storage backing; the default timeout is 300 seconds. Deployment timeouts require adjustment to the equipment they live on - I set it to 1200 seconds to accommodate spinning disks.

This playbook is parameterized, with variables identified by double curly brackets, {{ variable_name }}. Ansible Playbook examples don't always provide what output the code will expect, a consistent issue with community-generated code.

Ansible supports variable injection from an external file or the command line; the best way to manage these input variables for self-service is to use file-based inputs. CI tools ship with APIs that allow you to invoke a pipeline with a JSON payload, and this method gives you an easy way to convert back and forth. I have provided an example JSON document to show what format the playbook wants.

To invoke extra variables in Ansible, use the ext-vars feature:

1ansible-playbook build_vm.yml "@parameters.json"

Jenkins has a parameter injection feature that makes this consumable as an end user as well:

Consumers perceive infrastructure's quality with a heavy bias towards convenience. We should construct easy ways to provision resources and enable creativity to boost positive perception of infrastructure services. Ansible's vmware_rest module combines with an execution engine like Jenkins to create an open method for deploying Virtual Machines, a pivotal step towards making infrastructure appealing.

Why Automate? VM Deployment with vSphere's REST API

Thu, 05 Jan 2023 17:10:00 -0900

VMware introduced RESTful APIs with their vSphere 7.0 release train; migrating from the old MOB APIs is a welcome choice. REST clients provide a powerful tool for automating processes, but it's important to embody reliable practices when interacting with infrastructure.

The neat thing about published and documented REST APIs is that you're no longer restricted to a specific tool set to automate - both a blessing and a curse.

In the following section, I describe how I prefer to develop reliable infrastructure automation with integrated checking. The code is here.

How Much Automation?

The industry often provides persuasive guidance on the fact that automation should be a conscious choice at work (usually to sell automation products). The title of this blog article is also indicative of this trend - imagine the difference in Google Search hits for "How much automation is good for me and my company". This "peer pressure" alienates people who haven't researched much on automation before they can resolve any subconscious issues present.

The issues I hear are typically from infrastructure engineers, and the concerns raised with automation are common and valid. That doesn't mean we're all off the hook, though - the issues raised are all solvable:

How Safe: Automation safety is a huge concern - and rightly so. Taking the wrong action rapidly produces disastrous results. This is the highest priority when developing an automation practice, so peer review fundamentals and executing proofs are so important.
Which Things: Discussions like "This highly custom and variable process can't be automated" indicate that a company's IT Architecture doesn't fit well with automation approaches - prioritizing repetitive solutions on automatable infrastructure should buy companies time to redesign themselves. This is particularly true in networking, where automation models and tools may require hardware upgrades to implement. Infrastructure has always moved slowly, but we stand to benefit from a redesign incorporating everything we've learned since the last time around (~2014 for most).
How Much: Do you think you'd know the difference between a company with one engineer coding it all with Python, a company that broadly uses Chef/Puppet/Ansible, or one that uses commercial automation tools exclusively?
Going "hard mode" may be the right choice, or it might not - it's a better choice to make IT work fluidly with the business.
A company that typically accepts tools as-is and doesn't modify them much (commercial farming, manufacturing as examples) would benefit more from vendor-provided automation
A startup trying to rush an initial product to market is going to "code it all".
Most of us live somewhere in the middle.

Automation Requirements

Let's assume that the agreement is set appropriately. I frame this as a critical ability for my home lab - creating disposable virtual machines to avoid suffering consequences for my mistakes. We start by constructing a plan or requirements for our code:

Deploy a vSphere Virtual Machine
Without OVFtool. OVFTool deployments are slow, even if originated from tools like vRealize Orchestrator within the data center
Leverage host-storage copying optimization if available (vSAN, NFS Server-Side Copy, VAAI, etc.)
Build a VM from a central image repository
If the VM isn't possible to build, perform best-effort checking prior to deployment
Validate that the VM is successfully deployed
Return a pointer to enforce further customization

The final requirements may be present in a canned solution, but we quickly find that companies with multiple unlinked vCenters/clusters find unique challenges propagating a VM template to each compute resource in their administrative domain. VMware's Content Library feature enables enterprises to create a "build stack" that production clusters can subscribe to (and sync) for standardized images. The Publish Once, Sync Everywhere approach simplifies administration, in part because more complex automation doesn't need to be developed to perform these tasks.

Note: The approach of publishing a VM template requires a specific type of template to automatically populate in vSphere. Clicking "Clone as Template to Library" completes this conversion in one step:

Note: This post assumes that a usable template is already available for use in a content library.

Reviewing Canned Offerings

Always explore the easy path before the difficult one. In this case, Ansible and VMware's REST modules do not yet support template deployments - nearly everything after the deployment is covered.

When compared to Ansible or other idempotent commercial automation tools, self-authored API code has some heavy lifting to do.

Ensure it's safe to "do": This is difficult for a developer; it requires a level of infrastructure understanding not normally present.
Implement the "do" thing: With RESTful APIs, this part isn't particularly difficult. Implicitly trusting an endpoint's API reduces most code to "send this document to this IP", resulting in small code bases and little development effort (a handful of PowerCLI or Python lines)
Ensure the "do" was executed correctly: It's notably straightforward to verify with a REST call after the fact, once the correct checks are

Developing the Code

If it's useful to self-develop code, we need to keep a few things in mind. Anyone with formal software development experience will be familiar with this routine.

Begin by developing an outline/pseudocode: Getting a team to agree on the "what" with actual code may be feasible for experienced developers, but the rest of us could use a little help. Flow diagrams are well-supported with Visio/Omnigraffle/Draw.io/Inkscape. Coding with a software diagram is like having a map to guide you while writing code. The C4 Model helps by defining personas and interaction types with a common framework:

Develop the tests before the actions. Data structures and formatting magically lose their ability to change after users start consuming automation, and testing will improve the diagrams and data structures before the code is actually usable.

Develop the user interactions, documentation, and formatting after the tests. This would include any use of argparse, json.loads/dumps, or environment variables. Users and their inputs should be on the diagram created and relatively easy to code. CI/CD tooling should also come into play here.

Develop the action code. This should be a single-digit percentage of the overall effort!

The Python code to deploy from a content library is here.

Maintaining the code

Most of the effort to automate exists in maintaining code, not creating it. It takes considerably more skill to improve on yesterday's effort and to perform routine work like:

Steering/Directional choices for the code
Documentation
Logging any reported issues
Prioritizing issues
Allocating resources to develop solutions to functional issues
Allocating resources to any problems that may occur operationally

Making responsibilities clear is a critical part of any automation solution - a common approach is to establish a RACI chart or to formally elect a maintainer to decide product direction more centrally in community projects.

In short, if you're in the maintenance phase, Congratulations, you made it! 99% of the work is ahead of you!

TL;DR, Let's use the code!

Now that we covered the development practice, let's cover how to use this code with Jenkins!

We create a new "Freestyle Project", and set input parameters:

Configure Git Hooks (but not "Poll SCM", because this work is to be performed on-demand):

Configure Credential Injection (vCenter Credentials):

Finally, command execution:

This will create a new button - Build with Parameters:

Executing with a properly formatted, short JSON file then results in a new VM build. The tool also supports simple CLI invocation, and will suggest resources for each field (except name, which is up to the user):

 1Fetching vSphere Details...  
 2Payload:  
 3{  
 4    "id": {  
 5        "description": "The Content Library object to clone",  
 6        "suggestions": {  
 7            "05886bd8-7389-49e1-a53f-29353cd70186": {  
 8                "name": "debian11-base",  
 9                "guest_OS": "DEBIAN_11_64"  
10            },  
11            "ac6d7f50-0f50-4a1d-b9b5-0e6326f95bb2": {  
12                "name": "suse15.4-base",  
13                "guest_OS": "SLES_15_64"  
14            },  
15            "a26bb731-3ef6-4045-bf7f-3c0514fa343f": {  
16                "name": "ubuntu-22.04-base",  
17                "guest_OS": "UBUNTU_64"  
18            }  
19        }  
20    },  
21    "name": "Example",  
22    "datastore": {  
23        "description": "The vSphere datastore to put virtual disks on",  
24        "suggestions": {  
25            "datastore-1023": {  
26                "name": "datastore1"  
27            }  
28        }  
29    },  
30    "folder": {  
31        "description": "The vCenter folder to place the VM into",  
32        "suggestions": {  
33            "group-v1002": "vm",  
34            "group-v1095": "Infrastructure",  
35            "group-v1096": "Services",  
36            "group-v1097": "NSX",  
37            "group-v1098": "Management Plane",  
38            "group-v1099": "Edges",  
39            "group-v1100": "Experiments",  
40            "group-v1101": "Templates",  
41            "group-v1102": "Monitoring",  
42            "group-v1103": "Routing",  
43            "group-v1104": "CI-CD Pipeline",  
44            "group-v2001": "vCLS",  
45            "group-v26010": "Security",  
46        }  
47    },  
48    "cluster": {  
49        "description": "The vSphere compute cluster to put the VM into",  
50        "suggestions": {  
51            "domain-c1008": "cluster01"  
52        }  
53    }  
54}  
55Operation Complete!

As a footnote, I created this "suggestions" tool in multiple projects at this point - my major issue with most DevOps tool sets is that they don't effectively document ingress/egress schemas, leaving a user of their code "stuck" if they didn't write it. Even when using a tool like Ansible, I would probably create accompanying tools to "discover" or "suggest" inputs to help the user along.

What happens to packets with a VMware vSphere Distributed Switch?

Mon, 26 Dec 2022 16:15:00 -0900

Distributed Virtual Port-Groups (dvPGs) in vSphere are a powerful tool for controlling network traffic behavior. vSphere Distributed Switches (vDS) are non-transitive Layer 2 proxies and provide us the ability to modify packets in-flight in a variety of complex ways.

Note: Cisco UCS implements something similar with their Fabric Interconnects, but software control of behavior is key here.

Where do the packets go?

Let's start with a packet flow diagram:

ESXi evaluates a combination of source and destination dvPG/MAC address conditions and will ship the packet to one of the following "stages":

vDS Memory Bus: This is only an option if the source and destination VM are both on the same port-group and the same host
vDS Uplink Stage: This is where the vSphere Distributed Switch receives the traffic from the vnic and applies any proxy settings
UCS FI: In Cisco UCS Environments configured in end-host mode, traffic will depend on the vSphere Distributed Switch's uplink pinning, as Fabric Interconnects do not transit between redundant nodes. If they are configured in transitive node, they function as external layer 2 switches
External Switching: If the destination is in the same broadcast domain (determined by network/host bits) packets will flow via the access layer (or higher layers depending on the network design)
External Layer 3 Routing: Traffic outside of the broadcast domain is forwarded to the default gateway for handling

Testing The Hypothesis

vDS tries to optimize VM-to-VM traffic to the shortest possible path. If a Virtual Machine attempts to reach another Virtual Machine on the same host, and same dvPG, ESXi will open up a path via the host's local memory bus to transfer the Ethernet traffic.

This hypothesis is verifiable by creating two virtual machines on the same port-group. If the machines in question are on the same host, it will not matter if the VLAN in question isn't trunked to the host, an important thing to keep in mind when troubleshooting.

An easy method to test the hypothesis is to start an iperf session between two VMs, and change the layout accordingly. The bandwidth available between hosts will often differ between the memory bus and the network adapters provisioned.

For this example, we will execute an iPerf TCP test with default settings between 2 VMs on the same port-group, then vMotion the server to another host and repeat the test.

Output (Same Host, same dvPG):

1    root@host:~# iperf -c IP  
2    ------------------------------------------------------------  
3    Client connecting to IP, TCP port 5001  
4    TCP window size:  357 KByte (default)  
5    ------------------------------------------------------------  
6    [  3] local IP port 33260 connected with IP port 5001  
7    [ ID] Interval       Transfer     Bandwidth  
8    [  3] 0.0000-10.0013 sec  6.98 GBytes  6.00 Gbits/sec

Output (Different host, same dvPG):

1    root@host:~# iperf -c IP  
2            ------------------------------------------------------------  
3    Client connecting to IP, TCP port 5001  
4    TCP window size: 85.0 KByte (default)  
5    ------------------------------------------------------------  
6    [  3] local IP port 59478 connected with IP port 5001  
7    [ ID] Interval       Transfer     Bandwidth  
8    [  3] 0.0000-10.0009 sec  10.5 GBytes  9.06 Gbits/sec

Troubleshooting

Understanding vSphere Distributed Switch packet flow is key when trying to assess networking issues. The shared memory bus provided by ESXi is a powerful tool when ensuring short, inconsistent paths are used over longer, more consistent ones.

When constructing the VMware Validated Design (VVD) and the system defaults that ship with ESXi, the system architects chose a "one size fits most" strategy. This network behavior would be desirable with Gigabit data centers or edge PoPs, or anywhere the network speed would be less than the memory bus. In most server hardware, system memory buses will exceed backend network adapters' capacity, improving performance with small clusters. It's important to realize that VMware doesn't just sell software to large enterprises - cheaper, smaller deployments make up the majority of customers.

Impacts on Design

Shorter paths are not always desirable. In my lab, hardware offloads like TCP Segmentation Offload (TSO) are available and will make traffic more performant outwards. Newer hardware architectures, particularly 100GbE(802.3by), benefit from relying on the network adapter for encapsulation/decapsulation work instead of CPU resources better allocated to VMs.

This particular "feature" is straightforward to design around. The vSphere Distributed Switch provides us the requisite tools to achieve our aims, and we can follow several paths to control behavior to match design:

When engineering for high performance/network offload, creating multiple port-groups with tunable parameters is something a VI administrator should be comfortable doing. Automating port-group deployment and managing the configuration as code is even better.
If necessary, consider SR-IOV for truly performance intensive workloads.
The default is still good for 9 of 10 use cases. Design complexity doesn't make a VI administrator "leet"; consider any deviation from recommended carefully.

As always, it's important to know oneself (your environment) before making any design decision. Few localized Virtual Machines concentrate enough traffic to benefit from additional tuning. Real-world performance testing will indicate when these design practices are necessary.

Security patches are available for VMware vCenter 8.0 - Let's try the new vCenter Lifecycle Manager!

Sat, 17 Dec 2022 08:47:00 -0900

Let's take a look at the new lifecycle management process for vCenter.

The old process via the VAMI was easy to execute - the industry is upping the ante with automated pre- and post-testing. Cisco's NX-OS installer is another example - complex procedures (in Cisco's case, sequential PGA or microcode updates) invite problems and escalate a "simple" process to something only the senior-most engineer can safely operate.

vSphere 8 seeks to improve on vSphere 7's upgrade planner by including a "vCenter Lifecycle Manager" to administer package upgrades in an integrated, reliable fashion that includes available pre-checks and reduce "update anxiety".

Then navigate to Updates -> vCenter Server -> Update:

Under Select Version, it's now possible to view eligible updates, along with the type and Release Notes:

For those of us that use NSX Data Center or other integrated products, interoperability checks are part of the wizard

Unfortunately, the vCenter backup step is not included as part of the wizard at this time. (Note: You can back up directly to a filer with vSphere 7 or newer)

Looks like we're not quite ready to use this feature to its fullest potential yet. some notable limitations still exist and should be compensated for:

The "Go button" still requires the VAMI installer for execution
The Lifecycle checker isn't aware of everything (NSX ALB was missing from the matrix), so trips to the Interoperability Matrix are still a good idea
You can subscribe to get release information from VMware using RSS by visiting the product documentation page and copying the RSS link:

Most collaboration platforms (like Teams) can support attaching an RSS feed to a channel

Establishing processes and automatic patch notifications (RSS is a powerful tool for that) will go a long way toward making a New Year's resolution to keep our systems up to date!

Why Automate? Programmability is about solving new problems without fear of failure.

Sat, 03 Dec 2022 10:52:00 -0900

Have you ever heard someone say "I'm not a coder" at work?

The IT industry is changing again. Our humble origins began as polymaths moved from adjacent industries and created a new world from scratch. The pioneering phase led to unique opportunities, creating our transport protocols, programming languages, and ways of building.

The appetite for trying new things is fundamentally different now. We don't worry as much about functional quality with our IT products in this day and age. Even Windows, the butt of jokes in the 2000s provides a consistent and reliable user experience.

Is this the denouement for IT innovation? Neal Stephenson predicted this issue in 2011, examining the creativity and breakneck pace the aerospace industry developed in the 1960s/1970s.

More importantly, he brings to light a painful pattern that IT engineers often go through when trying to create new things for their company's goals - "done before" means something isn't worth doing. Don't we all buy products from (more or less) the same companies to achieve similar outcomes? Why should we care if an idea was executed before?

Liability for failure and high expectations both in quality and reduced risk are prolific in today's market. I'd argue that we have a new problem - after a decade or so of easy-to-implement, highly reliable products, we've forgotten what it feels like to try something new. We're told it costs too much, or might hurt the business when infrastructure engineers want to attempt something novel, and removing something costly is too much of a problem.

The software development market has this figured out. The shift to artistic creativity has provided some growing pains, but we see a potential bridge to the future here. Infrastructure engineers may not "be coders" but uncertain outcomes are what engineers (pragmatic creatives, not artistic creatives) excel at. Our analog in the industry, actual engineers incrementally improve a physical resource, creating safer cars or city designs that promote creative growth. They don't need to worry about the low-level components functioning.

IT is maturing, and our goals are changing, but we can't forget where we came from. Software Development is bifurcating from IT infrastructure. The internal focus for infrastructure is shifting to providing tools and resources to developers as typical customers. We need to find strength in our pragmatic creativity.

Through rose-tinted glasses, "melting pot" innovation imparts a culture of "can-do" wherever success lives - but the transition to disposable electronics/mechanical products is removing opportunities for the development of the required skills. Deep down we know this is bad, and key themes are being set by those who can, "maker spaces" and "hackerspaces" are good examples of this key trend. We need to teach new engineers not to fear failure or the practice of trying.

This doesn't mean that we can throw caution to the wind. While I admire a farmer's ability to innovate at work, creating some trailblazing (albeit somewhat unsafe) fixes in the field is not what we need in IT. (Check out FarmCraft101 for some of the stuff they do)

We need to change how the IT infrastructure industry operates. Educating new engineers will always be at least a little bit of trial and error, and the most important thing we can do is create an environment where we balance the values of trailblazing and reliable delivery. Programmability does this for us, but we don't know how to use it fully yet - but we can look at the other engineering industries to see what might work for us.

What Works

IT Engineers all seem to agree that setting up "labs" facilitates healthy innovation. Home labs offer an environment where an individual can break and rebuild, with an onus to fix it. Resources like VMware's Hands-on-Labs provide a zero-cost method to learn without consequence, albeit with product marketing.

Dyed-in-the-wool engineers love testing. New engineers learn by examining failure without negative context; engineers may work for years before actually building anything themselves. The search term "failure analysis" provides a wealth of information on the processes used by pragmatically creative individuals to steadily improve modular designs to the point where they achieve an artistically creative outcome.

Continuous Delivery practices (DevOps) supercharge these practices; We don't have to deal with physicality. If we manage to automate testing, it costs us virtually zero time, and we can pick up the failure modes as educational resources for new engineers.

How We Can Change

I'd like to see a practice that is two-thirds engineering and one-third "redneck farm repairs". With the FarmCraft101 example, we see an admirable attitude instead of apprehension towards trying new things, and we need to combine it with mature, reliable practices.

The combination of removing or reducing costs for failure and a drive to try new things is about to reach a critical point in the IT industry. We're seeing waves of new engineers enter the industry born in the 2000s, and they don't remember having to try and set IRQs, flip DIP switches with the right type of ballpoint pen, and 32-bit memory ceilings. Personal Computers have become throwaway devices that we don't have to understand well to use, and we need ways to preserve the resilience that comes with "I can solve any problem that comes my way". Raspberry Pi, Arduino, and their lookalikes revitalize this mindset and provide a quality of education that we wish we had when we were young - let's make sure the younglings use it.

I'd also like to see some self-awareness. Most people who "can't code" are in the same boat as those who "can't write", they just don't feel artistically creative. Pragmatic creativity is the backbone of modern engineering - a concept artist doesn't design a car or make it real beyond visual aesthetic and non-functional requirements. The inability to write creatively or "code" is fixed first by identifying a useful goal and achieving it. Infrastructure engineers already do this - just look at how network engineers make long-haul connectivity meet a business objective, or how HTTP forwarding rules make an application behave better.

Let's remove the belief that coding is impossible - most of the truly "propeller-hat" stuff has been done by vendors and community members already - so this leaves most actual software development as "exchanging text files over a network" or "making deterministic paths for behavior". The reason why Object-Oriented Programming and other best practices are emphasized is to ensure you know why it's there. Understanding how an automatic transmission works and should be appropriately used helps a driver improve their skills, but the point to which we learn varies from person to person, and a deep understanding isn't always necessary.

Know your strengths, and know that you can meaningfully contribute by using them. It might take a lifetime to figure out how, but it's worth it.

Tweak Application behaviors with NSX ALB

Sun, 06 Nov 2022 10:19:00 -0900

Not every application is designed to leverage a load balancer.

Early load balancing solutions capitalized on this fact - which may seem odd. The solution to the supportability issue creates new opportunity; we'll look at some examples here. Adjusting application behaviors quickly becomes a powerful tool when enhancing customer experience.

This was the story before digital transformation patterns began to emerge, at least. Business now prefer to use Commercial Off-The-Shelf (COTS) software and Content Management System (CMS) systems instead of writing raw code or web frameworks. This means that the software our SWEs are using is considerably less "tunable" than it used to be, and that the business line expects victory when adopting newer software.

They aren't wrong, the advantage at the other end of this transition is more maintainable software. It's important to be accommodating and compassionate when helping web developers improve their app - they didn't write it, and it's a new approach for them too.

Knowledge of how HTTP works is valuable here. For this exercise, I'm going to proxy Jenkins - it's not designed for a load balancer and is open source.

It's really important to standardize behavior with load balancers. Any profile or rule should be captured either in source control or a manager of managers. I prefer both, because tools like Git Pull Request provide a valuable place for peer review and seeking input on standards. It's good to have standards, it's better all qualified engineers agree on the standard (and there's a way to version/release new ones!)

We'll be using the following profiles for this application as a starting point:

http-profile-Jenkins-v1.0.0
This profile looks comprehensive - Avi Vantage stores all variables explicitly when anything is customized. Here are the highlights:
Enable HSTS: (Not required, I just like it. NSX ALB/Avi doesn't configure a separate service for port 80, so this ensures we don't run cleartext traffic.
Reset HTTP on HTTPS connections: This is an Avi oddity - by default, HTTPS services will return an HTTP 400 series code instead of resetting the connection. I don't like giving out thumbprint data on a given port when it asks for illegal output.
X-Forwarded-For: Avi will include an HTTP Header with this label and the client's IP address. This allows a web engineer to configure logging with the real address in 1-arm deployment scenarios
clienttls-v1.0.0
This client TLS profile is short because we can be strict on security.
Include a short acceptable cipher suite/cipher list, and enforce forward secrecy
Only support TLS 1.2 and 1.3
Prefer server cipher ordering (enforce good cryptography)

We've already covered significant ground in terms of standardization and user experience here. Many web apps don't plan to implement user-friendly TLS, for example, and benefit from NSX ALB/Avi in front of their web tier, acting as a security guard until the client successfully negotiates TLS and navigates to the appropriate port. Many offerings also layer WAF, DDoS protection into their products for this very reason.

Let's configure a basic virtual service:

This basic wizard will establish some acceptable defaults. Let's modify the virtual service and add our settings:

Application Profile
TLS Profile
Certificate

Let's also configure this service to listen on port 80.

Note: Avi will forward unencrypted traffic to the pool, so some loss of privacy may occur if you don't configure the redirect rule under Profiles -> HTTP Request prior to saving!

The new service should be healthy under Application -> Dashboard:

Let's try and use the app!

We encounter an error when clicking "Manage Jenkins":

This is the configured name of my Jenkins server - and it's no longer using the virtual server's name.

We encounter this issue commonly with CMS, and "hard linked" applications can prove difficult to resolve. There are two approaches to resolving this issue:

Leverage redirects, either:
Rewriting any redirects that flow through the load balancer
Redirecting from port 8080 to the correct service
Rewrite HTML as it streams through the load balancer
Avoid this if possible, it's computationally very costly and may not support production traffic levels. It's still useful while developing a new website, as it might be a quicker hypothesis test than a coded solution or a vendor request for change (RFC)

Before diving into a specific fix, it's quite easy to categorize the problem with Firefox. Press F12 on your keyboard to bring up the developer view, and select the Network tab:

Note: This is a big part of why web developers always think it's the network that broke their application.

Click the link again, and you will see a play-by-play of the HTTP transactions executed complete the transaction:

That's a bingo. Jenkins is forwarding a non-relative HTTP 302 (also known as: The link is in another castle HTTP code). Let's rewrite it with NSX ALB by creating an HTTP Response Policy.

Here's a quick cheat sheet:

HTTP Response policies let an engineer edit HTTP transactions (but not the body) from the server to the client
HTTP Request policies let an engineer edit HTTP transactions (but not the body) from the client to the server

The Location header in an HTTP transaction applies specifically to redirects (301,302) - and as such is easy for a load balancer to rewrite. In this case, it wasn't the HTML pages that had a bad link - the redirect chain /manage->/manage/ was the culprit.

VMware NSX ALB (and any other self-respecting ADC) presents an infrastructure engineer with near unlimited power to ensure that an application is delivered correctly (even if there's a flaw in the product). Check out the Developer Screen's network tab (F12) in Firefox/Chrome on some common websites - you will find that many of them employ similar "fixer" tactics.

Track Certificate Expiration with Jenkins and Python 3!

Sun, 23 Oct 2022 12:26:00 -0800

CI/CD tools aren't just for automatically deploying apps! Jenkins excels at enabling an engineer to automatically execute and test code - but, it has a hidden super-power: Automating boring and intensive IT tasks(removing toil).

Let's take a common and relatable IT problem - it doesn't matter if you're a DevOps engineer, a Agilista, or even a "normal" systems engineer. Tracking certificate expiration is not an enjoyable task, and can often involve either manual checking or (usually) outages to discover that a certificate has expired.

This solution will have several major elements:

An inventory of TLS-issued hosts
A Python 3 script (leveraging OpenSSL) to open up TLS connections and fetch certificates
A Jenkins pipeline to execute that script against that inventory daily, emailing the results

Inventory

Full transparency, this example is executed in a home lab. It's naive to assume that this task is trivial for any enterprise, but here are some potential approaches to building an inventory at scale:

Write a Python script to ingest DNS zone files, and loop curl to see if any listen on port 443
Fetch a report for a vulnerability scanner (Retina, Qualys, Nexpose)
Searching PKI issuance reports (if available)

We also want to write our inventory file in a way that's friendly to our execution approach. Python is dynamically typed, and most IT automation is fine with that - we're not doing any hardcore programming for most of it. The vast majority of IT automation involves sending and processing files and I/O.

Python will change a variable to any data type when you tell it to, so it's useful to map out what we want. Here are the relevant data types. I will also include the symbols JSON uses to signify them (if applicable):

String (""): This is a type that encapsulates a series of text characters
Integer (No wrapping): Whole number, and Signed. It can be positive or negative, but there's no decimal point (decimal points are their own unique flavor of complexity in computer programming)
List ([]): To a dyed in the wool software developer, this will be similar to an array. Lists are indexed by an integer, and can contain any data type below it
Python has a neat trick where a for loop can return a list item instead of the index, which saves a great deal of code
Python can sort a list by executing the function .sort() on that object
Dictionary ({}): This is an advanced construct, and provides an engineer with a great deal of capability (at the expense of performance, and code simplicity in some cases)
dicts store entries as key-value pairs, and the index is usually a string
Python can add to a dictionary by adding a new key, e.g. dictionary["newkey"] = "b33f"

When planning software functionality, we want to always use the right tool for the job. Downstream APIs (e.g. OpenSSL) want to see a particular format for a parameter(e.g. TCP port should be an integer), so documentation research is a must at this phase. I'll explain my logic for this file:

I want to easily iterate through the list, without addressing indexes, and I want it to be fast. I should use a list for the top-level data in the inventory ([])
I want to ensure that I don't accidentally address the keys wrong, so each individual entry should be a dictionary ({}) with the following keys: fqdn, port
fqdn should store a string
port should store an integer

Example:

[ { "fqdn": "vcenter.engyak.co", "port": 443 }, { "fqdn": "nsx.engyak.co", "port": 443 } ]

Python Code

Here's a copy of my code. To execute it, the following pip packages need to be installed:

fqdn
OpenSSL
ruamel.yaml

datetime in particular does quite a bit of heavy lifting here. The package providers in the Python community have managed to solve most of the truly difficult work, so interpreting expiration dates is a simple comparison operation.

I heavily rely on functions for this code to work in a maintainable format - this code is only 167 lines long, but most of the usage is for readability.

Another point of note - when writing Python to execute in a pipeline, it helps to be Perl levels of dramatic when crashing code. Jenkins doesn't evaluate output by default, and the easiest way to notify of a problem is by using sys.exit(""). This is why I placed a crash if errors exist at the end of the list.

Jenkins

This configuration example should provide some basic level of functionality. Jenkins has a lot of capability, so this tooling can be endlessly tweaked to your needs.

First, let's set up a SMTP server. With a default installation, the settings are under Dashboard -> Manage Jenkins -> Configure System:

Advanced Settings will allow you to configure SMTP auth, port, if applicable. If you use Gmail, you can still leverage MFA and app passwords, preserving MFA and avoiding password proliferation.

Now, let's set up a freestyle pipeline:

With Jenkins, all things should be executed from source code. This is the way.

We want to run this daily, irrespective of source code changes. This requires a slight deviation from the usual Poll SCM approach:

As always, amnesic workspaces are best:

Python and the inventory file simplify the Jenkins configuration as well. Just execute the script as-is:

The final step is to add a Post-Build Action to email if there is a failure:

It really is this simple. Jenkins will now execute daily and email you a list of expired, and soon to expire certificates!

Lessons Learned

I'm going to improve this code. Here are some of my ideas:

Provide a set of rules for each warning level, e.g. 14 days for Critical
JSON Schema validation will help make the ingest more intuitive
Write/Add a Jenkins parser to chart different severity issues, not unlike Jenkins' flake8 plugin

I'm continually amazed at what the open source community can achieve with this level of simplicity. Would you consider this approach out of reach or too challenging?

Gathering and Using Data from Cisco NX-OS with Ansible Modules

Sat, 15 Oct 2022 23:05:00 -0800

Reliably executing repetitive tasks with automation is easy (after the work is done)

Given enough work, self-built automation can be easy to consume. Non-consumers (engineers) need to focus on reliability and repeatability, but occasionally there's an opportunity to save time and simplify lives directly.

Information gathering with Ansible is a powerful tool, making the level of difficulty to perform a check on one network node roughly equal to the effort on 2, or even one hundred. Here's a quick and easy way to get started.

Ansible Inventory

Ansible likes to know where each managed node lives, and provides the inventory capability to organize similar devices for remote management. Not all network automation endpoints use the inventory feature, so ensure that you read the published documentation first.

Note: The easiest way to check inventory dependency is to verify if there are directives in the playbook named hostname, username, or password. If they exist, that module probably does not use inventory.

Ansible supports two formats for an on-controller inventory, conf (Windows-like) and YAML (Linux-like). Here's an example in YAML, I personally find it easier to read:

 1---  
 2  nxos_example_001:  
 3    hosts:  
 4      nexus_1:  
 5        ansible_host: "1.1.1.1"  
 6      nexus_2:  
 7        ansible_host: "2.2.2.2"  
 8      vars:  
 9        ansible_user: "admin"  
10  nxos_all:  
11    children:  
12      nxos_example_001:

We have a little bit to unpack here:

The first hierarchical tier is for groups, which can contain other groups if you use the children: directive (see nxos_all as an example)
vars: will specify variables to commonly use across all members of that group
ansible_host is used to specify an address - and is useful with dual stack environments (or ones that don't have DNS)

Ansible Facts

Ansible stores all of its runtime variables for a given playbook as facts. This is held as a Python dict at runtime by Ansible Engine, and the debug: module allows an engineer to print the output to stdout:

 1---  
 2- hosts: localhost  
 3  connection: local  
 4  tasks:  
 5    - name: "Print it!"  
 6      debug:  
 7        var: lookup('ansible.builtin.env', 'PATH')  
 8    - name: "Print it, but with msg!"  
 9      debug:  
10        msg:  
11          - "The system environment PATH is: {{ lookup('ansible.builtin.env', 'PATH') }}"  
12          - "Wise engineers don't use this feature to print passwords"

Running this playbook will produce the following:

 1ansible-playbook debug.yml   
 2[WARNING]: No inventory was parsed, only implicit localhost is available  
 3[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'  
 4  
 5PLAY [localhost] *************************************************************************************************************************************************************************************************************************************************************************  
 6  
 7TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************************  
 8ok: [localhost]  
 9  
10TASK [Print it!] *************************************************************************************************************************************************************************************************************************************************************************  
11ok: [localhost] => {  
12    "lookup('ansible.builtin.env', 'PATH')": "/root/.vscode-server/bin/d045a5eda657f4d7b676dedbfa7aab8207f8a075/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"  
13}  
14  
15TASK [Print it, but with msg!] ***********************************************************************************************************************************************************************************************************************************************************  
16ok: [localhost] => {  
17    "msg": [  
18        "The system environment PATH is: /root/.vscode-server/bin/d045a5eda657f4d7b676dedbfa7aab8207f8a075/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",  
19        "Wise engineers don't use this feature to print passwords"  
20    ]  
21}  
22  
23PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************************  
24localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

msg: is effective for formatted output, while var: is considerably simpler when dumping a large dictionary. var: does not require Jinja formatting, which may cause playbooks to be simpler.

Let's apply this to a Cisco NX-OS Node. We can register command output from the nxos_facts module.

Note: The example provided below is the "new way", where Network modules follow the Ansible rules. If using older versions of Ansible (Ansible 2), the following may not be fully available!

First, we need to update the Ansible inventory. We will be using the API method to collect data, and it requires multiple new variables:

ansible_network_os: Instructs Ansible on what module to use for that system
ansible_connection: Instructs Ansible on what transport to use (HTTP API, SSH)
ansible_httpapi_use_ssl: Instructs Ansible to use HTTPS

 1---  
 2  nxos_example_001:  
 3    hosts:  
 4      nexus_1:  
 5        ansible_host: "1.1.1.1"  
 6      nexus_2:  
 7        ansible_host: "2.2.2.2"  
 8      vars:  
 9        ansible_user: "admin"  
10        ansible_network_os: 'cisco.nxos.nxos'  
11        ansible_connection: ansible.netcommon.httpapi  
12        ansible_httpapi_password: ''  
13        ansible_httpapi_use_ssl: 'yes'  
14        ansible_httpapi_validate_certs: 'no'  
15  nxos_all:  
16    children:  
17      nxos_example_001:

The updated inventory allows us to run extremely simple playbooks to gather data

 1---  
 2- hosts: nxos_machines  
 3  tasks:  
 4    - name: "Gather facts via NXAPI"  
 5      cisco.nxos.nxos_facts:  
 6        gather_subset: 'min'  
 7        gather_network_resources:  
 8          - 'interfaces'  
 9      register: nxos_facts_gathered  
10    - name: "Print it!"  
11      debug:  
12        var: nxos_facts_gathered

 1ansible-playbook debug_nxos_facts.yml   
 2  
 3PLAY [nxos_machines] *************************************************************************************************************************************************************************************************************************************************************************************************  
 4  
 5TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************************************************************************************  
 6[WARNING]: Ignoring timeout(10) for cisco.nxos.nxos_facts  
 7ok: [nx-1]  
 8  
 9TASK [Gather facts via NXAPI] ****************************************************************************************************************************************************************************************************************************************************************************************  
10ok: [nx-1]  
11  
12TASK [Print it!] *****************************************************************************************************************************************************************************************************************************************************************************************************  
13ok: [nx-1] => {  
14    "nxos_facts_gathered": {  
15        "ansible_facts": {  
16            "ansible_net_api": "nxapi",  
17            "ansible_net_gather_network_resources": [  
18                "interfaces"  
19            ],  
20            "ansible_net_gather_subset": [  
21                "default"  
22            ],  
23            "ansible_net_hostname": "AnsLabN9k-1",  
24            "ansible_net_image": "bootflash:///nxos.9.3.8.bin",  
25            "ansible_net_license_hostid": "",  
26            "ansible_net_model": "Nexus9000 C9300v",  
27            "ansible_net_platform": "N9K-C9300v",  
28            "ansible_net_python_version": "3.9.2",  
29            "ansible_net_serialnum": "",  
30            "ansible_net_system": "nxos",  
31            "ansible_net_version": "9.3(8)",  
32            "ansible_network_resources": {  
33                "interfaces": [  
34                    {  
35                        "name": "Ethernet1/1"  
36                    },  
37                    {  
38                        "name": "mgmt0"  
39                    }  
40                ]  
41            }  
42        },  
43        "changed": false,  
44        "failed": false  
45    }  
46}  
47  
48PLAY RECAP ***********************************************************************************************************************************************************************************************************************************************************************************************************  
49nx-1                       : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Ansible's Inventory feature enables us to scale per node without any additional code - the previous playbook will execute once on every inventory object in the group, which allows an engineer to thoroughly test a playbook on lab resources with some level of separation.

Deliberate automation design will bear fruit here - as safety is key when developing and testing automation. Like with previous automation-centric posts, thorough, comprehensive testing of automation for reliability is a social responsibility when creating tools.

Establishing a separate CI/CD tooling set to target a lab (or CML, as in this case!) enables us to add additional safeguards against accidental changes, such as ACLs/Firewall policies preventing access from Test CI/CD -> Production network assets. Tools like CML take it even further by allowing an engineer to spin up amnesic NOS instances to run code against.

Here's an applicable instance. Recently, Cisco disclosed a vulnerability with Cisco Fabric Services - and most environments don't need that service running. This is an aggressive fix - but with Ansible we can check for the service and disable it only if it's running, and then check again afterwards. This illustrates the value of idempotency, or the practice of running repeated executions safely.

Using cloud-init with vSphere and openSUSE 15.4

Fri, 23 Sep 2022 16:30:00 -0800

Rapidly deploying Linux servers to meet a whim represents the essence of home lab activities, but we spend a great deal of time spinning/configuring machines to meet our specs.

Worse, we lose a great deal of time keeping them properly configured and up to date, and none have the privilege of unlimited lab time.

Let's explore a way to get a base template implemented in vSphere 7 and enable the machine to boot with customizations like hostname, IP address, startup scripts, etc.

Constructing a VM Template

First, let's pick up a fresh operating system installer ISO from opensuse.org. Since this is a home lab / server-style deployment, I'd recommend using the network image - we'll add everything we want later.

Upload the ISO file to a datastore. This step will allow the installation process to run unattended, even if you shut down the client:

Create a virtual machine, and name it accordingly. Attach the datastore ISO:

Boot the Linux machine. During the installation wizard, ensure that a logical volume manager (LVM2). I've found that when you build a clone template, any disk size you choose will be wrong in the application owner's mind, so plan for the future.

After the installation is complete, disconnect the CD/DVD virtual drive! If you fail to do this on shared infrastructure, the VI admins will have a difficult time with the VM - and in a home lab, that's you. Establish good habits to make responsible tenancy easy.

Start up the machine, and use zypper to install any packages or keys that may be required to administer the device. In a home lab, CA certificates and SSH keys are OK - but an enterprise environment should have an automated, repeatable way to lifecycle trust in the event of a compromise.

Once that's done, let's install cloud-init. This software package is incredibly useful, but it isn't available by default with OpenSUSE Leap:

After installing the package, ensure it's enabled with:

1systemctl enable cloud-init  
2systemctl enable cloud-init-local  
3systemctl enable cloud-config  
4systemctl enable cloud-final  
5cloud-init clean

Cloud-Init

Cloud-init is a project managed by Canonical to standardize VM customization on boot, making IaaS more "cloudy", regardless of hosted location. It is structured to receive configuration data from a datasource and abstracts the specific inputs from other "clouds" to the IaaS workload (VM) as consistent instructions. The customization software will use these data sources as "drop points" to transform the cloud-specific instructions (OVF, Azure, EC2) to a common configuration (Metadata, Userdata).

metadata should represent the workload's system configuration, like hostname, network configuration, and mounts.

userdata should represent the workload's user space configuration, like Ansible playbooks, SSH keys, and first-run scripts. With the current state, I would tend towards using automation to register a workload with Ansible and perform that configuration centrally. It's neat that this level of customization is offered, though - cloud-init can automatically register with centralized orchestrators like SaltStack and Puppet on startup.

cloud-init has a ton of goodness available as boot-time customization, and this will only scratch the surface of how it can be used. cloud-init accepts a YAML configuration that can include:

Users/Groups
CA certificates
SSH keys
Hostnames
Packages/Repositories
Ansible Playbooks
External mounts (NFS)

VMware offers two data sources for workloads provisioned on vSphere:

OVF (the scope of this post)
Direct Metadata (VMware Documentation)

VMware's new RESTful API has built-in documentation. From the vSphere GUI, select the triple ellipsis and select "Developer Center":

Unfortunately, VMware's new metadata source does not appear to function with this distribution. According to Canonical's changelog, cloud-init Version 21.3+ is required to recognize the new datasource. I tested with OpenSUSE 15.4 (Ships with cloud-init 21.4) and received the following error:

 1# A new feature in cloud-init identified possible datasources for        #  
 2# this system as:                                                        #  
 3#   []                                                                   #  
 4# However, the datasource used was: OVF                                  #  
 5#                                                                        #  
 6# In the future, cloud-init will only attempt to use datasources that    #  
 7# are identified or specifically configured.                             #  
 8# For more information see                                               #  
 9#   https://bugs.launchpad.net/bugs/1669675                              #  
10#                                                                        #  
11# If you are seeing this message, please file a bug against              #  
12# cloud-init at                                                          #  
13#    https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid      #  
14# Make sure to include the cloud provider your instance is               #  
15# running on.                                                            #  
16#                                                                        #  
17# After you have filed a bug, you can disable this warning by launching  #  
18# your instance with the cloud-config below, or putting that content     #  
19# into /etc/cloud/cloud.cfg.d/99-warnings.cfg                            #  
20#                                                                        #  
21# #cloud-config                                                          #  
22# warnings:                                                              #  
23#   dsid_missing_source: off                                             #  
24**************************************************************************

To view the provided and applied metadata for a system, cloud-init provides the following file handle:

1/run/cloud-init/instance-data.json

To view the userdata for a system, use the following command:

1cloud-init query userdata

This indicates that we probably have an upstream issue with the new data source type. Reviewing the change log we see several fixes applied to this data source.

Applying Workload Templates

Note: This feature is only available on vSphere 7 and up!

Here's how to leverage the OVF data source with vSphere and OpenSUSE.

The flag disable_vmware_customization is a directive that functions as a switch to choose between the metadata source and the OVF data source. following to /etc/cloud/cloud.cfg:

1disable_vmware_customization: false  
2datasource:  
3  OVF:  
4    allow_raw_data: true  
5vmware_cust_file_max_wait: 25

Once installed, shut the virtual machine down. Right-click on the VM, and select Clone -> Clone as Template to Library:

This vCenter feature will orchestrate the conversion to a template object and publish it to a Content Library as one step.

Deploying a customized machine

The next process needs to be executed via vCenter's Content Library vSphere API:

Establish API Session Key (required authentication for the endpoints used to deploy)
Deploy Content Library Object (/api/vcenter/vm-template/library-items/)
Find the correct content library
Find the correct content library item
Find the content library item via the vsphere API (ID to use in deployment command)
Find vSphere Cluster
Find vSphere Folder
Find vSphere Datastore
Deploy Content Library Item
Wait until deployment is complete, periodically checking to see if it's complete
Normally, an API will respond immediately that the command was successful, and subsequent calls would be required to validate readiness. Instead, vSphere's RESTful API responds with a 200 response only if and when the deployment is complete, which simplifies our code
Locate the Virtual Machine. The previous API call responds with a 200 OK, and Postman conveniently times the operation for you as well!
Apply Guest Customization
Start VM

To replicate this lab, the Postman Environment and Collection will be provided at the bottom of this post. Postman provides a powerful platform to educate engineers unfamiliar with a particular API by expanding the behaviors an HTTP client may have. Automated processes are typically very terse, and do not effectively explain each step and behavior. To import this collection and environment, download the files, and import them:

Postman Environments will stage variables for consumption by Collections.

I have sorted the Postman Collection based on the order of execution. The final customization step will return a 204 if successful, with an empty body. To verify that the configuration was correctly applied, browse to the individual VM in vCenter, and look under Monitor -> Events for an event of the type "Reconfigure VM". If you see the task on the correct VM, start it, and you will see the following:

Soon after, look at the virtual machine to review its customized attributes!

Debugging/Troubleshooting Tips

This process is slightly opaque, and a little confusing at first. Here are some key points for troubleshooting, and the methods to manage it:

The vSphere /vm/guest/customization URI will only respond with a 204 if working correctly.
If it returns a 400, the error will indicate what part of the JSON spec is having issues. Keep in mind that it may only give you the parent key - tools like JSONLint offer a method to quickly validate payloads as well
When locating resources, the Content Library and Templates are returned as a UUID with no description. GET the individual objects to match with names, or use the find API
All other resources (datastore, VM name) are listed with their MOB name, e.g. domain-c1008
Save the response from the deployment action, it has the VM ID when it finally completes
VM Customization can only be applied to a VM that is OFF, and doesn't customize until the VM starts.

Troubleshooting customization after boot can be done by viewing the metadata (/run/cloud-init/) or by reviewing logs at the following locations:

1/var/log/  
2/var/log/vmware/imc  
3journalctl -xe  
4systemctl restart cloud-init

The classic "wipe and restart" method is also quite valuable:

1cloud-init clean -l -s  
2systemctl restart cloud-init

Finally, after a host is successfully configured, I'd recommend disabling cloud-init to prevent further customization. This is just as easily achieved with an Ansible playbook

1systemctl disable cloud-init cloud-final cloud-init-local cloud-config

Code

Identity theft has gotten out of hand. Here are basic ways to protect yourself.

Sat, 13 Aug 2022 16:08:00 -0800

It's not a matter of if you will be the victim of a breach, but when.

Wired is starting to track breaches by halves (as a general tech publication), and security vendors are moving to monthly reporting due to the volume.

It's 2022, and it seems everyone loves to over-share on social media. This may feel good but introduces substantial risks. Let's talk about cyber hygiene.

Information security is a frame of mind, so the most effective way to protect yourself is by being smart. ISC2 has started an institution - The Center for Cyber Safety and Education - to provide further effective education on how to comprehensively protect yourself online.

Here are some brief tips to help keep an eye on when you shouldn't disclose information online. Always ask "Can I dial this back? Do I need to provide this much information?"

Personally Identifiable Information (PII) can provide adversaries with methods to fake your identity
Birthdays. social media companies love to collect them, and they're used for ID verification everywhere. Facebook doesn't need your exact birth date, and storing it there increases your risk. Avoid storing your full birth date whenever feasible
Credit Card Numbers, Expiration Dates, CVVs
Any image of any ID card you own. Driver's License numbers are particularly popular.
Hometowns or birth locations are fun to socialize, but fit in this same category
Full Middle Name
"Mother's maiden name" and other names unlisted and typically used by financial institutions or security questions. Social media quizzes aggressively try to steal information like this!
Previous Employers
Home address / shipping address. These are typically used to validate credit card transactions, particularly large charges
Personal Health Information (PHI) are typically protected by HIPAA, with large exceptions for non-medical institutions. Don't share any of this information without full disclosure on how that information will be used!
Medical history, surgeries, etc.
Ancestry information

It's worth re-iterating, your children are much more likely to be targeted as well. Here are some guidelines on how to protect them from discovering they have a mortgage and a compromised credit score in junior high.

This is the most important, but also the most difficult. We can use products or services to protect your identity and shore any gaps.

Credit Locking / Credit Freezes

Now that we're done scaring you, the good news is that providing some basic level of protection against identity theft isn't particularly hard. Crime does pay, and the most effective way to terminate the pattern is to pursue every avenue to prevent new credit being opened with your identity. Most banks, utilities and other services won't open a credit account without a credit report, so the most effective method of countering compromise is to disallow any and all credit report attempts. The neat thing about this method is that people who are providing legitimate services to you can be sneaky and execute reports without your consent, dinging your credit score in the process.

If you don't do anything else I suggest, do this. It's going to take 5-10 minutes to do all three. Here are the links to "freeze credit" (prevent credit reports from being executed with your information):

Note: You'll need to create a new account for each of these services! Don't lose this information!

Use a Password Manager

To quote Mel Brooks:

"12345! That's amazing! I have the same combination on my luggage!"

Cryptography isn't magic, and all the transport security and firewalls in the world can't protect you from weak identity material.

The most effective way (for the least effort) to de-risk yourself is to set up a password manager. We see some peripheral advantages outside of password storage like storing confidential documents, sharing passwords between family members, etc.

I'm not going to recommend a specific product here, because needs can vary quite a bit depending on needs. Here are some typical requirements I keep in mind when evaluating a password manager:

How strict is its MFA? Can you disable SMS TOTP? Is a hardware security token like Yubikey supported?
Does it support a family plan?
What is its breach response plans?
How securely to store their data?
Is it compatible with my devices? Personally, I use 1Password for the Yubikey support and family plan support. It gives me piece of mind, and has a feature where all passwords are released to my family if I fail to log in for a month. Here are some others, in no particular order:
Dashlane
LastPass
Keeper
Google Password Manager

Host-based security software included password managers:

Using one is better than not - so all of these would be an improvement over nothing at all. I've used Dashlane and LastPass and dropped them in favor of 1Password.

Multi-Factor Authentication

Multi-factor authentication can be broken out into the following major categories:

Something you know: Passwords are an example of this "authentication factor". If a credential is publicly exposed (e.g. used on the Internet) it should be unique to that service to ensure that your banks don't get compromised if your Twitter password leaks
Something you have: The most common MFA tools fit in this category. Yubikeys are fantastic (if supported), and the following Time-based One-time Pad (TOTP) apps are good options. I don't personally have any strong preference other than AVOID SMS / TEXT MESSAGE MFA!
Authy
Yubico Authenticator
Google Authenticator
Microsoft Authenticator
Apple Authenticator
Secure your second factor! If it's a phone, set a lock code and take measures to ensure it can't be stolen. If it's a hardware token (Yubikey, Titan Key, DoD Common Access Card (CAC), keep it on your person and physically secure. Treat it like your ID card.
Something you are: Murky waters abound here, because you must be completely fine with submitting your biometrics to a third party. I'm not keen on doing this, given its potential for misuse. Most consumer fingerprint scanners are "passable" at best, so I don't consider this a good standalone authentication factor.
Somewhere you are: Location-based services are usually somewhat iffy as well for private non-enterprise non-government, as they aren't particularly accurate. If you're consuming a service like Gmail, the company should provide this for you.
Something you do: This is a real propeller-hat scientific factor. Capturing behavior patterns can reveal whether you're behaving normally. Again, this is mostly the responsibility of the group providing you a service.
There's a low-tech way to provide this authentication factor in the real world - paying a security guard. They're good at this and don't need a Ph.D to do it.

Identity Theft Protection

Now, it's time to bring out the heavy hitters. We don't always have the time to keep an eye on the entire internet, or to research recommendations to reduce our online footprint.

Leaning on the experts in identity theft protection services is the way to go. The industry is awash with good options, and the providers of these services aggressively drive costs down to make it affordable.

Full disclosure, I am employed by Allstate, who provides ID theft services. These recommendations are my own and not my employer's.

Here are some guidelines when evaluating ID Theft Protection services:

Do they have a family plan? Children's ID theft is on the rise, mostly because it's easy to predict SSNs given a birth location, easily available information like birth date and addresses, etc. You'd think creditors would avoid opening up a credit card in a newborn's name, but you'd be wrong. Add them to your ID theft protection, freeze their credit!
What services do they monitor? A minimum should maintain tracking your credit score **without affecting it!
**
What insurance do they provide?
What guidance and periodic advice do they offer to customers and the public?
What recommendations do they make to improve your online presence? I'd avoid the ones provided by the credit industries - the Equifax breach impacted my confidence, and nothing brought it back.

As an aside, if you've been a victim of any of the wave of breaches recently, you're probably eligible for free ID theft protection services from multiple companies. Use this to shop around, if you like one, stick with it. If you don't find any you like, here are some popular ones:

Shop around! The worst thing you can do with your online presence is to do nothing, and there's a wide variety of good products to help you out. These services provide a trial, use it to evaluate if it's a good fit.

Conclusion

Society has passed the "age of innocence" with identity theft, and cybersecurity will need to become a routine for anyone living in it. Pandora's box has been opened, and criminals are not going to forget how easy and low-risk cybercrime is. Protecting yourself is a rabbit-hole where all effort is valuable - but you don't need to be a security expert to get the basics in place.

NSX Data Center 4.0.0.1 is now available!

Sat, 06 Aug 2022 22:12:00 -0800

NSX 4 is now available, and it was a surprisingly sparse release in terms of new capabilities.

NSX 4.0 appears to be a "clean house" initiative, so while it's missing "whizz-bang" new data plane features it does address a variety of issues I am happy to say are now closed:

Numerous documented API deprecations. Normally this wouldn't be that big of a deal, but NSX 3.x dropped several experiments (NSX ALB front-end, for example) that stayed available throughout the release train
Deprecating host-based N-VDS
Deprecating KVM and older Linux support (RHEL 7.8, 8.0,8.3) KVM was announced early in 3.0, and the affected EOL dates for RHEL releases have already been exceeded. It is an odd choice for physical servers, though.
Lifecycle Management improvements (I can't test these until the next upgrade).
IPv6 Management Plane support. Unfortunately, VTEPs aren't part of this release, and vSphere is still behind the curve in terms of IPv6 support, limiting efficacy. It's unsurprising to see the Network teams be ahead of the Virtualization teams on network goals.
HSTS is implemented for the WebUI as well. New installs will need to run an override prior to installing a new certificate.
API endpoint to replace API certificate: /api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=""
API endpoint to replace cluster certificate: /api/v1/node/services/http?action=apply_certificate&certificate_id=

Let's review how a new deployment may differ from previous installations:

IPv6 options have now been added to the OVA:

When deploying new workloads with IPv6 support - it's important to have a plan to access those addresses. The best strategy for enterprises and home labs is roughly the same, but with different products. Make your DNS dual-stack, and enter AAAA (IPv6 host records) for each service that supports IPv4 and IPv6. Let your client services do it seamlessly and transparently. End users shouldn't have to care about IPv6 being used. Configuring DNS as-code from a source repository makes this migration easy.

The browser add-on IPvFoo can tell you if you're using native IPv4 or a fallback mode. It'll also tell you what IP addresses you're talking to for a given page to load, which is incredibly useful.

To access an IP address with IPv6 in a web browser, the notation is a little different:

{{protocol}}://[{{site}}]/,

Example:

https://2001:dead:beef::2.

To fully leverage IPv6, you need to give vCenter the same treatment. VMware's documentation on it is here. I executed the change from the VAMI (https://vcenter:5480) under Networking using the supported wizard.

**Note: This will incur brief downtime for vCenter, and interrupt services like VCHA! Execute a vCenter backup before executing this work!
**

And that's about it! We can see NSX Manager with an IPv6 address in the Appliance UI:

And, IPvFoo reports all IPv6 for the front-end:

NSX 4.0 was a mellow release by VMware standards - but according to the Semantic Versioning rules, breaking changes automatically increment a major version. The API deprecations justify the version increment on these terms.

Note: The most important part (NSX Control Plane, VTEPs) are still to be completed.

The Role of Trust and Failure in Information Security

Sun, 03 Jul 2022 10:07:00 -0800

The principles that define the information security field are decades older than computing, and we'd do well to learn from the lessons that precede our industry.

We as security professionals naively construct an "our stuff versus them" model when attempting to defend our networks in our early career. As we develop more of a salty patina, the realization that we shouldn't trust everything begins to set in, transforming previous revisions of our security model from "assume a cow is a sphere in a vacuum at absolute zero" levels of oversimplification to something more worthy. How do we accelerate that learning process?

Confronting Failure

IT Professionals have a truly bizarre relationship with the concept of failure, causing some notably oppressive culture. Burying failure is making our systems vulnerable. This deep-seated crisis of ego has proven to undermine companies a great deal.

When re-reading Cyberpunk: Outlaws and Hackers on the Computer Frontier my experiential lens provided new insight - Kevin Mitnick probably would not have been as successful if DEC was more transparent about how compromised their systems were. Reviewing past through rose-tinted glasses, DEC was considered a company that provided full-service computing - all maintenance and loading was done by a DEC employee.

DEC needed total implicit trust from their customers to operate, and did not disclose their history of compromise to keep the ego-driven narrative ("we have no problems") going for a number of years. This choice empowered Kevin Mitnick and others to continue compromising DEC customers for years and evade capture.

The industry has learned quite a bit about its problems handling failure since 1991, but it could do much better. Vindictive behavior in the emergence of a new breach is common behavior nowadays, with language like "how could they be compromised?" being bandied about as if we didn't know about thankless dependence on somebody from Nebraska pattern, turns in a counter-productive direction. We need transparency from those who provide us paid software, but we punish them for providing it.

Google feels strongly about learning from failure, and so should we. Engineering professions (the truer, more long-lived ones) have long since begun to analyze failure as a method of teaching, proving that we leave a wealth of information wasted every time we revert to blame in the advent of a problem.

The entire industry needs to figure out how to constructively learn from failure, while simultaneously applying appropriate levels of pressure on all product vendors to ensure that vulnerabilities, breaches, and other problems are disclosed fairly and appropriately. Easy, right?

Building on a Foundation for Trust

Despite this cycle of abuse, the industry does want to see more from companies that provide tech products. Vulnerability disclosure programs are particularly successful and important due to significant pressure to improve. Locksmithing is of particular interest in this case, as is the Enigma story - technology doesn't passively improve over time, it requires conscious effort and does not progress until problems are acknowledged.

Vulnerability disclosure made big strides transitioning from the more negative past (see AT&T's stance here) where the courts would use the CFA as a sledgehammer to cover up or mask problems to the current day's model - "Heartbleed" and "Shellshock". Examine those websites - the vulnerability campaigns maintain blameless language, and focus consumers on how to resolve the issues, what questions to ask of their vendors. We complain about "vulnerability fatigue" often, forgetting that we only began to transform the industry to a more secure future a mere 8 years ago.

Let's commit to some meaningful changes to help us get to the future - We aren't there yet!

Encourage and Promote Transparency: When a company provides you information on a security problem, push for more information. CloudFlare publishes their post-mortems here as an example.
Don't be Punitive: This part doesn't specifically apply to security, or even IT. The person nearest to you probably has nothing to do with your issue.
For bonus points, don't allow others to paint you this way.
Focus on the Fix: Some people find this part easier than others - shift focus on solving problems and providing real results. Continually ask yourself the question "am I contributing to the objectives of this conversation" and ensure that emphasis stays on what to do next or how something will be prevented.
Persuade Others: Group-think begins working against you when building trust or establishing a culture of disclosure. Don't allow others to steer the conversation back into punitive territory:
Listen: If others paint your behavior as punitive, listen to what they have to say and example it objectively.
This conversation also needs to remain constructive, so cascading tactics may apply. Operating in good faith is key.
Recognize Contribution: It takes courage to share information about a problem, sincerely remind those who disclose via a direct verbal utterance.
Restate Commitments: A business relationship, like any other human relationship, requires maintenance. In times of strain, it's important to be forward and remind participants of their commitment to each other.
Sympathize: Find common ground with those who failed. We've all done it, blur the factional lines by reflecting on other failures - but only bring your own to avoid creating adversarial tension.

Learning from failure is a crucial aspect to improving oneself, improving others, and building trust. Don't let a good failure go to waste by fighting over it.

Scale datacenters past the number of VLAN IDs with NSX-T Tier-0 and Q-in-X

Sun, 22 May 2022 08:08:00 -0800

VMware introduced the ability to double-encapsulate layer 2 frames in via the "Access VLAN" option for VRF instances in NSX Data Center: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-4CB5796A-1CED-4F0E-ADE0-72BF7B3F762C.html

Q-in-VNI provides a capable infrastructure engineer the ability to to construct straightforward multitenant constructs. From the documentation and previous testing, we have demonstrated its capability outside of Layer 3 constructs. The objective of this post is to examine and test these capabilities with Tier-0 VRFs:

NSX Data Center provides the ability to pass a tag inside of a segment, which enables a few interesting design patterns:

Layer 3 VPN to customer's campus, with each 802.1q tag delineating a separate "tenant", e.g. PCI/Non-PCI
Inserting carrier workloads selectively to specific networks
Customer empowerment - let's enable the customer to use their cloud how they please

To validate this hypothesis, we will leverage the following isolated topology:

Note: VRF-Lite is required for this feature!

Q-in-VNI on NSX-T Routers

When configuring an interface on a VRF, the following option (Access VLAN ID) becomes available. Select the appropriate "inside" VLAN for each sub-interface:

We then configure the sub-interfaces - the tenant VM is unaware that it's being wrapped into an overlay:

Unsurprisingly, this feature just works. NSX-T is designed to provide a multi-tenant cloud-like environment, and VLAN caps are a huge problem in that space. In this example, we created 2 subinterfaces in the same VRF - normally tenants would not share a VLAN.

Q-in-VNI Design Patterns

Offering Q-in-VNI on a Tier-0 solves valuable use cases for multi-tenant platform services. The primary focus of these solultions is customer empowerment - VMware isn't taking sides on matters of :"vi vs emacs", "Juniper vs Cisco", etc. Instead, we as CSPs can provide a few design patterns that enable a customer to leverage their own chosen methods, or even to allow an ISP to integrate crisply and effectively with their telecommunications services.

NSX-T has some fairly small scalability limits for CSPs leveraging the default recommended design pattern (160 standalone Tier-0s), and the ultimate best solution is to leverage multiple NSX Data Center instances to accommodate. If the desired number of tenants is above, say, twice that, the VRF-Lite feature allows an infrastructure engineer to deploy 100 routing tables per Tier-0.

VRF-Lite enables scaling to 4,000 Tier-1 gateways at this level, and a highly theoretical maximum of 160,000, but the primary advantage of this approach is that customers can bring their own networking easily and smoothly, front-ending NSX components with their preferred Network OS. Customers and Infrastructure engineers extend the feature set and reducing strain on NSX at the same time, creating a scenario where both the customer and the infrastructure benefit cooperatively.

Note: VMware's current configuration maximums are provided here: https://configmax.esp.vmware.com/guest?vmwareproduct=VMware%20NSX>

VRF-Lite can also be built to provide a solution where customers can "hair-pin" their tenant routing tables to a virtual firewall over the same VN-Segment. Enterprise teams leveraging NSX Data Center benefit the most from this approach, because common virtual firewall deployments are limited by the number of interfaces available on a VM. This design pattern empowers customers by permitting infrastructure engineers to construct thousands of macrosegmentation zones if desired.

Q-in-Q on NSX-T Routers

Time to test out the more complex option!

When I attempt to configure an internal tag with VRF-Lite subinterfaces, the following error is displayed:

Sadly, it appears that Q-in-Q is not supported yet, only Q-in-VNI. Perhaps this feature will be provided at a later date.

Here's the VyOS configuration to perform Q-in-Q:

Retrospective

Learn, hypothesize, test is an important cycle for learning and design, and this is why we build home labs. NSX Data Center appeared to support Q-in-Q tagging - but the feature was ultimately for passing a trunk directly to a specific VLAN ID in a port-group.
vSphere vDS does not appear to allow Q-in-Q to trunk outwards to other port-groups that do not support VLAN trunking, either.
Make sure that MTU can hold inner and outer header without loss. I set the MTU to 1700, but you only need 16 bytes of extra MTU for the 802.1q header.

Different Methods to carry 802.1q tags with VMware vDS and NSX-T

Fri, 06 May 2022 15:00:00 -0800

VMware's vDS is a bit of a misnomer

In a previous post, I covered the concept of transitivity in networking - but in Layer 2 (Ethernet) land, transitivity is critically important to understanding how VMware's Virtual Distributed Switch (vDS) works.

The statement "VMware's Virtual Distributed Switch is not a switch" seems controversial, but let's take a moment to reflect - when you plug in the second uplink on an ESXi host, does the ESXi host participate in spanning tree?

Testing this concept at a basic level is straightforward. Enabling BPDU Guard on an ESXi host-facing port should take the host down immediately if it's actually a switch (it doesn't). This concept is actually quite useful to a capable infrastructure engineer.

A "Layer 2 Proxy"

VMware's vDS is quite a bit more useful than a simple Layer 2 transitive network device - each ESXi host accepts data from a virtual machine, and then leverages a "host proxy switch" to take each packet and re-write its Layer 2 header in a 3-stage process:

Note: For a more detailed explanation of VMware's vDS architecture and how it's implemented, VMware's documentation is here.

Note: VMware's naming for network interfaces can be a little confusing, here's a cheat sheet:

vnic: A workload's network adapter
vmnic: A hypervisor's uplink
vmknic: A hypervisor's Layer 3 adapter

A common misinterpretation of vDS is that the VLAN ID assigned to a virtual machine is some form of stored variable in vSphere - it isn't. vDS was designed with applying network policy in mind - and an 802.1q tag is simply another policy.

vDS is designed with tenancy considerations, so a port-group will not be allowed to transit traffic between different port-groups (but the same VLAN ID). Non-transitive behaviors achieve two goals at the same time - providing an infrastructure engineer total control of data egress on a vSphere host, and adequate segmentation to build a multi-tenant VMware Cloud.

Replacing the Layer 2 header on workload packets is extremely powerful - vDS essentially empowers an infrastructure engineer to write policy and change packet behavior. Here are some examples:

For a VM's attached vnic, apply an 802.1q tag (or don't!)
For a VM's attached vnic, limit traffic to 10 Megabits/s
For a VM's attached vnic, attach a DSCP tag
For a VM's attached vnic, deny promiscuous mode/MAC spoofing
For a VM's attached vnic, prefer specific vmnics
For a VM's attached vnic, export IPFix

NSX expands on this capability quite a bit by adding overlay network functions:

For a VM's attached vnic, publish the MAC to the global controller table (if it isn't already there) and send the data over a GENEVE or VXLAN tunnel
For a VM's attached vnic, only allow speakers with valid ARP and MAC entries (validated via VMware tools or Trust-on-First-Use) to speak on a given segment
For a VM's attached vnic,send traffic to the appropriate distributed or service router

NSX also enables a few things for NFV that are incredibly useful, NFV service chaining and Q-in-VNI encapsulation.

Q-in-VNI encapsulation is pretty neat - it allows an "inside" Virtual Network Function (VNF) to have total autonomy with inner 802.1q tags, empowering an infrastructure engineer to create a topology (with segments) and deliver complete control to the consumer of that app. Here's an example packet running inside a Q-in-VNI enabled segment (howto is here).

NSX Data Center is not just for virtualizing the data center anymore. This capability, combined with the other precursors (generating network configurations with a CI tool, automatically deploying changes, virtualization), is the future of reliable enterprise networking.

Network Experiments with VMware NSX-T and Cisco Modeling Labs

Fri, 29 Apr 2022 17:09:00 -0800

Cisco Modeling Labs (CML) has turned out to be a great tool for deploying virtual network resources, but the "only Cisco VNFs" limitation is a bit much.

Let's use this opportunity to really take advantage of the capabilities that NSX-T has for virtual network labs!

Overview

For the purpose of lab construction, I will use an existing Tier-0 router and uplinks to facilitate the "basics", e.g. internet connectivity && remote accessibility:

Constructing the NSX-T "Outside" Environment

NSX-T has a few super-powers when it comes to constructing network topologies. We need one in particular to execute this - Q-in-VNI encapsulation - which is enabled on a per-segment basis. Compared to NSX-V, this is simple and straightforward - a user simply applies the allowed VLAN range to the vn-segment, in this case, _eng-lab-vn-cml-trunk:

We'll need to disable some security features that protect the network by creating and applying the following segment profiles. The objective is to limit ARP snooping and force NSX to learn MAC addresses over the segment instead of from the controller.

I can't stress this enough, the Q-in-VNI feature gives us near unlimited power with other VFs, circumventing the 10 vNIC limit in vSphere and reducing the number of segments that must be deployed to what would normally be "physical pipes".

Here's what it looks like via the API:

Expose connectivity to CML

Generating the segments in NSX is complete, and now we need to get into the weeds a bit with CML. CML is, in essence, a Linux machine running containers, and has external interface support. Let's add the segments to the appropriate VM:

CML provides a Cockpit UI to make this easy, but the usual suspects exist to create a Linux bridge. We're going to create both segments as bridge adapters:

A word of warning - this is a Linux Software Bridge, and will copy everything flowing through the port. NSX helps by squelching a significant chunk of the "network storms" that result, but I would recommend not putting it on the same NSX manager or hosts as your production environments.

Leveraging CML connectivity in a Lab

The hard part's done! To consume this new network, add a node of type External Connector, and configure it to leverage the bridge we just created:

The next step would be to deploy resources and configure them to connect to this central point - I'd recommend this approach to make nodes reachable via their management interface, i.e. deploy a management VRF and connect all nodes to it for stuff like Ansible testing. CML has an API, so the end goal here would be to spin up a test topology and test code against it, or to build a mini service provider.

Lessons Learned

This is a network lab, and we're taking some of the guardrails off when building something this free-form. NSX does allow an engineer to do so on a per-segment basis, and insulates the "outside" network from any weird stuff you intend to do. The inherent dangers to spamming BPDUs outwards from a host or packet flooding can be contained with the built-in feature "segment security profiles", indicating that VMware had clear intent to support similar use-cases in the future.

NSX also enables a few other functions if in routed mode with a Tier-1. It's trivial to set up NAT policies, redistribute routes as Tier-1 statics, to control export of any internal connectivity you may or may not want touching the "real network".

I do think that this combination is a really impressive one-two punch for supporting enterprise deployments. If you ask anyone but Cisco, a typical enterprise network environment will involve many different network vendors, and having the flexibility to mix and match in a "testing environment" is a luxury that most infrastructure engineers don't have, but should.

CML Scenario

Here's the CML scenario I deployed to test this functionality:

Vendor interoperability with multiple STP instances

Sun, 17 Apr 2022 15:56:00 -0800

Spanning Tree is the all-important loop prevention method for Layer 2 topologies and source of ire to network engineers worldwide

Usually IT engineers list the Dunning-Kruger Effect in a negative context, depicting an oblivious junior or an unaware manager, but I like to focus on the opposite end of the curve with meta-cognition. Striving to developing meta-cognition and developing self-awareness is difficult and competes with the ego, but is an incredibly powerful tool for learning. I cannot stress enough how important getting comfortable with one's limitations is to a career.

Spanning Tree is a key topic that should be revisited frequently, building upon knowledge growth.

Let's examine some methods for plural instantiation with Spanning Tree.

The OSI Model and sublayers

Ethernet by itself is surprisingly limited in scope for the utility we glean from it. It provides:

Source/Destination forwarding
Variable length payloads
Segmentation with 802.1q (VLAN tagging) or 802.3ad (Q-in-Q tagging)

It also doesn't provide a Time-to-Live field at all, which is why switching loops are so critically dangerous in production networks.

Ethernet needs a supporting upper layer (the Logical Link Control sublayer) to relay instructions on how to handle packets. Internal to a switch ASIC, the hardware itself needs some form of indicator to select pipelines for forwarding or processing.

802.2 LLC and Subnetwork Access Protocol (SNAP) are typically used in concert to bridge this gap and allow a forwarding plane to classify frames and send to the appropriate pipeline. Examples are provided with this article, where LLC and SNAP are used to say "this is a control plane packet, don't copy it, process it" or "this packet is for hosts, copy and forward it".

Multiple Ethernet Versions

Over the years, network vendors implemented multiple versions of the Lower Ethernet sublayer, and in many cases did not update the Control Plane code on network equipment. manage to proliferate throughout computer networks for , which appear to resemble simply stitching together ancient PDU types for compatibility . It's not entirely surprising that multiple ethernet editions exist given the fragmentation throughout the industry.

I'd strongly recommend reading this study by Muhammad Farooq-i-Azam from the COMSATS Institute of Information Technology. The author outlines methods of testing common forms of Ethernet in production formats, and provides a detailed overview of our progress to standardization. Spanning Tree is a major cause for the remaining consolidation work, as it turns out.

Generally, Ethernet II is what you want to see in a production network, and most host frames will follow this standard. Variable-length fields over 1536 bytes are supported by this protocol, which is a big advantage in data centers.

The original Ethernet standard, 802.3 did not support ethertypes or frames larger than 1536 bytes, and is typically used by legacy code in a switching control plane. Two major variants are used within this protocol, extended by LLC and (optionally) with SNAP as an extension.

So, why does this matter when talking about spanning tree? Network Equipment Providers (NEP) haven't all updated their Layer 2 control plane protocols in a long time. Bridge Protocol Data Units (BPDUs) are inconsistently (consistently in their hardware, but inconsistent with others) transmitted, causing a wide variety of interoperability issues.

Per-VLAN Spanning Tree

From a protocol standpoint, Per-VLAN STP and RSTP are probably the simplest method with the fewest design implications and most intuitive protocol layout - but some dangers are inherent when running multi-vendor networks.

Let's examine a few captured packets containing the STP control plane

Cisco PVRST+ Dissection

Cisco structures the per-VLAN control plane by wrapping instantiated BPDUs:

With an Ethernet II Layer 2 Lower
With an 802.1q tag encapsulating the VLAN ID
With a SNAP Layer 2 Upper, PID of PVSTP+ (0x010b)
Destination of 0100.0ccc.cccd

Arista and Cisco Vendor Interoperability

Interestingly enough, I discovered that Arista appears to implement compatible PVRST to Cisco (with some adjustments covered in Arista's whitepaper). To validate this, I executed another packet dissection with Arista's vEOS, which is available to the public. I have provided the results here, but the PDUs are nearly identical to the Cisco implementation.

MST Dissection

For the majority of vendor interoperable spanning-tree implementations, this will be a network engineer's best option. MST allows an engineer to specify up to 16 separate instances of Spanning Tree, either 802.1d or 802.1w. The primary hazards with leveraging MST have a great deal to do with trunking edge ports, as each topology must be accounted for and carefully planned. BPDU Guard, Loop Guard, and VLAN pruning are absolutely necessary when planning MST, in addition to diagramming each topology that will be instantiated.

IEEE's MST standard is implemented per-instance, and relayed with one common BPDU. It's pleasingly efficient, but...

With an 802.3 Ethernet Layer 2 Lower
With no 802.1q tag
With an LLC header, ID of BPDU (0x42)
With a destination MAC 0180.c200.0000
After the BPDU typical attributes, MST attaches an extension indicating priorities, but no unique bridge IDs. If a topology differs, it would be separated onto its own control plane frame.

CST Dissection

For comparison, I also compared against a Mikrotik Routerboard, which should follow the implements RSTP as a single instance (Common Spanning Tree) and optionally supports Multiple Spanning-Tree (MST). I found the following attributes with default settings:

Destination of 0180.c200.0000 (STP All-Bridges Destination Address)
802.3 Ethernet Frame
Spanning-tree BPDU Type

Conclusion

Spanning Tree is a foundation to all enterprise networks, but there really seems to be some legacy code and general weirdness in place here. The industry, starting with the data center, is moving to a more deterministic control plane to replace it, whether that be EVPN or a controller-based model like NSX.

Campus enterprise deployments are beginning to do the same as businesses realize that a shared, multi-tenant campus network can increase the value against the cost of the same equipment. With the downsizing of corporate offices, the only thing stopping office providers from also providing a consolidated campus network is a general lack of expertise and an industry full of under-developed solutions. As platforms converge, the same pattern will emerge in campus deployments soon.

In terms of design implications, supplanting Layer 2 legacy control planes is no small feat. Even EVPN requires STP at the edge, but containment and exceptions management are both clear design decisions to make when building enterprise networks.

Cisco Modeling Labs

Sat, 26 Mar 2022 08:00:00 -0800

Ever wonder what it would be like to have a platform dedicated to Continuous Improvement / Testing / Labbing?

Cisco's put a lot of thought into good ways to do that, and Cisco Modeling Labs(CML) is the latest iteration of their solutions to provide this as a service to enterprises and casual users alike.
CML is the next logical iteration of Virtual Internet Routing Labs (VIRL) and is officially backed by Cisco with legal VNF licensing. It's a KVM type-2 hypervisor, and automatically handles VNF wiring with an HTML5/REST interface.

Cisco's mission is to provide a NetDevOps platform to network engineers, upgrading the industry's skillset to provide an entirely new level of reliability to infrastructure. Hardware improves over time, and refresh cycles complete - transferring the "downtime" problem from hardware/transceiver failure to engineer mistakes. NetDevOps is the antidote for this problem, infrastructure engineers should leverage automation to make any operation done on production equipment absolutely safe.

Solution Overview

Cisco Modeling Labs provides you the ability to:

Provision up to 20/40 Cisco NOS nodes (personal licensing) as you see fit
Execute "What if?" scenarios without having to pre-provision or purchase network hardware, improving service reliability
Develop IaC tools, Ansible playbooks and other automation on systems other than the production network
Leverage TRex (Cisco's network traffic generator) to make simulations more real
Deploy workloads to the CML fabric to take a closer look or add capabilities inside
Save and share labbed topologies
Do everything via the API. Cisco DevNet even supplies a Python client for CML to make the API adoption easy!

Some important considerations for CML:

Set up a new segment for management interfaces, ensuring that external CI Tooling/Ansible/etc can reach it.
VNFs are hungry. NX-OSv images are at the high end (8GB of memory each), and IOSv/CSR1000v will monopolize CPU. Make sure that plenty of resources are allocated
Leverages Cisco Smart Licensing. CML uses legitimate VNF images with legitimate licensing, but will need internet access
CML does not provide SD-WAN features, Wireless, or Firepower appliance licensing, but does support deploying them

Let's Install CML! Cisco provides an ESXi (vSphere) compatible OVA:

After It's deployed, I assigned 4 vCPUs and 24 GB of memory, and attached the platform ISO. Search under software.cisco.com for Modeling Labs:

Once mounted, the wizard continues from there. CML will ask you for passwords, IP configurations, the typical accoutrements:

The installer will take about 10 minutes to run, as it copies all of the base images into your virtual machine. Once it boots up, CML has two interfaces:

https://{{ ip }}/ : The CML "Lab" Interface
https://{{ ip }}:9090/ : The Ubuntu "cockpit" view, manage the appliance and its software updates from here. Cisco's integrated CML into this GUI as well.

CML's primary interface will not allow workloads to be built until a license is installed. Licenses are fetched from the Cisco Learning Network Store under settings:

CML is ready to go! Happy Labbing!

Deploy Root Certificates to Debian-based Linux systems with Ansible

Sat, 19 Mar 2022 09:47:00 -0800

There are numerous advantages to deploying an internal root CA to an enterprise:

Autonomy: Enterprises can control how their certificates are issued, structured, and revoked independently of a third party.
Slow or fast replacement cycles are permissible if you control the infrastructure, letting you customize the CA to the business needs
Want to set rules for what asymmetric cryptography to use? Don't like SHA1? You're in control!
Cost: Services like Let's Encrypt break this a bit, but require a publicly auditable service. Most paid CAs charge per-certificate, which can really add up
Better than self-signed: Training users to ignore certificate errors is extremely poor cyber hygiene, leaving your users vulnerable to all kinds of problems
Multi-purpose: Certificates can be used for users, services, email encryption, getting rid of passwords. They're not just to authenticate web servers.

The only major obstacle to internal CAs happens to be a pretty old one - finding a scalable way to deliver the root Certificate Authority to appropriate "trust stores" (they do exactly what it sounds like they do) on all managed systems. Here are a few "hot-spots" that I've found over the years, ordered from high-value, low effort to low-value, high effort. They're all worthwhile, so please consider it an "order of operations" and not an elimination list:

Windows Certificate Store: With Microsoft Windows' SChannel library, just about everything on the system will install a certificate in one move. I'm not a Windows expert, but this delivery is always the most valuable up-front.
Linux Trust Store: Linux provides a trust store in different locations depending on distribution base.
Firefox: Mozilla's NSS will store independently from Windows or Linux, and will need to be automated independently.
Java Trust Stores are also independently held and specific to deployed version. This will require extensive deployment automation (do it on install, and do it once).
Python also has a self-deployed trust store when using libraries like requests, but Debian/Ubuntu specific packages are tweaked to use the system. There are a ton of tweaks to just make it use the system store, but the easiest is to leverage REQUESTS_CA_BUNDLE as an environment variable pointing to your system trust store.

Hopefully it's pretty clear that automation is about to become your new best friend when it comes to internal CA administration. Let's outline how we'd want to tackle the Linux aspects of this problem:

Pick up the root certificate, and deliver from the Controller to the managed node
Either Git or an Artifacts store would be adequate for publishing a root certificate for delivery. For simplicity's sake, I'll be adding it to the Git repository.
Ansible's copy module enables us to easily complete this task, and is idempotent.
Install any software packages necessary to import certificates into the trust store
Ansible's apt module enables us to easily complete this task, and is idempotent.
Install the certificate into a system's trust store
Locations differ based on distribution. Some handling to detect operating system and act accordingly will be worthwhile in mixed environment
Ansible's shell module can be used, but only as a fallback. It's not idempotent, and can be distribution-specific.
Restart any necessary services

Here's where the beauty of idempotency really starts to shine. With Ansible, it's possible to just set a schedule for the playbook to execute in a CI tool like Jenkins. CI tools add some neat features here, like only executing on a source control change, which may not apply when using an artifacts store to deploy the root certificate.

In this example, I will be adding the play to my nightly update playbook to illustrate how easy this is:

After completion, this action can be tested by a wide variety of means - my favorite would be cURLing a web service that leverages the root CA:

1curl https://nsx.engyak.co/

Cloud-Scale Networking: NSX Datacenter Hierarchical Tier-0s, blending telecom with cloud

Sat, 12 Mar 2022 16:43:00 -0900

VMware's NSX Datacenter product is designed for a bit more than single enterprise virtual networking and security.

When reviewing platform maximums (NSX-T 3.2 ConfigMax), the listed maximum number of Tier-1 routers is 4,000 logical routers. Achieving that number takes a degree of intentional design, however.

When building a multi-tenant cloud network leveraging NSX Data Center, the primary design elements are straightforward:

Shared Services Tenant
A multi-tenant data center will always have common services like outbound connectivity, orchestration tooling, object storage, DNS.
This tenant is commonly offered as a component to a physical fabric with a dedicated Workload Domain (WLD), but can be fully virtualized and run on the commodity compute
**Packaging shared services within a WLD will require repetitive instantiation of common services, but makes the service "anycast-like" in that it will be more resilient for relatively little effort
**
**Implementation with hierarchical Tier-0s is comically easy, just attach the shared Tier-1 to the Infrastructure Tier-0!
**
**When designing a shared services tenant, a "Globally Routable" prefix is highly recommended in IPv4 to ensure that no conflicts occur. With IPv6, all networks should have globally routable allocations
**
Scaling: Tenants, Tenants, and more Tenants!
Most fabric routers have a BGP Peer cap of 256 speakers:
Divide that number in half for dual-stack: 128 speakers
Remove n spine nodes, 124 speakers
Add 4-way redundancy (2 Edge Transport Nodes): 31 Speakers, or 8-way, 15 Speakers
For customer-owned routing, the scalability maximum of 4,000 logical routers is achievable without good planning

Let's take a look at an infrastructure blueprint for scaling out network tenancy:

The more "boring" version of tenancy in this model supports highly scalable networking, where a customer owns the Tier-1 firewall and can self-service with vCloud Director:

VRF-Lite allows an NSX Engineer to support 100 network namespaces per Edge Transport Node cluster. When leveraging this feature, Tier-1 Logical Routers can connect to a non-default network namespace and peer traditionally with infrastructure that is not owned by the Cloud Provider via Layer 2 Interconnection or something more scalable (like MPLS or EVPN).

Empowering Cloud-Scale Networking with a feature like drop-shipping customer workloads into MPLS is an incredibly powerful tool, not just with scalability,but with ease of management. NSX-T VRFs can peer directly with the PE, simplifying any LDP or SR implementations required to support tenancy at scale.

With this design, we'd simply add a VRF to the "Customer Tier-0" construct, and peer BGP directly with the MPLS Provider Edge (PE). An NSX-T WLD with 3.0 or newer can support 1,600 instances this way, where most Clos fabrics can support ~4,000. The only difference here is that WLDs can be scaled horizontally with little effort, particularly when leveraging VCF.

A tenant VRF or network namespace will still receive infrastructure routes, but BGP engineering, Longest Prefix Match (LPM), AS-Path manipulation all can be used to ensure appropriate pathing for traffic to customer premise, shared infrastructure or other tenants with traditional telecommunications practices. Optionally, the customer's VRF can even override the advertised default route, steering internet-bound traffic to an external appliance.

This reference design solves another substantial maintainability problem with VRF-Lite implementations - VRF Leaking. The vast majority of hardware based routing platforms do not have a good path to traverse traffic between network namespaces, and software based routing platforms struggle with the maintainability issues associated with using the internal memory bus as a "warp dimension".

With overlay networking, this is easily controlled. VRF constructs in NSX-T inherit the parent BGP ASN, which prevents transit by default. The deterministic method to control route propagation between VRFs with eBGP is to replace the ASN of the tenant route with its own, e.g.:

Original AS-Path	New AS-Path
64906	64905
64905	64905

AS-Path rewrites provide an excellent balance between preventing transit by default and easily, safely, and maintainably providing transitive capabilities.

The more canonical, CCIE-worthy approach to solving this problem is not yet viable at scale. Inter-Tier-0 meshing of iBGP peers is the only feature available, confederations and route reflectors are not yet exposed to NSX-T from FRRouting. When this capability is included in NSX Data Center, iBGP will be the way to go.

Let's build the topology:

NSX Data Center's super-power is creating segments cheaply and easily, so a mesh of this fashion can be executed two ways.

Creating many /31s, one for each router member as overlay segments. NSX-T does this automatically for Tier-0 clustering and Tier-1 inter-links, and GENEVE IDs work to our advantage here. Meshing in this manner is best approach overall, and should be done with automation in a production scenario. I will probably write a mesh generator in a future post.

In this case, we'll do it IPv6-style, by creating a single interconnection segment and attaching a /24/64 to it. Tier-0 routers will mesh BGP with each other over this link. I'm using Ansible here to build this segment, not many of the knobs and dials are necessary so it saves both time and physical space.

 1 tasks:
 2 -  name: "eng-lab-vn-segment-ix-10.7.200.0_24:"
 3   vmware.ansible_for_nsxt.nsxt_policy_segment:
 4   hostname: "nsx.lab.engyak.net"
 5   username: '`{{ lookup("env", "APIUSER") }}`'
 6   password: '`{{ lookup("env", "APIPASS") }}`'
 7   validate_certs: False
 8   state: present
 9   display_name: "eng-lab-vn-segment-ix-10.7.200.0_24"
10   transport_zone_display_name: "POTATOFABRIC-overlay-tz"
11   replication_mode: "SOURCE"
12   admin_state: UP
13 -  name: "eng-lab-vn-segment-ix-10.7.201.0_24:"
14   vmware.ansible_for_nsxt.nsxt_policy_segment:
15   hostname: "nsx.lab.engyak.net"
16   username: '`{{ lookup("env", "APIUSER") }}`'
17   password: '`{{ lookup("env", "APIPASS") }}`'
18   validate_certs: False
19   state: present
20   display_name: "eng-lab-vn-segment-ix-10.7.201.0_24"
21   transport_zone_display_name: "POTATOFABRIC-overlay-tz"
22   replication_mode: "SOURCE"
23   admin_state: UP

From here, we configure the following external ports and peers on the infrastructure Tier-0:

eng-lab-t0r00-ix-1: 10.7.200.1/24 (AS64905, no peer)
eng-lab-t0r00-ix-vrfs-1: 10.7.201.1/24 (AS64905, no peer)
eng-lab-t0r00-ix-2: 10.7.200.2/24 (AS64905, no peer)
eng-lab-t0r00-ix-vrfs-2: 10.7.201.2/24 (AS64905, no peer)
eng-lab-t0r01-ix-1: 10.7.200.11/24 (AS64906)
eng-lab-t0r10-ix-1: 10.7.201.100/24 (AS64906)

Let's build the tenant default Tier-0:

Then configure BGP and BGP Peers:

Voila! BGP peerings are up without any VLANs whatsoever! Next, the VRF:

For the sake of brevity, I'm skipping some of the configuration after that. The notable differences are that you cannot change the BGP ASN:

That's about it! Peering between NSX-T Tier-0 routers is a snap.

The next stage to building a cloud must be automation at this point. Enabling self-service instantiating Tier-1s and/or VRFs will empower a business to onboard customers quickly, so consistency is key. Building the infrastructure is just the beginning of the journey, as always.

Lessons Learned

The Service Provider community has only just scratched the surface of what VMware's NSBU has made possible. NSX Data Center is built from the ground up to provide carrier-grade telecommunications features at scale, and blends the two "SPs" (Internet and Cloud Service Providers) into one software suite. I envision that this new form of company will become some type of "Value-Added Telecom" and take the world by the horns in the near future.

Diving deeper into NSX-T's Service Provider features is a rewarding experience. The sky is the limit! I did discover a few neat possibilities with this structure and design pattern that may be interesting (or make/break a deployment!)

Any part of this can be replaced with a Virtual Network Function. Customers like a preferred Network Operating System (NOS), or simply want a NGFW in-place. VMware doesn't even try to prevent this practice, enabling it as a VM or a CNF (someday). If a Service Provider has 200 Fortinet customers, 1,000 Palo Alto customers, and 400 Checkpoint customers, all of them will be happy to know they can simply drop whatever they want, wherever they want.
Orchestration and automation tools can build fully functional simulated networks for a quick "what if?" lab as a vApp.

Credit where credit is due

The team at 27 Virtual provided the design scenario and the community opportunity to fully realize this idea, and were extremely tolerant of me taking an exercise completely off the rails. You can see my team's development work here: https://github.com/ngschmidt/nsx-ninja-design-v3

VMware NSX-T and Ansible

Sun, 06 Mar 2022 08:23:00 -0900

What is the point of all this software-defined infrastructure if you don't use it?

In prior examples, it's a fairly straightforward path to SDN when deploying NSX Data Center, allowing a VI admin or network engineer to deploy virtual network resources via a GUI.

This isn't the end of an effort, but the start of a journey. Once the API is available, deployment of services on top of a virtual cloud network become easier

Setup

First, we need our CI tooling to be capable of leveraging VMware's NSX-T Community module:

1ansible-galaxy collection install git+https://github.com/vmware/ansible-for-nsxt

**Note: This requires Ansible 3.0 or higher to leverage the "Galaxy install from git+https" feature. This software package is not hosted on Ansible Galaxy
**

Building the playbook

To an Ansible engineer, this part might be a bit bothersome. Since we're interacting with an appliance, there are several differences compared to canonical Ansible playbooks:

Inventory: Playbooks for network providers specify targets inside of their module, not using Ansible's inventory. If an inventory is used, the playbook will execute once on every inventory host, targeting the same destination device - not good.
Credentials are stored internally to the leveraged module
Any parameters that would constitute the thing that the playbook should build

Now that we have the required software, the next step would be to map out what to build. Playbooks typically have documentation on what parameters it will accept to perform work. Example here.

Let's take what will probably be the most common deployment to automate - network segments:

Executing the Playbook

When writing playbooks that will be frequently re-used, I like to leverage Jinja in the playbook, denoted by {{}} to morph to whatever need I have at the moment. Ansible supports loading variables with the --extra-vars "@{{ filename }}" statement:

1ansible-playbook create_segments.yml --extra-vars "@segment_vars.yml"

Credential Management

I've glossed over a particularly important aspect of automation here - what credentials do we use?

Generally, I prefer to perform "hands-free" execution of aspects like this, so the provided playbook is designed to leverage the "Credentials as Environment Variables" feature in Jenkins automatically.

What I think of the Module

ansible-for-nsxt is a community maintained module, so expectations for the automation features should be set at the price paid for the software. I've tested this platform quite a bit, and the issues that I encountered appeared to be code obfuscation.

Digging deeper into the Ansible modules themselves, the biggest reason for this is NSX-T's declarative API - it makes more sense to not code those things at all and simply leverage the API. This resonates with me quite a bit!

VMware's community also has built testing into the repository, which seems to indicate that that testing is automated!

I do have two complaints about this module:

No maintainer for Ansible Galaxy means that Ansible 2 users (Red Hat shops) will have a difficult time installing the software from GitHub
Not all modules fully support IPv6 yet

Lessons Learned

Testing Idempotency

When looking to leverage automation at work, change safety is often provided as the primary reason not to do it.

It's okay not to trust automation with your production network, that's what testing is for.

When implementing an automation play like this in a production network, we need to evaluate whether or not it's safe to execute.

The first aspect to test when planning to automate a play will be idempotency, or how executing a play should consistently cause the desired state defined in the playbook, and not to impact services unnecessarily (ex. by deleting and re-creating something).

Idempotency is extremely important with infrastructure, we can't afford downtime as easily as other IT professions. The good news is that it's pretty easy to achieve:

1ansible-playbook create_segments.yml --extra-vars "@segment_vars.yml"  
2ansible-playbook create_segments.yml --extra-vars "@segment_vars.yml"`

The second executed playbook should return a status of "ok" instead of "changed".

If this test is passed, we'll evaluate the next aspect of idempotency by executing the playbook, then changing the segment's configuration in some way (GUI, API, Ansible playbook), and re-executing to ensure that the delta was detected by Ansible and can be resolved by it. I just create a second segment_vars_deviation.yml file and execute thusly:

1ansible-playbook create_segments.yml --extra-vars "@segment_vars.yml"  
2ansible-playbook create_segments.yml --extra-vars "@segment_vars_deviation.yml"  
3ansible-playbook create_segments.yml --extra-vars "@segment_vars.yml"`

This test should return "changed" for all three attempts.

More Testing

These two tests have extremely good coverage, indicating a high level of change safety for almost no effort. Additional tests to execute for a play like this can be mind-mapped or brain-stormed, and then coded from there. Here are some examples:

Check Tier-0 routes to ensure that the prefix built populated
Check Looking Glass to ensure the prefix is reachable everywhere
Check vCenter for the port-group created

Bogons, and how to leverage public IP feeds with NSX-T

Sun, 16 Jan 2022 08:37:00 -0900

Have you ever wondered what happened to all the privately-addressed traffic coming from any home network?

Well, if it isn't explicitly blocked by the business, it's routed, and this is not good. Imagine what data leakage can occur when a user mistypes a destination IP - the traffic goes out to the Service Provider, who will probably drop it somewhere, but it's inviting wiretapping/hijacking to occur.

RFC 1918 over the internet is part of a larger family of addresses called "bogons", an industry term to indicate a short list of prefixes that shouldn't be publicly routed.

Many network attacks traversing the public internet flow from what the industry calls "fullbogons", or addresses that, while publicly routable, aren't legitimate. These addresses are obviously block-able, with no legitimate uses.

As it turns out, the industry calls these types of network traffic Internet background noise, and recent IPv4 shortages have pushed some providers (Cloudflare in particular) into implementing on previous fullbogon space and shouldering the noise from an internet-load of mis-configured network devices.

The solution for mitigating both problems is the same: filtering that network traffic. Team Cymru provides public services that list all bogon types for public ingestion, all that needs to be done here is implementation and automation.

Bogon strategies

Given that the bogon list is extremely short, bogons SHOULD be implemented as null routes on perimeter routing. Due care may be required when filtering RFC 1918 in enterprise deployments with this method - Longest Prefix Match (LPM) will ensure that any specifically routed prefix will stay reachable, as long as dynamic routing is present and not automatically summarizing to the RFC 1918 parent. If this is a concern, implement what's possible today and build a plan for what isn't later.

Here's an example of how to implement with VyOS:

 1protocols {  
 2    static {  
 3        route 10.0.0.0/8 {  
 4            blackhole {  
 5            }  
 6        }  
 7        route 10.66.0.0/16 {  
 8            blackhole {  
 9            }  
10        }  
11        route 100.64.0.0/10 {  
12            blackhole {  
13            }  
14        }  
15        route 169.254.0.0/16 {  
16            blackhole {  
17            }  
18        }  
19        route 172.16.0.0/12 {  
20            blackhole {  
21            }  
22        }  
23        route 192.0.2.0/24 {  
24            blackhole {  
25            }  
26        }  
27        route 192.88.99.0/24 {  
28            blackhole {  
29            }  
30        }  
31        route 192.168.0.0/16 {  
32            blackhole {  
33            }  
34        }  
35        route 198.18.0.0/15 {  
36            blackhole {  
37            }  
38        }  
39        route 198.51.100.0/24 {  
40            blackhole {  
41            }  
42        }  
43        route 203.0.113.0/24 {  
44            blackhole {  
45            }  
46        }  
47    }  
48}

This approach is extremely straightforward and provides almost instant value.

Fullbogon strategies

For smaller enterprises and below (in this case, "smaller enterprise" means unable to support 250k+ prefixes via BGP, so nearly everybody) the most effective path to mitigate fullbogons isn't routing. Modern policy based firewalls typically have features that can subscribe to a list and perform policy-level packet filtering. The following are examples of firewall platform built-ins that let you just subscribe to a service:

Palo Alto Networks https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/built-in-edls.html
Fortinet (Note: This isn't built in, but is supported): https://fusecommunity.fortinet.com/blogs/yuri1/2020/07/26/using-external-threat-feeds-in-fortigate
Checkpoint (Note: This is mostly built-in, and the threat feeds are provided by checkpoint): https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk103154

In all of these cases, policies must be configured to enforce on traffic in addition to ingesting the threat feeds.

We can build this on our own, though. Since NSX-T has a policy API, let's apply it to a manager:

The method I provided here can be applied to any IP list with some minimal customization. There is only really one key drawback to this population method - the 4,000 object limit.

GitOps with NSX Advanced Load Balancer and Jenkins

Sun, 16 Jan 2022 08:25:00 -0900

GitOps

GitOps, a term coined in 2017, describes the practice of performing infrastructure operations from a Git repository. In this practice, we easily develop the ability to re-deploy any broken infrastructure (like application managers), but that doesn't really help infrastructure engineers.

From the perspective of an Infrastructure Engineer, Git has a great deal to offer us:

Versioning: Particularly with the load balancer example, many NEPs (Network Equipment Providers) expose object-oriented profiles, allowing services consuming the network to leverage versioned profiles by simply applying them to the service.
Release Management: Most enterprises don't have non-production networks to test code, but having a release management strategy is a must for any infrastructure engineer. At a minimum, Git provides the following major tools for helping an infrastructure engineer ensure quality when releasing changes:
Collaboration/Testing: Git's Branch/Checkout features contribute a great deal to allowing teams to plan changes on their own infrastructure**.** If virtual (simulated production) instances of infrastructure are available, this becomes an incredibly powerful tool
Versioning: Git's Tags feature provides an engineer the capability of declaring "safe points" and clear roll-backs sets in the case of disaster.
Peer Review: Git's Pull Request feature is about as good as it gets in terms of peer review tooling. When releasing from the "planning" branch to a "production" branch, just create a Pull Request providing notification that you want the team to take a look at what you indent to do. Bonus Points for performing automated testing to help the team more effectively review the code.

Applying GitOps

On Tooling

Before visiting this individual implementation, none of these tools are specific or non-replaceable. The practice is what matters more than the tooling, and there are many equivalents here:

Jenkins: Harness, Travis CI, etc
GitHub: GitLab, Atlassian, Gitea, etc.
Python: Ansible, Terraform, Ruby, etc.

GitOps is pretty easy to implement (mechanically speaking). Any code designed to deploy infrastructure will execute smoothly from source control when the CI tooling is completely set up. All of the examples provided in this article are simple and portable to other platforms.

On Practice

This is just an example to show how the work can be executed. The majority of the work in implementing GitOps lies with developing release strategy, testing, and peer review processes. The objective is to improve reliability, not to recover an application if it's destroyed.

It does help deploy consistently to new facilities, though.

Let's go!

Since we've already developed the code in a previous post, most of the work is already completed - the remaining portion is simply configuring a CI tool to execute and report.

A brief review of the code (https://github.com/ngschmidt/python-restify/blob/main/nsx-alb/apply_idempotent_profiles.py) shows it was designed to be run headless and create application profiles. Here are some key features for pipeline executed code to keep in mind:

If there's a big enough problem, crash the application so there's an obvious failure. Choosing to crash may feel overly dramatic in other applications, but anything deeper than pass/fail takes more comprehensive reporting. Attempt to identify "minor" versus "major" failures when deciding to crash the build. It's OK to consider everything "major".
Plan the code to leverage environment variables where possible, as opposed to arguments
Generate some form of "what was performed" report in the code. CI tools can email or webhook notify, and it's good to get a notification of a change and what happened (as opposed to digging into the audit logs on many systems!)
Get a test environment. In terms of branching strategy, there will be a lot of build failures and you don't want that to affect production infrastructure.
Leverage publicly available code where possible! Ansible (when fully idempotent) fits right into this strategy, just drop the playbooks into your Git repository and pipeline.

Pipeline Execution

Here's the plan. It's an (oversimplified) example of a CI/CD pipeline - we don't really need many of the features required by a CI tool here:

Pull code from a Git Repository + branch
Jenkins can support a schedule as well, but with GitOps you typically just have the pipeline check in to SCM and watch for changes.
Clear workspace of all existing data to ensure we don't end up with any unexpected artifacts
Load Secrets (username/password)
"Build". This stage, since we don't really have to compile, simply lets us execute shell commands.
"Post-build actions". Reporting on changed results is valuable and important, but the code will also have to be tuned to provide a coherent change report that turns to code. Numerous static analysis tools can also be run and reported on from here.

The configuration is not particularly complex because the code is designed for it:

This will perform unit testing first, then execute and provide a report on what changed.

Building from Code

The next phase to GitOps would be branch management. since the production or main branch now represents production, it's not particularly wise to simply commit to it when we attempt to create a new feature or capability. We're going to prototype next:

Outline what change we want to make with a problem statement
Identify the changes desired, and build a prototype. Avi is particularly good at this, because each object created can be exported as JSON once we're happy with it.
This can be done either as-code, or by the GUI with an export. Whichever works best.
Determine any versioning desired. Since we're going to make a functional but not breaking change, SemVer doesn't let us increment the third number. Instead, we'll target version v0.1.0 for this release.
Create a new branch, and label in a way that's useful, e.g. clienttls-v0.1.0-release
Generate the code required. Note: If you use the REST client, this is particularly easy to export:

1python3 -m restify -f nsx-alb/settings.json get_tls_profile --vars '{\"id\": \"clienttls-prototype-v0.1.0\"}'

Place this as a JSON file in the desired profile folder.
Add the new branch to whatever testing loop (preferably the non-prod instance!) is currently used to ensure that the build doesn't break anything.
After a clean result from the pipeline, create a pull request (Example: https://github.com/ngschmidt/python-restify/pull/17). Notice how easy it is to establish peer reviews with this method!

After the application, we'll see the generated profiles here:

What's the difference?

When discussing this approach with other infrastructure engineers, the answer is "not much". GitOps is not useful without good practice. GitOps, put simply, makes disciplined process easier:

Peer Review: Instead of meetings, advance reading, some kind of Microsoft Office document versioning and comments, a git pull request is fundamentally better in every way, and easier too. GitHub even has a mobile app to make peer review as frictionless as possible
Testing: Testing is normally a manual process in infrastructure if performed at all. Git tools like GitHub and Bitbucket support in-line reporting, meaning that tests not only cost zero effort, but the results are automatically added to your pull requests!
Sleep all night: It's really easy to set up a 24-hour pipeline release schedule, so that roll to production could happen at 3 AM with no engineers awake unless there's a problem

To summarize, I just provided a tool-oriented example, but the discipline and process is what matters. The same process would apply to:

Bamboo and Ansible
Harness and Nornir

The only thing missing is more systems with declarative APIs.

Leverage Idempotent, Declarative Profiles with the NSX-ALB (Avi) REST API

Sun, 02 Jan 2022 11:13:00 -0900

Idempotence and Declarative Methods - not just buzzwords

Idempotence

Coined by Benjamin Peirce, this term indicates that a mathematical operation will produce a consistent result, even with repetition.

Idempotence is much more complicated subject in mathematics and computer science. IT and DevOps use a simplified version of this concept, commonly leveraging flow logic instead of Masters-level Algebra.

Typically, an idempotent function in DevOps-land adds a few other requirements to the mix:

If a change is introduced, convergence (the act of making an object match what the consumer asked for) should be non-invasive and safe
It's the responsibility of the consumer to adequately test this
Provide a "What If?" function of some kind, indicating how far off from desired state a system is
It's the responsibility of the consumer to adequately test this. Idempotent systems should provide a method for indicating what will change, but won't always provide a statement of impact

Ansible's modules are a good example of idempotent functions, but Ansible doesn't require that everything be idempotent. Some good examples exist of methods that cannot be idempotent, re-defining it to add the "do no harm" requirement:

Restarting a service
Deleting and re-adding a file

As a result, many contributed modules are not pressured to be idempotent when they should be. It's the responsibility of the consumer (probably you) to verify things don't cause harmful change.

Declarative Methods

Lori MacVittie (F5 Networks) provides an excellent detailed explanation of Declarative Models here:

https://www.f5.com/company/blog/why-is-a-declarative-model-important-for-netops-automation

Declarative Methods provide a system interface that can be leveraged by a non-Expert, by allowing a consumer to specify what the consumer wants instead of how to build it (an Imperative method).

This is a huge issue in the IT industry in general, because we (incorrectly) conflate rote memorization of individual imperative methods with capability. In the future, the IT industry will be forced to transform away from this highly negative cultural pattern.

We as professionals need to solve two major problems to assist in this transition:

Find a way to somehow teach fundamental concepts without imperative methods
Teach others to value the ability to effectively define what they desire in a complete and comprehensive way

If you've ever been frustrated by an IT support ticket that has some specific steps and a completely vague definition of success, declarative methods are for you. The single most important aspect of declarative methods is that the user/consumer's intent is captured in a complete and comprehensive way. If a user fails to define their intent in modern systems like Kubernetes, the service will fail to build. In my experience, problem #1 feeds into problem #2, and some people just think they're being helpful by requesting imperative things.

Obviously the IT industry won't accept that a computer system is allowed to deny them if they failed to define everything they need to. This is where expertise comes in.

How we can use it in DevOps

Here's the good news - designing and providing systems to provide idempotent, declarative methods of cyclical convergence isn't really an enterprise engineer's responsibility. Network Equipment Providers (NEP) and systems vendors like VMware are on the hook for that part. We can interact with provided functions leveraging some relatively simple flow logic:

Well-designed APIs (NSX ALB and NSX-T Data Center are good examples) provide a declarative method, ideally versioned (minor gripe with NSX ALB here, the message body contains the version and may be vestigial), and all we have to do is execute and test.

In a previous post, I covered that implementing reliability is the consumer's responsibility, transforming a systems engineer's role into one of testing, ensuring quality and alignment of vision as opposed to taking on all of the complex methods ourselves

TL;DR Example Time, managing Application Profiles as Code (IaC)

Let's start by preparing NSX ALB(Avi) for API access. The REST client I'm using uses HTTP Basic Authentication, so it must be enabled - the following setting is under System -> Settings -> Access Settings:

Note: In a production deployment other methods like JWT ought to be used.

The best place to begin here with a given target is to consult the API documentation, provided here: https://avinetworks.com/docs/21.1/api-guide/ApplicationProfile/index.html

When reviewing the documentation VMware provides, declarative CRUD methods are all provided (GET, PUT, PATCH, DELETE) for an individual application profile. Let's leverage the workflow above as code (Python 3)

 1# Recursively converge application profiles  
 2def converge_app_profile(app_profile_dict):  
 3    # First, grab a copy of the existing application profile  
 4    before_app_profile = json.loads(  
 5        cogitation_interface.namshub(  
 6            "get_app_profile", namshub_variables={"id": app_profile_dict["uuid"]}  
 7        )  
 8    )  
 9  
10    # Fastest and cheapest compare operation first  
11    if not app_profile_dict["profile"] == before_app_profile:  
12        # Build a deep difference of the two dictionaries, removing attributes that are not part of the profile, but the API generates  
13        diff_app_profile = deepdiff.DeepDiff(  
14            before_app_profile,  
15            app_profile_dict["profile"],  
16            exclude_paths=[  
17                "root['uuid']",  
18                "root['url']",  
19                "root['uuid']",  
20                "root['_last_modified']",  
21                "root['tenant_ref']",  
22            ],  
23        )  
24  
25        # If there are differences, try to fix them at least 3 times  
26        if len(diff_app_profile) > 0 and app_profile_dict["retries"] < 3:  
27            print("Difference between dictionaries found: " + str(diff_app_profile))  
28            print(  
29                "Converging "  
30                + app_profile_dict["profile"]["name"]  
31                + " attempt # "  
32                + str(app_profile_dict["retries"] + 1)  
33            )  
34            # Increment retry counter  
35            app_profile_dict["retries"] += 1  
36            # Then perform Update verb on profile  
37            cogitation_interface.namshub(  
38                "update_app_profile",  
39                namshub_payload=app_profile_dict["profile"],  
40                namshub_variables={"id": app_profile_dict["uuid"]},  
41            )  
42            # Perform recursion  
43            converge_app_profile(app_profile_dict)  
44        else:  
45            return before_app_profile

Idempotency is easy to achieve, we leverage the deepdiff library to process data handled by a READ action, and then execute a re-apply action if it doesn't match. This method will allow me to just mash the execute key until I feel good with the results. I've included a retry counter as well to prevent looping.

That's actually all there is to it - this method can be combined with Semantically Versioned Profiles. I have provided public examples on how to execute that in the source code: https://github.com/ngschmidt/python-restify/tree/main/nsx-alb

The winds of change in cloud operations, and why integrations like NSX Data Center 3.2 + Advanced Load Balancer are important

Sun, 26 Dec 2021 00:00:00 -0900

**Note: This feature has been deprecated by VMware

The Jetstreams

Cloud operators now provide two completely different classes of service to customers:

Self-Service, VMs, Operating System Templates
- Generally mature, some private cloud operators are smoothing out CMPs or such, but work as intended from a customer perspective
- Bringup is automated
- Operating System level configuration is usually automated
- Application-level configuration is not always automated or managed as code
- Cloud Provider typically will hold responsibility for a widget working
Containers, Service Definitions, no GUI
- Kubernetes fits squarely here, but other services exist
- Not the most customer friendly, nascent
- Application Owner has to hold responsibility for a widget working

Infrastructure Engineers as Agents of Change

The industry cannot transition responsibility for "stuff working" to creative types (Web Developers, App Developers). Have you ever heard "it's the network"? How about "This must be a <issue I'm not responsible for>"?

This is a call for help. Once the current trends with Automation and reliability engineering slow down (Type 1 above), the second kind of automation is going to necessitate leveraging infrastructure expertise elsewhere. Services like Kubernetes both require a "distribution" of sorts, but there's nobody to blame when something fails to deploy.

Why This Matters

NSX-T's 3.2 release has provided a ton of goodies, with an emphasis on centralized management and provisioning. We're starting to see tools that will potentially support multiple inbound declarative interfaces to achieve similar types of work, and NSX Data Center Manager has all the right moving parts to provide that.

NSX ALB's Controller interface provides comprehensive self-service and troubleshooting information, and a "Lite" service portal.

NSX Datacenter + ALB presents a really unique value set, with one provisioning point for all services, in addition to the previously provided Layer 3 fabric integration. It's good to see this type of write-many implementation

Let's try it out!

First, let's cover some prerequisites (detailed list here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/installation/GUID-3D1F107D-15C0-423B-8E79-68498B757779.html):

Greenfield Deployment only. This doesn't allow you to pick up existing NSX ALB installations
NSX ALB version 20.1.7+ or 21.1.2+ OVA
NSX Data Center 3.2.0+
NSX ALB and NSX Data Center controllers must exist on the same subnet
NSX Data Center and NSX ALB clusters should have a vIP configured
...it also can't support third party CAs

Once these are met, the usual prerequisites also matter.

Deployment is extremely straightforward, and managed under System -> Appliances. This wizard will require you to upload the OVA, so get that rolling before filling out any forms:

The NSX Manager will take care of the VM deployment for you. Interestingly enough, this will allow us to potentially get rid of tools like PyVmOmi and let us deploy everything with Ansible/Terraform someday.

Once it's done deploying the first appliance, it'll report a "Degraded" state until 3 controllers are deployed.

Once installed, the NSX ALB objects should appear under Networking -> Advanced Load Balancer:

At this point, NSX Datacenter -> NSX ALB is integrated, but not ALB -> Data Center. The next step is to configure an NSX-T cloud. I've covered the procedure for configuring an NSX-T cloud here: https://blog.engyak.co/2021/09/vmware-nsx-alb-avi-networks-and-nsx-t/

Note: Using a non-default CA Certificate for NSX ALB here will break the integration. It can be reverted back by reverting the certificate, and there doesn't appear to be an obvious way to change that yet. A Principal Identity is formed for the connection between systems, indicating that the feature is just not fully exposed to users yet.

Viewing Services

A cursory review of the new ALB section indicates that existing vIPs don't appear via the ALB GUI, but the inverse is true. Let's try and build one for Jenkins! The constructs are essentially the same as with the ALB UI, but the process is considerably simpler:

First, create a vIP

Then, create the pool:

Finally, we will create the virtual service. Note: nullPointerException seems to mean that the SE Group is incorrect, and may need to be manually resolved on the ALB controller.

Unlike most VMware products, NSX Data Center seems to handle multi-write (changes from BOTH the ALB and the Manager) fairly well.

That's it!

Footnote: To use custom TLS profiles, it must be invoked via the API only. I am building a method to manage that here: https://github.com/ngschmidt/python-restify/tree/main/nsx-t/profiles/tls

NSX-T 3.2 and NSX ALB (Avi) Deployment Error - "Controller is not reachable. {0}"

Wed, 22 Dec 2021 21:00:00 -0900

Note: This feature has been deprecated by VMware

NSX-T 3.2 has been released, and has a ton of spiffy features. The NSX ALB integration is particularly neat, but while repeatedly (repeatably) breaking the integration to learn more about it, I ran into this error:

When deploying NSX ALB appliances from the NSX Manager, it's very important to keep the NSX ALB Controller appliances where NSX Manager can see them. In addition, the appliances must exist on the same Layer 2 Segment.

Detailed requirements for running the two together are here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/installation/GUID-3D1F107D-15C0-423B-8E79-68498B757779.html

This post is not about the integration, however.

The following error:

1NSX Advanced Load Balancer Controller is not reachable {0}

Indicates that NSX-T has orphaned appliances. NSX-T has API invocations for cleaning this up, but not GUI integrations. This is similar to other objects, and is because programmatic checking should be used to allow this work to be reliable.

To fix this, we must perform the following steps:

Get the list of NSX ALB appliances, if there isn't any, exit
Iterate through the list of appliances, prompting the user to delete
After deleting, check to make sure that it was deleted

The first step for any API invocations should be consulting the documentation. The NSX ALB Appliance management section is 3.7.1.4. After researching the procedure, I found the following endpoints:

Performing this procedure with programmatic interfaces is a good example of when to use APIs - the task is well defined, the results are easy to test, and work to prevent user mistakes is rewarding.

TL;DR - I wrote the code here, integrating it with the REST client: https://github.com/ngschmidt/python-restify/blob/main/nsx-t/nsxalb_deployment_cleanup.py

VyOS and other Linux builds unable to use `vmxnet3` or "VMware Paravirtual SCSI" adapter on vSphere

Sat, 04 Dec 2021 09:55:00 -0900

Have you seen this selector when building machines on vSphere?

This causes some fairly common issues in NOS VMs, as most don't really know what distribution the NOS is based on.

"Guest OS Version" doesn't just categorize your workload, though. Selecting "Other Linux" instructs vSphere to maximize compatibility and ensure the VI admin receives a reliable deployment - which means it'll run some pretty old virtual hardware.

VMware curates its lifecycle "Guest OS" settings here. Note that "Other" isn't described: https://partnerweb.vmware.com/GOSIG/home.html

Two commonly preferred settings for virtual hardware aren't available with this particular OS setting, and they both cause potential performance issues:

LSI Logic Virtual SCSI
Intel E1000 NIC <---If you're wondering, it will drop your VM's throughput

Let's cover how we'd fix this in vSphere 7 with a VM. The example in this procedure is VyOS 1.4.

Updating Paravirtualized Hardware

First, let's change the Guest OS version to something more specific. Generally, Linux distributions fall under two categories, Red-Hat, and Debian derivatives - Gentoo/Arch users won't be covered here because they should be able to find their own way out.

Since we know VyOS is a well-maintained distribution, I'll change it to "Debian 11." While this is technically lying, we're trying to provide a reference hardware version to the virtual machine, not accurately represent the workload. This menu can be reached by selecting "edit VM" on the vSphere console:

Second, let's change the SCSI Adapter:

Replacing network adapters will take a little bit more work. Re-typing existing interfaces is not currently supported in vSphere 7, so we'll need to delete and re-create. In this example, we can set a static MAC address so that the guest distribution can correlate the new adapter to the same interface by setting the MAC Address field to static. Since I'm life cycling a VM template, I don't want to do that!

If you're editing an existing VM, make a backup. If it's a NOS, export the configuration. There is no guarantee that the configurations will port over perfectly, and you will want a restore point. Fortunately, lots of options exist in the VMware ecosystem to handle this!

Refactoring / Recovering from the change

With my template VM, the only issues presented were that the interface re-numbered and the VRF needed to be re-assigned:

1set interfaces ethernet eth2 address dhcp
2set interfaces ethernet eth2 vrf mgmt
3commit
4save

Since we have the VM awake and in non-template-form, we can update the NOS too. (Guide here: https://docs.vyos.io/en/latest/installation/update.html)

 1vyos@vyos:~$ add system image https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso vrf mgmt  
 2Trying to fetch ISO file from https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso  
 3% Total % Received % Xferd Average Speed Time Time Time Current  
 4Dload Upload Total Spent Left Speed  
 5100 436M 100 436M 0 0 12.0M 0 0:00:36 0:00:36 --:--:-- 11.7M  
 6ISO download succeeded.  
 7Checking SHA256 (256-bit) checksum...  
 8% Total % Received % Xferd Average Speed Time Time Time Current  
 9Dload Upload Total Spent Left Speed  
10100 106 100 106 0 0 215 0 --:--:-- --:--:-- --:--:-- 215  
11Found it. Verifying checksum...  
12SHA256 checksum valid.  
13Checking for digital signature file...  
14% Total % Received % Xferd Average Speed Time Time Time Current  
15Dload Upload Total Spent Left Speed  
160 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0  
17curl: (22) The requested URL returned error: 404  
18Unable to fetch digital signature file.  
19Do you want to continue without signature check? (yes/no) \[yes\]  
20Checking MD5 checksums of files on the ISO image...OK.  
21Done!  
22What would you like to name this image? \[1.4-rolling-202112040649\]:  
23OK. This image will be named: 1.4-rolling-202112040649  
24Installing "1.4-rolling-202112040649" image.  
25Copying new release files...  
26Would you like to save the current configuration  
27directory and config file? (Yes/No) \[Yes\]: Yes  
28Copying current configuration...  
29Would you like to save the SSH host keys from your  
30current configuration? (Yes/No) \[Yes\]: No  
31Running post-install script...  
32Setting up grub configuration...  
33Done.  
34vyos@vyos:~$ show system image  
35The system currently has the following image(s) installed:  
36  
371: 1.4-rolling-202112040649 (default boot)  
382: 1.4-rolling-202103130218 (running image)  
39  
40vyos@vyos:~$ reboot  
41Are you sure you want to reboot this system? \[y/N\] y

Recap

To cover the major points of this article:

Selecting "Guest OS" in vSphere can present significant performance improvements or problems depending on what you choose. The selector chooses what PV hardware to provide to a VM, and it'll try to preserve compatibility and be conservative.
VM Hardware is a separate knob entirely, updating it won't make the newer hardware available without the "Guest OS" selector
Consult your NOS vendor on what to select here, if applicable, and require them to provide you documentation on why.

Some additional tangential benefits are present as a result of this change. For example, VM power actions work:

Since we're done, let's check this change into the image library:

Reference: https://blog.engyak.co/2020/10/using-vm-templates-and-nsx-t-for/

Why Automate? Reliability Approaches with the VMware NSX-T API

Wed, 24 Nov 2021 15:13:00 -0900

Why should an infrastructure engineer leverage REST APIs?

I'm sure most IT workers have at least heard of REST APIs, or heard a sales pitch where a vendor insists that while a requested functionality doesn't exist, you could build it yourself by "using the API".

Or, participate in discussions where people seemed to try and offer you a copy of The DevOps Handbook or The Unicorn Project.

They're right, but software development and deployment methods have completely different guiding values than infrastructure management. Speed of delivery is almost completely worthless with infrastructure, where downtime is typically the only metric that infrastructure is evaluated on.

We need to transform the industry.

The industry has proven that "value-added infrastructure" is a thing that people want, otherwise, services like Amazon AWS, Azure, Lumen would not be profitable. Our biggest barrier to success right now is the perceptions around reliability because there clearly is demand for what we'd call abstraction of infrastructure. We can't move as slow as we used to, but we can't make mistakes either.

Stuck between a rock and a hard place?

I have some good news - everybody's just figuring this out as they go, and you don't have to start by replacing all of your day-to-day tasks with Ansible playbooks. Let's use automation tools to ensure Quality First, Speed Second. Machines excel at comparison operators, allowing an infrastructure administrator to test every possible aspect of infrastructure when executing a change. Here are some examples where I've personally seen a need for automation:

Large-scale routing changes: if 1,000 routes successfully migrate, and a handful of routes fail, manual checks tend to depend overly (unfairly) on the operator to eyeball the entire lot
Check: Before and after routes, export a difference
Check: All dynamic routing peers, export a difference
Reverse the process if anything fails
Certificate renewals
Check: If certificate exists
Check: If the certificate was uploaded
Check: If the certificate has a valid CA chain
Check: If the certificate was successfully installed
Reverse the process if anything fails
Adding a new VLAN or VNI to a fabric
Check: VLAN Spanning-Tree topology, export a difference
Check: EVPN AFI Peers, export a difference
Check: MAC Address Table, export a difference
Reverse the process if anything fails

The neat thing about this capability is the configuration reversal - API calls are incredibly easy to process in common programming languages (particularly compared to expect) and take fractions of a second to run - so if a tested process (it's easy to test, too!) does fail, reversion is straightforward. Let's cover the REST methods before exploring the deeper stuff like gNMI or YANG.

Anatomy of a REST call

When implementing a REST API call, a client request will have several key components:

Headers: Important meta-data about your request go here, the server should adhere to any specification provided in HTTP headers. If you're building API code or otherwise, I'd recommend just setting up a standard when reviewing the list of supported fields. Examples:
Authentication Attributes
{'content-type': 'application/xml'}
{'content-type': 'application/json'}
{'Accept-Encoding': 'application/json'}
{'Cache-Control': 'no-cache'}
Resource: This is specified by the Uniform Resource Indicator, the URL component after the system is specified. A resource is the "what" of a RESTful interaction.
Body: Free-form optional text, this component provides a payload for the API call. It's important to make sure that the server actually wants it!
Web Application Firewalls (WAF) can inspect header, verb, and body to determine if an API call is safe and proper.

When implementing a REST API call, a server response will have several key components:

Headers: Identical use case, but keep in mind that headers from server to client will be following a different list.
Response Code: This should provide detail on the status of the API call.
In network automation, I strongly discourage simply trusting the response code as a means of testing for changes. It's better to make multiple GET requests to verify that the change was executed and provided the intended effects.
If implementing API-specific code, vendors will provide what each error code means specifically to them. Python supports constructing a dictionary with numeric indexes, a useful mechanism for mapping the vendor list, ex:

1httperrors = {  
2    1: ('Unknown Command', 'The specific config or operational command is not recognized.'),  
3    2: ('Internal Error', 'Check with technical support when seeing these errors.'),  
4    3: ('Internal Error', 'Check with technical support when seeing these errors.'),  
5    4: ('Internal Error', 'Check with technical support when seeing these errors.'),  
6    5: ('Internal Error', 'Check with technical support when seeing these errors.')  
7    }

Body: Ideally used for any additional detail on why the response provided executed with the status provided, but not mandatory.

Verb

In a REST API, it's important to specify the TYPE of change you intend to make prior to actually invoking it. F5 Administrators will be familiar with this, with actions like tmsh create. We have 4 major REST verbs:

Create
Read
Modify/Update
Delete

When you use a particular transport, you need to implement these verbs in a method native to that transport. This is significant when using other remote command methods like SSH (tmsh does this) or NetCONF or RESTCONF, all of which need a different method to implement.

Fortunately for us, HTTP 1.1 seems like it's been made for this! HTTP has plenty of verbs that match the above, here's a brief decoder ring.

GET: READ-only request, typically does not include a message body.
This will normally use a URI to specify what details you want to grab.
Since you're "getting" information here, typically you'd want to JSON pretty-print the output
POST: CREATE request, if you're making a new object on a remote system a message body is typically required and POST conveniently supports that.
POST should not overwrite existing data, but REST implementations vary!
POST: READ request, occasionally used when a query requires a message body.
URIs don't always cut it when it comes to remote filtered requests or complex multi-tier queries.
Cisco NX-API avoids GET as a READ verb, and primarily uses POST instead with the REST verbs in the body
PUT: UPDATE request, is idempotent. Generally does not contain a lot of change safety, as it will implement or fully replace an object.
Situations definitely exist that you want to be idempotent, and this is the verb for that.
Doesn't require a body
PATCH: MODIFY request, will only modify an existing object.
This will take considerably more work to structure, as PATCH can optionally be safely executed, but the responsibility for assembling requests safely in this manner is on the developer.
Most API implementations simply use POST instead and implement change safety in the back-end.
DELETE: DELETE request, does exactly what it sounds like, it makes a resource disappear.

Nota Bene: None of this is a mandatory convention, so vendors may implement deviations from the REST spec. For example, Palo Alto will use XML and 0-100 series HTTP codes.

Executing a REST Call

Once the rules are set, the execution of a REST call is extremely easy, here's an example:

 1curl -k -u admin https://nsx.lab.engyak.net/api/v1/alarms  
 2Enter host password for user 'admin':  
 3{  
 4"results" : \[ {  
 5"id" : "3e79618a-c89e-477b-8872-f4c87120585b",  
 6"feature\_name" : "certificates",  
 7"event\_type" : "certificate\_expiration\_approaching",  
 8"feature\_display\_name" : "Certificates",  
 9"event\_type\_display\_name" : "Certificate Expiration Approaching",  
10"summary" : "A certificate is approaching expiration.",  
11"description" : "Certificate 5c9565d8-2cfa-4a28-86cc-e095acba5ba2 is approaching expiration.",  
12"recommended\_action" : "Ensure services that are currently using the certificate are updated to use a new, non-expiring certificate. For example, to apply a new certificate to the HTTP service, invoke the  
13following NSX API POST /api/v1/node/services/http?action=apply\_certificate&certificate\_id=<cert-id> where <cert-id> is the ID of a valid certificate reported by the GET /api/v1/trust-management/certificates NS  
14X API. Once the expiring certificate is no longer in use, it should be deleted by invoking the DELETE /api/v1/trust-management/certificates/5c9565d8-2cfa-4a28-86cc-e095acba5ba2 NSX API.",  
15"node\_id" : "37e90542-f8b8-136e-59bc-5dd3b79b122b",  
16"node\_resource\_type" : "ClusterNodeConfig",  
17"entity\_id" : "5c9565d8-2cfa-4a28-86cc-e095acba5ba2",  
18"last\_reported\_time" : 1637510695463,  
19"status" : "OPEN",  
20"severity" : "MEDIUM",  
21"node\_display\_name" : "nsx",  
22"node\_ip\_addresses" : \[ "10.66.0.204" \],  
23"reoccurrences\_while\_suppressed" : 0,  
24"entity\_resource\_type" : "certificate\_self\_signed",  
25"alarm\_source\_type" : "ENTITY\_ID",  
26"alarm\_source" : \[ "5c9565d8-2cfa-4a28-86cc-e095acba5ba2" \],  
27"resource\_type" : "Alarm",  
28"display\_name" : "3e79618a-c89e-477b-8872-f4c87120585b",  
29"\_create\_user" : "system",  
30"\_create\_time" : 1635035211215,  
31"\_last\_modified\_user" : "system",  
32"\_last\_modified\_time" : 1637510695464,  
33"\_system\_owned" : false,  
34"\_protection" : "NOT\_PROTECTED",  
35"\_revision" : 353  
36}

Now - saving the cURL commands can be very administratively intensive - So I recommend some form of method to save and automate custom API calls. Quite a few more complex calls will require JSON payloads, variables, stuff like that.

Executing a Procedure

Planning the Procedure

Here we'll use the API to resolve the following alarm. I'm going to use my own REST client, found here, because it's familiar. Let's write the desired result in pseudo-code first to develop a plan:

GET current cluster certificate ID
GET certificate store
PUT a replacement certificate with a new name
GET certificate store (validate PUT)
GET certificate ID (to further validate PUT). For idempotency, multiple runs should be supported.
POST update cluster certificate
GET current cluster certificate ID

This process seems tedious, but computers don't ever get bored, and the objective here is to be more thorough than is reasonably feasible with manual execution! If you're thinking, "Gee, this is an awful lot of work!" trick rocks into doing it for you.

Let's Trick Those Rocks

Some general guidelines when scripting API calls:

Use a familiar language. An infrastructure engineer's goal with automation is reliability. Hiring trends, hipster cred, don't matter here. If you do best with a slide rule, use that.
Use libraries. An infrastructure engineer's goal with automation is reliability. Leverage libraries with publicly available testing results.
Log and Report: An infrastructure engineer's goal with automation is reliability. Report every little thing your code does to your infrastructure, and test code thoroughly.

In this case, I published a wrapper for Python requests that allows me to save API settings here, and built a script on that library. Install it first:

1python3 \-m pip install restify-ENGYAK

From here, it's important to research the API calls required for this procedure (good thing we have the steps!). For NSX-T, the API Documentation is available here: https://developer.vmware.com/apis/1163/nsx-t

NSX-T's Certificate management API also has a couple of quirks, where the Web UI and the API leverage different certificates. It's outlined here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-50C36862-A29D-48FA-8CE7-697E64E10E37.html

Since we're writing code for reliability

I'd like to outline a rough idea of where my time investment was for this procedure. I hope it helps because the focus really isn't on writing code.

50%: Testing and planning testing. I used Jenkins CI for this, and I'm not the most capable with it. This effort reduces over time, but does not reduce importance! Write your test cases before everything!
30%: Research. Consulting the VMware API docs and official documentation was worth every yoctosecond - avoiding potential problems with planned work is critical (and there were some major caveats with the API implementation).
10%: Updating the parent library, setting up the python environment. Most of this work is 100% re-usable.
5%: Managing source code, Git branching, basically generating a bread-crumb trail for the implementation for when I don't remember it.
5%: Actually writing code!

**I'm saving useful API examples in my public repository:https://github.com/ngschmidt/python-restify
**

The Code

 1# JSON Parsing tool  
 2import json  
 3  
 4# Import Restify Library  
 5from restify.RuminatingCogitation import Reliquary  
 6  
 7# Import OS - let's use this for passwords and usernames  
 8# APIUSER = Username  
 9# APIPASS = Password  
10import os  
11  
12api_user = os.getenv("APIUSER")  
13api_pass = os.getenv("APIPASS")  
14  
15# Set the interface - apply from variables no matter what  
16cogitation_interface = Reliquary(  
17    "settings.json", input_user=api_user, input_pass=api_pass  
18)  
19  
20# Build Results Dictionary  
21stack = {  
22    "old_cluster_certificate_id": False,  
23    "old_certificate_list": [],  
24    "upload_result": False,  
25    "new_certificate_id": False,  
26    "new_certificate_list": [],  
27    "new_cluster_certificate_id": False,  
28}  
29  
30# GET current cluster certificate ID  
31stack["old_cluster_certificate_id"] = json.loads(  
32    cogitation_interface.namshub("get_cluster_certificate_id")  
33)["certificate_id"]  
34  
35# GET certificate store  
36for i in json.loads(cogitation_interface.namshub("get_cluster_certificates"))[  
37    "results"  
38]:  
39    stack["old_certificate_list"].append(i["id"])  
40# We need to compare lists, so let's sort it first  
41stack["old_certificate_list"].sort()  
42  
43# PUT a replacement certificate with a new name  
44print(cogitation_interface.namshub("put_certificate", namshub_variables="cert.json"))  
45  
46# GET certificate store (validate PUT)  
47for i in json.loads(cogitation_interface.namshub("get_cluster_certificates"))[  
48    "results"  
49]:  
50    stack["new_certificate_list"].append(i["id"])  
51# We need to compare lists, so let's sort it first, then make it the difference between new and old  
52stack["old_certificate_list"].sort()  
53stack["new_certificate_list"] = list(  
54    set(stack["new_certificate_list"]) - set(stack["old_certificate_list"])  
55)  
56  
57# Be Idempotent - this may be run multiple times, and should handle it accordingly.  
58if len(stack["new_certificate_list"]) == 0:  
59    stack["new_certificate_id"] = input(  
60        "Change not detected! Please select a certificate to replace with: "  
61    )  
62else:  
63    stack["new_certificate_id"] = stack["new_certificate_list"][0]  
64  
65# GET certificate ID (to further validate PUT)  
66print(  
67    cogitation_interface.namshub(  
68        "get_cluster_certificate",  
69        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
70    )  
71)  
72# POST update cluster certificate  
73print(  
74    cogitation_interface.namshub(  
75        "post_cluster_certificate",  
76        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
77    )  
78)  
79print(  
80    cogitation_interface.namshub(  
81        "post_webui_certificate",  
82        namshub_variables=json.dumps({"id": stack["new_certificate_id"]}),  
83    )  
84)  
85  
86# GET current cluster certificate ID  
87stack["new_cluster_certificate_id"] = json.loads(  
88    cogitation_interface.namshub("get_cluster_certificate_id")  
89)["certificate_id"]  
90  
91# Show the results  
92print(json.dumps(stack, indent=4))

Get rid of certificate errors with Avi (NSX-ALB) and Hashicorp Vault!

Sun, 10 Oct 2021 17:22:00 -0800

Have you ever seen this error before?

This is a really important issue in enterprise infrastructure because unauthenticated TLS connections teach our end users to be complacent and ignore this error.

TLS Authentication

SSL/TLS for internal enterprise administration typically only addresses the confidentiality aspects of an organizational need - yet the integrity aspects are not well realized:

This is an important aspect of our sense of enterprise security, but the level of effort to authenticating information endpoints is high for TLS, so we make do with what we have.

The practice of ignoring authentication errors for decades has promoted complacency

Here's another error that enterprise systems administrators see all the time:

1ssh {{ ip }}  
2The authenticity of host '{{ ip }} ({{ ip }})' can't be established.  
3RSA key fingerprint is SHA256:{{ hash }}.  
4Are you sure you want to continue connecting (yes/no)?

This probably looks familiar too - Secure Shell (SSH) follows a different method of establishing trust, where the user should verify that hash is correct by some method, and if it changes, it'll throw an error that we hopefully don't ignore:

 1ssh {{ ip }}  
 2@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  
 3@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @  
 4@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  
 5IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!  
 6Someone could be eavesdropping on you right now (man-in-the-middle attack)!  
 7It is also possible that a host key has just been changed.  
 8The fingerprint for the RSA key sent by the remote host is  
 9SHA256:{{ hash }}.  
10Please contact your system administrator.  
11Add correct host key in known_hosts to get rid of this message.  
12Offending ECDSA key in known_hosts  
13{{ cipher }} host key for {{ ip }} has changed and you have requested strict checking.  
14Host key verification failed.

SSH is performing something very valuable here - authentication. By default, SSH will record a node's SSH hash in a file called known_hosts to ensure that the server is in fact the same as the last time you accessed it. In turn, once the server authenticates, you provide some level of authentication (user, key) afterward to ensure that you are who you say you are too. Always ensure that the service you're giving a secret to (like your password!) is authenticated or validated in some way first!

Web of Trust versus Centralized Identity

Web-of-Trust (WoT)

Web-of-Trust (WoT) is typically the easiest form of authentication scheme to start out with but results in factorial scaling issues later on if executed properly. In this model, it's on the individual to validate identities from each peer they interact with since WoT neither requires nor wants a centralized authority to validate against.

Typically enterprises use WoT because it's baked into a product, not specifically due to any particular need. Certain applications work well with it - so generally you should:

Keep your circle small
Replace crypto regularly
Leverage multiple identities for multiple tasks
e.g. separate your code signing keys from your SSH authentication keys

Pros

Easy initial set-up
Doesn't depend on a third party to establish infrastructure

Cons

The user is empowered to make both good and bad decisions, and the vast majority of users don't care enough about security to maintain vigilance
If you're in an organization with hundreds of "things to validate", you have to personally validate a lot of keys
It's a lot of work to properly validate - Ex. You probably don't ask for a person's ID every time you share emails with them
Revocation: If a key is compromised, you're relying on every single user to revoke it (or renew it, change your crypto folks) in a timely manner. This is a lot of work depending on how much a key is used.

Examples: SSH, PGP

Centralized Identity

Centralized Identity services are the sweetheart of large enterprises. Put your security officers in charge of one of these babies and they'll make it sing.

In this model, it's on the Identity Administrator to ensure the integrity of any Identity Store. They'll typically do quite a bit better than your average WoT user because it's their job to do so.

Centralized Identity services handle routine changes like ID refreshes/revocations much more easily with dedicated staffing - mostly because the application and maintainer are easy to define. But here's the rub, you have to be able to afford it. Most of the products that fit in this category are not free and require at least part-time supervision by a capable administrator.

It's not impossible, though. One can build centralized authentication mechanisms with open source tooling, it just takes work. If you aren't the person doing this work, you should help them by being a vigilant user - if an identity was compromised, report it quickly, even if it was your fault - the time to respond here is vital. Try to shoulder some of this weight whenever you can - it's an uphill hike for the people doing it and every little contribution counts.

Back to TLS and Certificates

In the case of an Application Delivery Administrator, an individual is responsible for the integrity and confidentiality of the services they deliver. This role must work hand-in-glove with Identity administrators in principle, and both are security administrators at heart.

This is really just a flowery way to say "get used to renewing and filing Certificate Signing Requests (CSRs)".

In an ideal world, an Application Delivery Controller**(ADC)** will validate the integrity of a backend server (Real Server) before passing traffic to it, in addition to providing the whole "CIA Triad" to clients. Availability is an ADC's thing, after all.

Realistically an ADC Administrator will only control one of these two legs - and it's plenty on its own. Here's one way to execute this model.

Certificate Management

Enough theory, let's do some things. First, we'll build a PKI inside of HashiCorp Vault - this assumes a full Vault installation. Here's a view of the planned Certificate Hierarchy:

From the HashiCorp Vault GUI - let's set up a PKI secrets engine for the root CA:

Note: Default duration is 30 days, so I've overridden this by setting the default and max-lifetime under each CA labeled as "TTL"
Let's create the services and user CAs:

This will provide a CSR - we need to sign it under the root CA:

Copy the resulting certificate into your clipboard - these secrets engines are autonomous, and don't interoperate - so we'll have to install it into the intermediate CA.

We install the certificate via the "Set signed intermediate" button in Vault:

Now, we have a hierarchical CA!

NB: You will need to create a Vault "role" here - https://www.vaultproject.io/docs/secrets/pki

Mega NB: The root CA should nominally be "offline" and at a minimum part of a separate Vault instance!

For this post, we'll just be issuing certificates manually. We need to extract the intermediate and root certificates to install in NSX ALB and participating clients. These can be pulled from the root-ca module:

Note: Vault doesn't come with a certificate reader as of 1.8.3. You can read these certificates with online tools or by performing the following command with OpenSSL:

1openssl x509 -in cert1.crt -noout -text

Once we have the files, let's upload them to Avi:

For each certificate, click "Root/Intermediate CA Certificate" and Import. Note that you do need to click on Validate before importing.

Now that we have the CA available, we should start by authenticating Avi itself and create a controller certificate:

Fulfilling the role of PKI Administrator, let's sign the CSR after verifying authenticity.

Back to the role of Application Administrator! We've received the certificate, let's install it in the Avi GUI!

Once we've verified the certificate is healthy, let's apply it to the management plane under Administration -> Settings -> Access Settings:

At this point, we'll need to trust the root certificate created in Vault - else we'll still see certificate errors. Once that's done, we'll be bidirectionally authenticated with the Avi controller!

From here on out - we'll be able to leverage the same process, in short:

Under Avi -> Templates -> Security -> TLS/SSL Certificates, create a new Application CSR
Ensure that all appropriate Subject Alternative Names (SANs)are captured!
Under Vault -> svc-ca -> issued-certificates -> Sign Certificate
Copy issued certificate to TLS Certificate created in the previous step
Assign to a virtual service. Unlike F5 LTM, this is decoupled from the clientssl profile.

Get an A on ssllabs.com with VMware Avi / NSX ALB (and keep it that way with SemVer!)

Sun, 19 Sep 2021 14:06:00 -0800

Cryptographic security is an important aspect of hosting any business-critical service.

When hosting a public service secured by TLS, it is important to strike a balance between compatibility (The Availability aspect of CIA), and strong cryptography (the Integrity or Authentication and Confidentiality aspects of CIA). To illustrate, let's look at the CIA model:

In this case, we need to balance backward compatibility with using good quality cryptography - here's a brief and probably soon-to-be-dated overview of what we ought to use and why.

Protocols

This block is fairly easy, as older protocols are worse, right?

TLS 1.3

As a protocol, TLS 1.3 has quite a few great improvements and is fundamentally simpler to manage with fewer knobs and dials. There is a major concern with TLS 1.3 currently - security tooling in the large enterprise hasn't caught up with this protocol yet as new ciphers like ChaCha20 don't have hardware-assisted lanes for decryption. Here are some of the new capabilities you'll like::

Simplified Crypto sets: TLS 1.3 deprecates a ton of less-than-secure crypto - TLS 1.2 supports up to 356 cipher suites, 37 of which are new with TLS 1.2. This is a mess - TLS 1.3 supports five.
Note: The designers for TLS 1.3 achieved this by removing forward secrecy methods from the cipher suite, and they must be separately selected.
Simplified handshake: TLS 1.3 connections require fewer round-trips, and session resumption features allow a 0-RTT handshake.
AEAD Support: AEAD ciphers both support integrity and confidentiality. AES Galois Counter Mode (GCM) and Google's ChaCha20 serve this purpose.
Forward Secrecy: If a cipher suite doesn't have PFS (I disagree with perfect) support, it means that a user can capture your network traffic and replay it to decrypt if the private keys are acquired. PFS support is mandatory in TLS 1.2

Here are some of the things you can do to mitigate the risk if you're in a large enterprise that performs decryption:

Use a load balancer - since this is about a load balancer, you can protect your customer's traffic in transit by performing SSL/TLS bridging. Set the LB-to-Server (serverssl) profile to a high-efficiency cipher suite (TLS 1.2 + AES-CBC) to maintain confidentiality while still protecting privacy.

TLS 1.2

TLS 1.2 is like the Toyota Corolla of TLS, it's run for forever and not everyone maintains it properly.

It can still perform well if properly configured and maintained - we'll go into more detail on how in the next section. The practices outlined here are good for all editions of TLS.

Generally, TLS 1.0 and 1.1 should not be used. Two OS providers (Windows XP, Android 4, and below) were disturbingly slow to adopt TLS 1.2, so if this is part of your customer base, beware.

Ciphers

This information is much more likely to be dated. I'll try to keep this short:

Confidentiality

(AEAD) AES-GCM: This is usually my all-around cipher. It's decently fast and supports partial acceleration with hardware ADCs / CPUs. AES is generally pretty fast, so it's a good balance of performance and confidentiality. I don't personally think it's worth running anything but 256-bit on modern hardware.
(AEAD) ChaCha20: This was developed by Google, and is still "being proven". Generally trusted by the public, this novel cipher suite is fast despite a lack of hardware acceleration.
AES-CBC: This has been the "advanced" cipher for confidentiality before AES-GCM. Developed in 1993, this crypto is highly performant and motivated users to move from suites like DES and RC4 by being both more performant and stronger. Like with AES-GCM, I prefer not to use anything but 256-bit on modern hardware
Everything else: This is the "don't bother" bucket: RC4, DES, 3DES

Integrity

Generally, AEAD provides an advantage here - SHA3 isn't generally available yet but SHA2 variants should be the only thing used. The more bits the better!

Forward Secrecy

ECDHE (Elliptic Curve Diffie Hellman): This should be mandatory with TLS 1.2 unless you have customers with old Android phones and Windows XP.
TLS 1.3 lets you select multiple PFS algorithms that are EC-based.

Matters of Practice

Before we move into the Avi-specific configuration, I have a recommendation that is true for all platforms:

Semantic Versioning

Cryptography practices change over time - and some of these changes break compatibility. Semantic versioning provides the capability to support three scales of change:

Major Changes: First number in a version. Since the specification is focused on APIs, I'll be more clear here. This is what you'd iterate if you are removing cipher suites or negotiation parameters that might break existing clients
Minor Changes: This category would be for tuning and adding support for something new that won't break compatibility. Examples here would be cipher order preference changes or adding new ciphers.
Patch Changes: This won't be used much in this case - here's where we'd document a change that matches the Minor Change's intent, like mistakes on cipher order preference.

Let's do it!

Let's move into an example leveraging NSX ALB (Avi Vantage). Here, I'll be creating a "first version," but the practices are the same. First, navigate to Templates -> Security -> SSL/TLS Profile:

Note: I really like this about Avi Vantage, even if I'm not using it here. The security scores here are accurate, albeit capped out - VMware is probably doing this to encourage use of AEAD ciphers:

...but, I'm somewhat old-school. I like using Apache-style cipher strings because they can apply to anything, and everything will run TLS eventually. Here are the cipher strings I'm using - the first is TLS 1.2, the second is TLS 1.3.

ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384

TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

One gripe I have here is that Avi won't add the "What If" analysis like F5's TM-OS does (14+ only). Conversely, applying this profile is much easier. To do this, open the virtual service, and navigate to the bottom right:

That's it! Later on, we'll provide examples of coverage reporting for these profiles. In a production-like deployment, these services should be managed with release strategies given that versioning is applied.

Static IPv4/IPv6 Addresses - Debian 11

Fri, 17 Sep 2021 16:45:00 -0800

Here's how to set both static IPv4 and IPv6 addressing on Debian 11. The new portions are outlined in italics.

First, edit /etc/network/interfaces

 1auto lo  
 2auto ens192  
 3iface lo inet loopback  
 4# The primary network interface  
 5allow-hotplug ens192  
 6iface ens192 inet static  
 7    address _**{{ ipv4.address }}**_  
 8    gateway _**{{ ipv4.gateway }}**_  
 9iface ens192 inet6 static  
10    address _**{{ ipv6.address }}**_  
11    gateway _**{{ ipv6.gateway }}**_

Then, restart your networking stack:

1systemctl restart networking

VMware NSX ALB (Avi Networks) and NSX-T Integration, Installation

Fri, 10 Sep 2021 09:00:00 -0800

Note: I created a common baseline for pre-requisites in this previous post. We'll be following VMware's Avi + NSX-T Design guide.

This will be a complete re-install. Avi Vantage appears to develop some tight coupling issues with using the same vCenter for both Layer 2 and NSX-T deployments - which is not an issue that most people will typically have. Let's start with the OVA deployment:

Initial setup here will be very different compared to a typical vCenter standalone or read-only deployment. The setup wizard should be very minimally followed:

With a more "standard" deployment methodology, the Avi Service Engines will be running on their own Tier-1 router, and leveraging Source-NAT (misnomer, since it's a TCP proxy) for "one-arm load balancing":

To perform this, we'll need to add two segments to the ALB Tier-1. one for management, and one for vIPs. I have created the following NSX-T segments, with 10.7.80.0/24 running DHCP and 10.7.81.0/24 for vIPs:

Note: I used underscores in this segment name, in my own testing both(_-) are illegal characters. Avi's NSX-T Cloud Connector will report "No Transport Nodes Found" if it cannot match the segment name due to these characters.

Note: If you configure an NSX-T cloud and discover this issue, you will need to delete and re-add the cloud after fixing the names!

Note: IPv6 is being used, but I will not share my globally routable prefixes.

First off, let's create NSX-T Manager and vCenter Credentials:

There is one thing that needs to be created on vCenter as well - a content library. Just create a blank one and label it accordingly, then proceed with the following steps:

Click Save, and get ready to wait. The Avi controller has automated quite a few steps, and it will take a while to run. If you want, the way to track any issue in NSX ALB is to navigate to Operations -> Events -> Show Internal:

Once the NSX Cloud is reporting as "Complete" under Infrastructure -> Dashboard, we need to specify some additional data to ensure that the service engines will deploy. To do this, we navigate to Infrastructure -> Cloud Resources -> Service Engine Groups, and select the Cloud:

Then let's build a Service Engine Group. This will be the compute resource attached to our vIPs. Here I configured a naming convention and a compute target - and it can automatically drop SEs into a specific folder.

The next step here is to configure the built-in IPAM. Let's add an IP range under Infrastructure -> Cloud Resources -> Networks by editing the appropriate network ID. Note that you will need to select the NSX-T cloud to see the correct network:

Those of you who have been LTM Admins will appreciate this. Avi SE also perform "Auto Last Hop," so you can reach a vIP without a default route, but monitors (health checks) will fail. The spot to configure the custom routes is under Infrastructure -> Cloud Resources -> Routing:

Finally, let's verify that the NSX-T Cloud is fully configured. An interesting thing I saw here is that Avi 21 shows an unconfigured or "In Progress" cloud as green now, so we'll have to mouse over the cloud status to check in on it.

Now that everything is configured (at least in terms of infrastructure), Avi will not deploy Service Engines until there's something to do! So let's do that:

Let's define a pool (back-end server resources):

Let's set a HTTP-to-HTTPS redirect as well:

Finally, let's make sure that the correct SE group is selected:

And that's it! You're up and running with Avi Vantage 21! After a few minutes, you should see deployed service engines:

The service I configured is also now up - In this case, I'm using Hyperglass, and I can leverage the load-balanced vIP to check and see what the route advertisement from Avi looks like. As you can see, it's firing a multipath BGP host address:

vCenter - File system `/storage/log` is low on storage space

Fri, 03 Sep 2021 20:37:00 -0800

After a recent VCSA reboot, I was seeing the infamous no healthy upstream error from vCenter.

The first place to check for issues like this is VMware's Virtual Appliance Management Interface (VAMI), located by default via HTTPS on port 5480. An administrator can use the appliance root password for this particular interface.

When reviewing this issue with the VAMI, I saw the following error:

Now, VCSA by design automatically rotates most logs available on the appliance using the open-source tool logrotate, but nothing in this directory appears to be managed:

1root@vcenter# grep /storage/log/etc/logrotate.d/*

I'd say this particular log partition is going to need some manual cleanup every now and then. To open up the CLI, SSH into vCenter and execute the following command:

1Command> shell  
2Shell access is granted to root

First, let's get an idea of how full the disks are:

Note: The -m switch converts units into Megabytes

 1root@vcenter[~]# df -m  
 2Filesystem 1M-blocks Used Available Use% Mounted on  
 3devtmpfs 5982 0 5982 0% /dev  
 4tmpfs 5993 1 5992 1% /dev/shm  
 5tmpfs 5993 2 5992 1% /run  
 6tmpfs 5993 0 5993 0% /sys/fs/cgroup  
 7/dev/sda3 46988 7199 37374 17% /  
 8tmpfs 5993 5 5988 1% /tmp  
 9/dev/mapper/dblog\_vg-dblog 15047 185 14080 2% /storage/dblog  
10/dev/mapper/vtsdb\_vg-vtsdb 10008 68 9412 1% /storage/vtsdb  
11/dev/mapper/vtsdblog\_vg-vtsdblog 4968 36 4661 1% /storage/vtsdblog  
12/dev/sda2 120 30 82 27% /boot  
13/dev/mapper/log\_vg-log 10008 9475 6 100% /storage/log  
14/dev/mapper/core\_vg-core 25063 45 23723 1% /storage/core  
15/dev/mapper/db\_vg-db 10008 507 8974 6% /storage/db  
16/dev/mapper/updatemgr\_vg-updatemgr 100273 1953 93185 3% /storage/updatemgr  
17/dev/mapper/netdump\_vg-netdump 985 3 915 1% /storage/netdump  
18/dev/mapper/lifecycle\_vg-lifecycle 100273 3364 91775 4% /storage/lifecycle  
19/dev/mapper/autodeploy\_vg-autodeploy 10008 37 9444 1% /storage/autodeploy  
20/dev/mapper/imagebuilder\_vg-imagebuilder 10008 37 9444 1% /storage/imagebuilder  
21/dev/mapper/seat\_vg-seat 10008 1185 8295 13% /storage/seat  
22/dev/mapper/archive\_vg-archive 50133 16373 31185 35% /storage/archive

The log partition is definitely full. To take an inventory of disk usage, we'll use the du utility, with the s (summarize) and m (megabytes) switches enabled, and then pass the output to sort with the n (numerical) and r (reverse) switches enabled to focus on the most important first.

1root@vcenter[/]# du -sm /storage/log/vmware/\* | sort -n -r  
22578 /storage/log/vmware/eam  
32286 /storage/log/vmware/lookupsvc  
4785 /storage/log/vmware/sso  
5781 /storage/log/vmware/vsphere-ui  
6530 /storage/log/vmware/vmware-updatemgr

Examining these folders further, quite a few of these are old and never rotated. VMware provides the following guidance on what's safe or isn't. Generally, Linux has issues with files being deleted out from under it, so obviously rotated logs can be safely removed. If this is a production system, I'd recommend calling VMware GSS instead of taking it upon yourself. The above command (du -sm * | sort -nr) can be used in any working directory to see what is filling up the logs the most. Here are a few examples of what I deleted to make room:

1rm -rf /storage/log/vmware/eam/web/localhost-2020-*  
2rm -rf /storage/log/vmware/eam/web/localhost_access.2020*  
3rm -rf /storage/log/vmware/eam/web/catalina-2020*

From here, I like to verify that space is cleared:

1root@vcenter[/]# df -m | grep /storage/log  
2/dev/mapper/log_vg-log 10008 5793 3688 62% /storage/log

Catalina and Tomcat are names for the same thing. This software package proxies inbound HTTP requests to specific applications, allowing many developers to build code without having to construct a soup-to-nuts HTTP server. Other similar (but more recent) projects include Python's Flask.

With HTTP Proxies and servers, it is useful to keep comprehensive records indicating "who did what", both for security reasons ("whodunit") and for debugging reasons. As a result, Tomcat is a serious log-hog wherever it exists, and it almost never reviews old logs. This is why I evaluated the change as relatively safe.

If this was not an appliance, I would have added a logrotate spec to automatically delete old files from this directory, but it is not recommended to alter VCSA in this way.

VMworld 2021 is right around the corner! Here are my top 10 sessions!

Wed, 25 Aug 2021 23:20:00 -0800

VMworld 2021 is online this year

I'll really miss some of the sessions and exploration we've had in past years in person, but I think VMware made the right call this year. We can expect to see a fundamental shift with online conventions - and this will need some unique strategy compared to previous years.

The Basics

I attended my first VMworld in 2016, and to describe it as information overload would be an understatement. It's only been a few years, but here's what I have to say to new VMworld attendees:

Give yourself time between sessions: it's too easy to switch between video streams at home - but it's a trap. Your brain needs time to process new information, and normally stretching your legs and walking around would help with that. After a particularly heavy session, get away from your keyboard and give yourself time to think. It's like college, if you take too many classes you will perform less effectively than if you capped out your class time.
Talk to people: The Orbital Jigsaw Discord server can serve as a water cooler of sorts here - remember that you always can learn more with others than on your own.
Be kind to your mind: I'm mentioning it twice, and I don't care. trying to absorb everything will be stressful, the single most important thing you can do is take care of yourself. Don't skip meals, don't skip time with the kids, don't skip out on rest.

VMware has provided a lot more content in the breakout sessions this year, and it's because we can't do stuff like the fun run. Here are my sessions of interest:

Fundamentally Important Sessions

At its core - I'd like to break out sessions that would be of critical importance, aforementioned biases notwithstanding:

Enhance Data Center Network Design with NSX and VMware Cloud Foundation [NET1789]
Nimish Desai is an extremely colorful presenter. In my first VMworld, I was actually wandering around the halls and heard yelling from one of the auditoriums, and decided to wander in and take a look. It turns out he was asking some questions about OSPF and I answered one right and ended up with some trucker cap he'd glued a marketing-noncompliant NSX logo onto and didn't leave the auditorium for about 3 hours. This was on NSX-V Fundamentals - for a director he is an extremely capable teacher and presenter.
I consider this (other names before it, it's basically NSX fundamentals) session every year a foundation for just about everything VMware and SDN.
NSX-T Design, Performance and Sizing for Stateful Services [NET1212]
This one has to be good. My other favorite presenter on NSX has always been Samuel Kommu, he specializes in flaying whatever SDN platform crosses his desk within an inch of its life, and then squeezing a little bit more than that out of it. He was the first engineer to get NSX-V past 40 Gigabits/s. Nicolas Michel is a capable engineer in the newer NSX-T team, they appear to be based out of EMEA, and is a total Linux and Open Source guy too. NSX-T is based almost completely on open source software and his team is working to recreate the old NSX functionality with F/OSS.
In this case, we're visiting how to build out the stateful back-end (Tier-1) services, essentially the bits that make a network "smart". NSX-T has some highly unique next-gen scaling capabilities for these service types. Packet inspection devices are the bottleneck in nearly all modern enterprise networks, this will present a fresh perspective on solving this problem!
Extreme Performance Series: vSphere Advanced Performance Boot Camp [MCL2033]
This class every year is basically required for anyone interested in their VCAP (DCV) as it handles the most important subject for virtualization - getting the absolute most value out of your equipment. It is a Tech+ pass session but probably justifies it by itself. If you're having trouble putting together the in-book subjects while studying for VCAP/VCP, this is where you want to go.

Interesting Sessions

Apply SRE’s Golden Signals for Monitoring Toward Network Operations [NET1088]
The title more or less says it all, this would be step 4 after a round-trip of fundamentals. The first thing I try to do when encountering a new technology is to make it reliable, and this is a logical progression.
(Tech+)Future-Proof Your Network with IPv6, Platform Security and Compliance [EDG1024]
If you haven't guessed, IPv6 is coming and you can't avoid it. With that out of the way, VMware's Networking and Security Business Unit (NSBU) has covered significant ground getting the rest of the company IPv6-ready. This is a Tech+ session primarily focused on SD-WAN, so if you're interested in how an enterprise can become IPv6-ready, this is where to start.
(Tech+)NSX-T Reference Designs for vSphere with Tanzu [NET1426]
NSX-T's hidden superpower is actually container networking. It's designed from the ground up with two Container Plugins - Antrea and NCP - that support container networking without complex Flannel/IPTables configurations simply to get stuff to work.
Getting Started with NSX Infrastructure as Code [NET2272]
I'll be blunt here, I've made several series of blog posts on this already, but NSX-T is a complicated animal, and it's important to build it right. In my opinion, the best way to do this is to prototype your deployment repeatedly until it's as close to perfect as you can get it.
There are two major paths to automate NSX-T here:
The platform: Ansible/Terraform helps us here to maintain configured state. In a previous life I crushed concrete cylinders to see if they're strong enough, this is like that but digital (and safer!)
The services: vRealize Automation / vCloud Director provides services on top of the base networking we provide, it is important to understand how people consume networks we build.
NSX-T and Infrastructure as Code [CODE2741]
Yes, this will take more than one session to absorb. VMware understands that - Nicolas Michel is front-ending this one too, he's working on a YouTube channel called vPackets to capture some of this automation knowledge.

Telecom Sessions

I'm breaking this out because "there are dozens of us!"

Apparently, VMware thinks there are more of us than that - and is diving head-first into the breach. VMware has developed a robust hosting and automation suite of services to help accelerate telecommunication delivery.

I'm hoping this will possibly transform smaller ISPs into more of an Edge model, where the telecom provides the pipe and "stuff" on top of it as an additional revenue source. It'd be pretty exciting - even if you don't have a 4-post rack and some cooling, you could loan some cycles from a colocation space as needed. Despite most complaints, telecommunications companies have a few strengths here, namely:

Drive. Telecom engineers do what they do to connect people to information - regardless of how one will often complain about how their internet sucks, these guys are out there working nonstop to help make things just that little bit better.
Connectivity. While this ought to be a given, do you as a customer want to deal with the stress of relocating your server farm while down-sizing offices due to COVID?
Connectivity (people) believe it or not, running cable in every major city will build up quite the Rolodex. If anyone can find a viable physical space to fit your equipment/services, it'd be the telecom company.

Before I go too far, there is a ton of sensationalism on "The Edge!(tm)" All this really means is what I've explained here - your telecommunications provider would be empowered to deploy distributed compute stacks regionally to fit your (low latency? more like cost-effective!) workload needs. This is especially important in Alaska, where reaching out to the data center the "next town over" is a microwave relay system reaching hundreds of miles.

There's also quite a bit of misinformation on 5G, which fits into my top priority session in this category:

A Tour of the Heart of the 5G Network with Nokia and VMware [EDG1935]
You probably haven't heard of this Nokia Networks. It doesn't matter, attend this session if you're interested in 5G - the architecture changes from 4G to 5G are myriad, the organization maintaining the standards (3GPP) made dramatic improvements in terms of technical design, and this will give you a bird's eye view.
Nokia Networks is a name to track in the future, VMware's NSX-T platform and Nokia's new SR-Linux platform are going to take the data center by storm. Nokia's recent interest in Open Source has culminated in a telecommunication grade workload based on Linux - and they seem to have thought of everything, model-based configuration, automated testing in a container pipeline, the sky is the limit!
Demystifying Performance: Meeting Stringent Latency Requirements for RAN [EDG2872]
I still groan every time someone states that it's "impossible to virtualize x because of latency!" We wouldn't have a connected Alaska today if we felt that wasn't a good enough reason to try. These guys succeeded.

I look forward to seeing you all there! I'll try my best to be reachable via Twitter @engyak907 and in the Orbital Jigsaw server when I can.

Managing DNS Servers with Ansible and Jenkins (Unbound, BIND)

Sun, 22 Aug 2021 22:08:00 -0800

DNS is a vital component of all computer networks. Also known as the "Internet Yellow Pages," this service is consumed by every household.

DNS services are typically deployed in several patterns to support users and systems:

DNS Forwarder: This deployment method is the most common. Everybody needs name resolution - caching and forwarding DNS results can save you bandwidth and improve localized performance. Most appliances can do this out of the box, and if they don't, try it out! It's really easy and will help you learn how DNS works.
- Use case: You don't have your own domain and use computers.
Managed Public DNS: This deployment method is a significant majority of public domains are managed this way. You pay a third-party provider to manage the authoritative registration of public DNS records
- Use case: You have a business and own a domain, but don't have any internal resources that you need to resolve.
- Use case: You have a business and own a domain, but don't want to manage publicly resolvable nameservers
Private/Internal Nameserver: This deployment method is typically enterprise-specific, but is also required for home labs and all manner of weird experiments. Since it's not on the internet, we can violate any and all manner of Internet conventions.
- The first component here is a recursive nameserver because even if you run a second server for recursive lookups, you still need a second server for recursive lookups.
- Authoritative zones: For any given domain, keep a zone file to resolve against. This will include name-to-record (forward) objects and record-to-name (reverse) objects in separate files.
A method to change everything above, this has a high benefit:effort ratio.

For this post, we'll build the structure to have an internal nameserver managed completely from source control. This is surprisingly easy to get started - performing this work with abstraction is a welcome convenience, but not initially necessary as zone files are typically very simple and the application (Bind 9 or Unbound) is only one service.

To perform this, we'll follow this procedure:

Install the service - in this case, we'll use CentOS for Bind9 (my old setup), and Debian 11 for Unbound (because Debian 11 is new).
Extract the configuration file, and then export it into source control.
Create zone files, and then export it into source control
Automate delivery from source control to what we'll now call the "DNS Worker Node"

Bind9

 1dnf install bind  
 2find / -name 'named.conf'  
 3cat /etc/named/named.conf
 4```Example named configuration file (Credit where it's due, the vast majority of this configuration has been provided by CentOS and Bind9 - I set the _forwarders, allow-query, listen-on,_ and _zone_ directives:```
 5options {  
 6        listen-on { any; };  
 7        listen-on-v6 { any; };  
 8        directory       "/var/named";  
 9        dump-file       "/var/named/data/cache_dump.db";  
10        statistics-file "/var/named/data/named_stats.txt";  
11        memstatistics-file "/var/named/data/named_mem_stats.txt";  
12        secroots-file   "/var/named/data/named.secroots";  
13        recursing-file  "/var/named/data/named.recursing";  
14        allow-query { 10.0.0.0/8; 127.0.0.1; 2000::/3; };  
15        forwarders { 1.1.1.1; 9.9.9.9; };  
16        /*  
17         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.  
18         - If you are building a RECURSIVE (caching) DNS server, you need to enable  
19           recursion.  
20         - If your recursive DNS server has a public IP address, you MUST enable access  
21           control to limit queries to your legitimate users. Failing to do so will  
22           cause your server to become part of large scale DNS amplification  
23           attacks. Implementing BCP38 within your network would greatly  
24           reduce such attack surface  
25        */  
26        recursion yes;  
27  
28        dnssec-enable yes;  
29        dnssec-validation yes;  
30  
31        managed-keys-directory "/var/named/dynamic";  
32  
33        pid-file "/run/named/named.pid";  
34        session-keyfile "/run/named/session.key";  
35  
36        /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */  
37        include "/etc/crypto-policies/back-ends/bind.config";  
38          
39};  
40  
41zone "engyak.net" in {  
42        allow-transfer { any; };  
43        file "/etc/named/engyak.net.zone";  
44        type master;  
45};

Then, let's build a zone file in source control. Please note that there are additional conventions that should be followed when creating new DNS zone records, this is just an example file that will run!

 1$TTL 2d  
 2@               SOA             ns.engyak.net. hostmaster.engyak.net  (  
 3                                1      ; serial  
 4                                3600            ; refresh  
 5                                600             ; retry  
 6                                608400          ; expiry  
 7                                3600 ) ;  
 8;  
 9;  
10engyak.net.     IN NS           ns.engyak.net.  
11ns              IN A            10.0.0.1  
12johnnyfive      IN A            10.1.1.1  
13duncanidaho     IN A            10.2.2.2

Copy the named.conf contents into a new source code repository or your existing one, preferably in an organized fashion. Ansible playbook execution is very straightforward. I'd recommend building this in source control as well - see above note about potential process improvements

 1---  
 2- hosts: ns.engyak.net  
 3  tasks:  
 4    - name: "Update DNS Zones!"  
 5      copy:  
 6        src: zonefiles/engyak.net  
 7        dest: /etc/named/engyak.net.zone  
 8        mode: "0644"  
 9    - name: "Update DNS Config!"  
10      copy:  
11        src: conf.d/ns.engyak.net/named.conf  
12        dest: /etc/named.conf  
13        mode: "0640"  
14    - name: "Restart Named!"  
15      service:  
16        name: "named"  
17        state: "restarted"

Any time you run this playbook it will download a fresh configuration and zone file, then restart Bind9.

As a cherry on top, let's make this process smart - if we want to automatically deploy changes to DNS from source control, we need a CI Tool like Jenkins. Start off by creating a new Freeform pipeline to "Watch SCM" - yes, this isn't a real repository.

That's it - add entries, live long, and prosper! Since the Ansible playbook and supporting files are fetched via source control, the only setup required on a DNS worker node is to establish a relationship between it and the CI tool, ex. SSH authentication.

Unbound

Unbound is a newer DNS server project and has quite a few interesting properties. I've been using BIND for well over a decade - and Unbound aims to change a few things, notably:

Robust public security auditing (!!!)
DNS-over-TLS (DoT)
DNS-over-HTTPS (DoH)
All Configurations are YAML
Published RFC Conformance
Snazzy Documentation
All kinds of modern software development, automated builds/fuzzing

Oddly enough, there is no features list for this software package, but pretty much everything else is impressively documented. Let's start the installation:

1apt install unbound  
2cat /usr/share/doc/unbound/examples/unbound.conf

Unbound can use the same zonefile format as BIND, so we only need to create a new config file to migrate things over. Note: This is not a production-ready configuration, it's just enough to get me started.

As I learn more about Unbound, I'll be using source control to implement changes / implement a rollback - an important benefit when making lots of mistakes!

 1# The server clause sets the main parameters.  
 2server:  
 3        verbosity: 1  
 4        num-threads: 2  
 5        interface: 0.0.0.0  
 6        interface: ::0  
 7        port: 53  
 8        prefer-ip4: no  
 9        edns-buffer-size: 1232  
10  
11        # Maximum UDP response size (not applied to TCP response).  
12        # Suggested values are 512 to 4096. Default is 4096. 65536 disables it.  
13        max-udp-size: 4096  
14        msg-buffer-size: 65552  
15        udp-connect: yes  
16        unknown-server-time-limit: 376  
17  
18        do-ip4: yes  
19        do-ip6: yes  
20        do-udp: yes  
21        do-tcp: yes  
22  
23        # control which clients are allowed to make (recursive) queries  
24        # to this server. Specify classless netblocks with /size and action.  
25        # By default everything is refused, except for localhost.  
26        access-control: 10.0.0.0/8 allow  
27        access-control: 127.0.0.0/8 allow  
28  
29        private-domain: "engyak.net"  
30        caps-exempt: "engyak.net"  
31        domain-insecure: "engyak.net"  
32  
33        private-address: 10.0.0.0/8  
34  
35        # cipher setting for TLSv1.2  
36        tls-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256"  
37        # cipher setting for TLSv1.3  
38        tls-ciphersuites: "TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_8_SHA256:TLS_AES_128_CCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256"  
39  
40# Python config section. To enable:  
41# o use --with-pythonmodule to configure before compiling.  
42# o list python in the module-config string (above) to enable.  
43#   It can be at the start, it gets validated results, or just before  
44#   the iterator and process before DNSSEC validation.  
45# o and give a python-script to run.  
46python:  
47        # Script file to load  
48        # python-script: "/etc/unbound/ubmodule-tst.py"  
49  
50# Dynamic library config section. To enable:  
51# o use --with-dynlibmodule to configure before compiling.  
52# o list dynlib in the module-config string (above) to enable.  
53#   It can be placed anywhere, the dynlib module is only a very thin wrapper  
54#   to load modules dynamically.  
55# o and give a dynlib-file to run. If more than one dynlib entry is listed in  
56#   the module-config then you need one dynlib-file per instance.  
57dynlib:  
58        # Script file to load  
59        # dynlib-file: "/etc/unbound/dynlib.so"  
60  
61# Remote control config section.  
62remote-control:  
63        # Enable remote control with unbound-control(8) here.  
64        # set up the keys and certificates with unbound-control-setup.  
65        control-enable: no  
66  
67# Authority zones  
68# The data for these zones is kept locally, from a file or downloaded.  
69# The data can be served to downstream clients, or used instead of the  
70# upstream (which saves a lookup to the upstream).  The first example  
71# has a copy of the root for local usage.  The second serves example.org  
72# authoritatively.  zonefile: reads from file (and writes to it if you also  
73# download it), primary: fetches with AXFR and IXFR, or url to zonefile.  
74# With allow-notify: you can give additional (apart from primaries) sources of  
75# notifies.  
76forward-zone:  
77      name: "."  
78      forward-addr: 1.1.1.1  
79      forward-addr: 9.9.9.9  
80auth-zone:  
81      name: "engyak.net"  
82      for-downstream: yes  
83      for-upstream: yes  
84      zonefile: "engyak.net.zone"

To automate file delivery here, we'll use a (similar) playbook for Unbound. The Jenkins configuration will not need to be modified, because the playbook will automatically be re-executed.

 1---  
 2- hosts: ns.engyak.net  
 3  tasks:  
 4    - name: "Update DNS Zones!"  
 5      copy:  
 6        src: zonefiles/engyak.net  
 7        dest: /etc/unbound/engyak.net.zone  
 8        mode: "0644"  
 9    - name: "Update DNS Config!"  
10      copy:  
11        src: conf.d/ns.engyak.net/unbound.conf  
12        dest: /etc/unbound.conf  
13        mode: "0640"  
14    - name: "Restart Unbound!"  
15      service:  
16        name: "unbound"  
17        state: "restarted"

Some Thoughts

This method of building DNS records from a source of truth does replace the master-slave (sorry guys, BIND's terms are not my own!) relationship older name servers will typically use. Personally, I like this method of propagation.

The biggest upside here is that a DNS worker node being unavailable does not prevent an engineer from adding/modifying records as long as recursive name servers support multiple resolvers.

It is eventually consistent, as the orchestrator will update every worker node for you. This may be slower or faster, depending on TTL.

The Ansible playbook I used here will kill your DNS node if you push it into an invalid configuration, so this is probably not production-worthy without additional work.

If you would rather purchase a platform instead of building this capability with F/OSS components, this is basically how Infoblox Grid works.

It'd be really neat to abstract software-specific constructs, which can be done with Python and Jinja2 (or just Ansible and Jinja2!)

NSX Advanced Load Balancer - NSX-T Service Engine Creation Failures: `CC_SE_CREATION_FAILURE` and `Transport Node Not Found to create service engine`

Mon, 05 Jul 2021 16:11:00 -0800

TL;DR

If you see either of these errors, check grep 'ERROR' /opt/avi/log/cc_agent_go_{{ cloud }} for the potential cause. In my case, the / character was not correctly processed by Avi's Golang client (facing vCenter).

The Problem

When trying to configure NSX ALB + NSX-T on my home lab, I am presented nothing but the following error:

CC_SE_CREATION_FAILURE

The Process

Avi Vantage appears to be treating this as a retriable error, attempting to deploy a service engine five times, which can be re-executed with a controller restart:

Oddly enough, vCenter doesn't report any OVA deploy attempts. The next thing to check here would be the vSphere content library:**

So far, so good. vCenter knows where to deploy the image from.

Now here's a problem - Avi doesn't provide any documentation on how to troubleshoot this yet - so I did a bit of digging and found that you can bump yourself to root by performing a:

1sudo su

Useful note: Avi Vantage is running bullseye/sid with only 821 packages listed under dpkg -l | wc -l. They did do a pretty good job with pre-release cleanup, but there are still a few oddball packages in there. I'd give it a 9/10, I'd like to see X11 not be installed but am pleased to see only Python 3!

Avi's logs are located in:

1/var/lib/avi/log
2/opt/avi/log

what I found in alert_notifications_debug.log:

 1summary: "Syslog for System Events occured"  
 2event_pages: "EVENT_PAGE_VS"  
 3event_pages: "EVENT_PAGE_CNTLR"  
 4event_pages: "EVENT_PAGE_ALL"  
 5obj_name: "avi_-Avi-se-rctbp"  
 6tenant_uuid: "admin"  
 7related uuids ['avi_-Avi-se-rctbp']  
 8[2021-04-09 20:06:30,923] INFO [alert_engine.processAlertInstance:225] [uuid: ""  
 9alert_config_uuid: "alertconfig-938cf267-e20d-4d8e-a50a-21f0f5a5b633"  
10timestamp: 1617998694.0  
11obj_uuid: "avi_-Avi-se-rctbp"  
12threshold: 0  
13events {  
14  report_timestamp: 1617998694  
15  obj_type: SEVM  
16  event_id: CC_SE_CREATION_FAILURE  
17  module: CLOUD_CONNECTOR  
18  internal: EVENT_EXTERNAL  
19  context: EVENT_CONTEXT_SYSTEM  
20  obj_uuid: "avi_-Avi-se-rctbp"  
21  obj_name: "avi_-Avi-se-rctbp"  
22  event_details {  
23    cc_se_vm_details {  
24      cc_id: "cloud-022c7b90-f987-4b15-91bb-1f1405715580"  
25      se_vm_uuid: "avi_-Avi-se-rctbp"  
26      error_string: "Transport node not found to create serviceengine avi_-Avi-se-rctbp"  
27    }  
28  }  
29  event_description: "Service Engine creation failure"  
30  event_pages: "EVENT_PAGE_VS"  
31  event_pages: "EVENT_PAGE_CNTLR"  
32  event_pages: "EVENT_PAGE_ALL"  
33  tenant_name: ""  
34  tenant: "admin"  
35}  
36reason: "threshold_exceeded"  
37state: ALERT_STATE_ON  
38related_uuids: "avi_-Avi-se-rctbp"  
39level: ALERT_LOW  
40name: "Syslog-System-Events-avi_-Avi-se-rctbp-1617998694.0-1617998694-45824571"  
41summary: "Syslog for System Events occured"  
42event_pages: "EVENT_PAGE_VS"  
43event_pages: "EVENT_PAGE_CNTLR"  
44event_pages: "EVENT_PAGE_ALL"  
45obj_name: "avi_-Avi-se-rctbp"  
46tenant_uuid: "admin"

From the looks of things - Avi is talking with NSX-T before vCenter to determine appropriate placement, which makes sense.

Update and Root Cause

With the Avi 20.1.6 release, VMware has made a lot of improvements to logging! We're now seeing this error in the GUI (Ensure that "Internal Events" is checked:

Let's take a look at the new logging. Avi's controller system leverages a series of Go modules called "cloud connectors" dedicated to that specific interface. Each one has its own log file in``` /opt/avi/log/cc_

12021-07-04T20:20:42.801Z        ERROR   vcenterlib/vcenter_utils.go:606 [10.66.0.202][avi-mgt-vni-10.7.80.0/24] object references is empty  
22021-07-04T20:20:42.819Z        ERROR   vcenterlib/vcenter_utils.go:578 [10.66.0.202][avi-mgt-vni-10.7.80.0/24] object references is empty  
32021-07-04T20:20:42.822Z        ERROR   vcenterlib/vcenter_se_lifecycle.go:432  [10.66.0.202][QH] [10.66.0.202] Network 'avi-mgt-vni-10.7.80.0/24' matching not found in Vcenter  
42021-07-04T20:20:42.822Z        ERROR   vcenterlib/vcenter_se_lifecycle.go:891  [10.66.0.202] [10.66.0.202] Network 'avi-mgt-vni-10.7.80.0/24' matching not found in Vcenter

Now, this vn-segment does exist in vCenter, so I tried the "non-escaped shell character" knowledge from years of Linux/Unix administration and reformatted it to avi-mgt-vni-10.7.80.0_24.

Since we don't get a Redeploy (please VMware!) button, I restarted the controller and all SE deployments succeeded after that.

World WiFi Day 2021!

Sun, 20 Jun 2021 09:22:00 -0800

World WiFi Day

We (human beings) have several weird superpowers, but the ability to communicate over vast distances has always fascinated me the most.

I've had the privilege of meeting some of the most truly capable pioneers in this field - but the reality here is that we're faced with a very unequal world.

Authors like William Gibson and Neal Stephenson have the right of things as well and while we're not quite living in that dystopian future, technology can become a great equalizer.

So yeah - as telecommunications operators we have the responsibility to bridge this gap!

Learn More

I'm always surprised by how much there still is to learn, even in fields I feel like I already know. Here are a few learning approaches that will help you build out a good foundation for learning Wi-Fi (and more!):

Amateur Radio: It's cheesy, sure. I'm amazed at how much my grown-up self can now do with amateur radio - I've been licensed since the '90s (KL0NS) and the community is doing so much more now than ever. When I was very young, this was a good opportunity to learn the principles of radio outdoors.
- In Alaska we have it pretty good - KL7AA is a self-provided test provider, and they sell the study book I used way back when
- Everywhere else in the US, the test costs $15 and probably takes about 2 days to study for. There's no reason not to try it out and participate. Most of the study material is free, and we even get practice tests
- ARRL provides good ways to participate
- If you join a radio club they'll find different ways to exercise your brain, and they're usually pretty fun. In addition, you'll be helping maintain emergency communication networks.
Get Certified. Also pretty cheesy, I know how people feel about IT certs and still would argue for in this case. For Wi-Fi, the CWNP organization tends to serve the same role as the Linux Professional Institute - employers don't know about them but it's really effective in terms of education.
- CWNA: Yeah, it ends in the same letter as CCNA, but I wouldn't really consider it part of the associate level tier.
  - The best resource to prepare is David Coleman's book latest edition here, but there are E-Learning bundles too: https://www.cwnp.com/buy-products/cwna

Do More

Let's just cover some volunteer opportunities here - because there's no point in building skills if you don't use them:

World Wi-Fi Day
ITDRC These guys are really neat. The IT Disaster Resource Center leverages oldie-but-goodie enterprise telecom/IT equipment to provide disaster relief all over the continental US. Check out their deployment map!
Airheads Volunteer Corp. I know this is a vendor plug, but this approach is really cool if you can travel!
United Way

On Volunteering

One of the passive effects of these approaches - you'll get better as you go. Employers nearly always constrain your learning path to what they need at the moment, often to their own detriment. They may not know what they'll need you to do next year, COVID showed us that. Volunteering not only gives you an opportunity to help others but also passively improves your skills outside of the usual "corporate playbook".

XML, JSON, YAML - Python data structures and visualization for infrastructure engineers

Sun, 06 Jun 2021 17:29:00 -0800

At some point, we can't "do it all" with one block of code.

As developers, we need to store persistent data for a variety of reasons:

We want it for later execution (or to compare it to another result)
We're sick of storing variables in code. This matters a lot more in compiled languages than runtime ones
We want the results to end up in some form of a deliverable report

Let's cover a computer science concept being used here - semaphores. Edsger Dijkstra coined this term from Greek sema(sign) and phero(bearer) (you may remember him from OSPF) to solve Inter-Process Communications(IPC) issues.

To provide a reductionist example, process A and process B need to communicate somehow, but shouldn't access each other's memory or, in the '60s, it wasn't available. To solve this problem, the developer needs to develop a method of storing variables in a manner that is both efficient and can be consistently interpreted.

Dijkstra's example, in this case, was binary, and required a schema to interpret the same data - but was not specifically limited to single binary blocks. This specific need actually influenced one of the three data types we're comparing here - consequently the oldest.

But which one do I use? TL;DR?

Spoiler alert - anyone working with automation MUST learn all three to be truly effective. My general guidance would be:

This is a personal preference, but I would highly recommend YAML for human inputs. It's extremely easy to write, and while I generally prefer JSON it's much easier to first write a document into YAML and then convert it. If you take user input or just want to get a big JSON document started, I'd do it this way.
- YAML User input drivers can also parse JSON, making this an extremely flexible approach.
JSON is good for storing machine inputs/outputs. Because all typing is pretty explicit with JSON, json.dumps(dict, indent=4) is pretty handy for previewing what your code thinks structured data looks like. Technically this is possible with YAML, but conventions on, say, a string literal can be squishy.
- YAML with name: True could be interpreted as:
  - JSON of "name": true, indicating a Boolean value
  - JSON of "name": "True", indicating a String
- Sure, this is oversimplified, and YAML can be explicitly typed, but generally, YAML is awesome for its speed low initial friction. If an engineer knows YAML really well (and writes their own classes for it) going all-YAML here is completely possible - but to me that's just too much work.
- If you use it in the way I recommend, just learn to interpret JSON and use Python's JSON library natively, and remember json.dumps(dict, indent=4) for outputs. You'll pick it up in about half an hour and just passively get better over time.
Use XML if that's what the other code is using, Python's Element and ElementTree constructs are more nuanced than dictionaries, so a package like defusedXML is probably the best way to get started. There are a lot of binary invocations/security issues with XML, so using basic XML libraries by themselves is ill-advised. xmltodict is pretty handy if you just want to convert it into another format.

Note: JSON and XML both support Schema Validation, an important aspect of semaphores. YAML doesn't have a native function like this, but I have used Python's Cerberus modules to do the same thing here.

YAML

YAML was initially released in 2001 and has gained recent popularity with projects like Ansible. YAML 1.2 was released in 2009 and is publicly maintained by the community, so it won't have industry bias (but also won't change as quickly). YAML writes a lot like Python, consuming a ton of whitespace and being particular about tags. Users either love or hate it - I typically only use it for human inputs and objects that are frequently peer-reviewed.

NOTE: one big upside to YAML with people processes is comment support. YAML supports comments, but JSON does not.

YAML is pretty easy to start using in Python. I'm a big fan of the ruamel.YAML library, which adds on some interesting capabilities when parsing human inputs. I've found a nifty way to parse using try/except blocks - making a parser that is supremely agnostic, ingesting JSON or YAML, as a string or a file!

1---  
2message:  
3 items:  
4 item:  
5      "@tag": Blue  
6      "#text": Hello, World!

 1#!/usr/bin/python3  
 2  
 3import json  
 4  
 5from ruamel.yaml import YAML  
 6from ruamel.yaml import scanner  
 7  
 8# Load Definition Classes  
 9yaml_input = YAML(typ='safe')  
10yaml_dict = {}  
11  
12# Input can take a file first, but will fall back to YAML processing of a string  
13try:  
14    yaml_dict = yaml_input.load(open('example.yml', 'r'))  
15except FileNotFoundError:  
16    print('Not found as file, trying as a string...')  
17    yaml_dict = yaml_input.load('example.yml')  
18finally:  
19    print(json.dumps(yaml_dict, indent=4))

JSON

JSON was first implemented in 2006 and is currently maintained by the IETF. Currently, Python 3 will visually represent dicts using JSON as well - making things pretty intuitive. In my experience, writing JSON is pretty annoying because it's picky.

 1{  
 2    "message": {  
 3        "items": {  
 4            "item": {  
 5                "@tag": "Blue",  
 6                "#text": "Hello, World!"  
 7            }  
 8        }  
 9    }  
10}

1#!/usr/bin/python3  
2  
3import json  
4  
5with open('example.json', 'r') as file:  
6    print(json.dumps(json.loads(file.read())))

Typically, I'll just use json.dumps(dict, indent=4) on a live dict when I'm done with it - dumping it to a file. JSON is a well-defined standard and software support for it is excellent.

Due to its IETF bias, JSON's future seems to focus on streaming/logging required for infrastructure management. JSON-serialized Syslog is a neat application here, as you can write it to a file as a single line, but also explode for readability, infuriating grep users everywhere.

XML

XML is the oldest data language typically used for automation/data ingestion, and it really shows. XML was originally established by the W3C in 1998 and is used for many document types like Microsoft Office.

XML's document and W3C bias read very strongly. Older Java-oriented platforms like Jenkins CI heavily leverage XML for semaphores, document reporting, and configuration management. Strict validation (MUST be well-formed) required for compiled languages to synergize well with the capabilities provided. XML also heavily uses HTML-style escaping and tagging approaches, making it familiar to some web developers.

XML has plenty of downsides. Crashing on invalid input is generally considered excessive or "Steve-Ballmer"-esque, making the language favorable for mission-critical applications where misinterpretation of data MUST not be processed, and miserable everywhere else. For human inputs, it's pretty wordy which impacts readability quite a bit.

Schemas

XML has two tiers of schema - Document Type Definition (DTD) and XML Schema. DTD is very similar to HTML DTDs and provides a method of validating that the language is correctly used. XML Schema definitions (XSD) provide typing and structures for validation and is a more commonly used tool.

Python Example

XML Leverages the Element and ElementTree constructs in Python instead of dicts. This is due to XML being capable of so much more, but it's still pretty easy to use:

XML Document:

1<?xml version="1.0" encoding="ISO-8859-1" ?>  
2<message>  
3    <items>  
4        <item tag="Blue">Hello, World!</item>  
5    </items>  
6</message>

 1#!/usr/bin/python3  
 2  
 3from defusedxml.ElementTree import parse  
 4import xmltodict  
 5import json  
 6  
 7document = parse('example.xml').getroot()  
 8  
 9print(document[0][0].text + " " + json.dumps(document[0][0].attrib))  
10  
11file = open('example.xml', "r").read()  
12print(json.dumps(xmltodict.parse(file), indent=4))

After using both methods, I generally prefer using xmltodict for data processing - it lets me use a common language, Python lists and dicts to process all data regardless of source, allowing me to focus more on the payload. We're really fortunate to have this fantastic F/OSS community enabling that!

Troubleshooting with VMware NSX ALB/Avi Vantage

Sat, 22 May 2021 15:37:00 -0800

NSX Advanced Load Balancer - Logging and Troubleshooting Cheat Sheet

Get into the OS Shell (all elements)

1sudo su

Controller Log Locations

Note: Everything in /var/lib/avi/logs is managed by Elasticsearch. I wouldn't mess with it.

Events published to the GUI: /var/lib/avi/logs/ALL-EVENTS/

The primary log directory for Avi Vantage Controllers is /opt/avi/log. As this feeds into Elasticsearch, they have file outputs for every severity level. An easy way to get data on a specific object would be to build a grep statement like this:

1grep {{ regex }} /opt/avi/log/{{ target }}

alert_notifications_*: Summarized problems log. Events are in a json format!

Troubleshooting Deployment Failures

avi-nsx.*: Presumably for NSX-T integration. further investigation required
cloudconnectorgo.*: Avi's cloud connector is pretty important given their architecture. This is where you can troubleshoot any issues getting a cloud turned up, or any initial provisioning issues.
vCenter*: vCenter write mode activity logs. Look here for SE deployment failures in a traditional vSphere cloud.

Service Engines

Troubleshooting

Checking the Routing Table

NSX ALB / Avi uses FRRouting (7.0 as of release 20.1) over network namespaces to achieve management/data plane separation and VRF-Lite. To access the data plane, you will need to change namespaces! Unlike NSX-T, this doesn't happen over docker namespaces. This means that the follow commands work in both as root:

Show all VRF+Namespaces ip netns show
Send a one-shot command to the namespace: ip netns exec {{ namespace }} {{ command }} Example: ip netns exec 'ip route show'
Start a shell in the desired namespace: ip netns exec {{ namespace }} {{ shell }} Example: ip netns exec avi_ns1 bash

After in the bash shell, all normal commands apply as if there was no namespace/VRF.

For more information on Linux Network Namespaces, here's a pretty good guide: https://www.opencloudblog.com/?p=42

Logging

All SE logging is contained in /var/lib/avi/log. Here are the significant log directories there:

IMPORTANT! bgp: This is where all the routing protocol namespace logging from FRRouting lands.
traffic: This one's pretty touch to parse and it's better to use Avi's Elasticsearch instead.

Conclusion

Avi Vantage has a pretty solid logging schema, but is very much a growing product. These logs will eventually be exposed more fully to the GUI/API, but for now it's handy to grep away. I'll be updating this list as I find more.

VMware NSX Advanced Load Balancer - Installation

Sat, 15 May 2021 20:54:00 -0800

Pre-Requisites

Before beginning the Avi installer, I configured the following in my environment:

Management Segment (NSX-T Overlay). This is set up with DHCP for quick automatic provisioning - no ephemeral addresses required
Data Segments (NSX-T Overlay). Avi will build direct routes to IPs in this network for vIP processing. I built 3 -
Layer 2 Cloud (attached to Tier-1)
NSX Integrated (attached to Tier-1)
Layer 3 Cloud (attached to Tier-0)

Avi also supports automatic SE deployment - which means that automatic IP configuration is important. Avi supports SLAAC (IPv6) and DHCP (IPv4) for this purpose.

NSX-T is unsurprisingly symbiotic here. I have built a dedicated Tier-1 for NSX ALB, and we're going to provide DHCP services via the Tier-1 router. If this was a production deployment or a VVD-compliant SDDC, this should be performed with a DHCP relay. I just haven't set aside time to deploy DHCP/IPAM tools for reasons that are beyond me.

The following changes are performed on the Tier-1 Logical Router. This step is not required for external DHCP servers!

The following changes are to be performed on the Logical Segment.

If production, DHCP relay is selectable from the following screen:

Installation

Avi Controller

VMware provides a prepackaged OVA for the Vantage controller - and it's a pretty large appliance. 24 GB of memory and 8 vCPUs is a lot of resourcing for a home lab. There are no sizing options here.

Installation is pretty easy - once the OVA is deployed, I used my CI/CD pipeline and GitHub to deploy DNS updates and logged right into the installation wizard.

AVI version 20.1.5 has changed the installer approach from the above to this. When "No cloud setup" is selected, it still insists on configuring a new cloud. This isn't too much of a problem:

Note: This passphrase is for backups - make sure to store it somewhere safe!

From here, we configure vCenter's integration:

Let's ensure that Avi is connected to vCenter and has no issues. Note: VMware recommends write-mode for vCenter clouds.

After install, it's useful to get a little oriented. Up in the top left of the Avi Vantage GUI. In the top left, you'll find the major configuration branches by expanding the triple ellipsis. Get familiar with this part of the GUI - you'll be using it a lot!

Patching

Before we build anything, I prefer to load any patches (if applicable) prior to building anything. This should help avoid any software issues on deployment, and patching is usually simpler/lower impact if you have no configuration yet.

Avi Vantage really excels here - this upgrade process is pretty much fully automated, with extensive testing. As a result, it's probably going to be slower than "manual" upgrades, but is definitely more reliable. Our industry really needs to get over this - If you have a good way to keep an eye on things while keeping busy, you're ahead of the curve!

We'll hop on over to Administration -> Controller -> Software:

While this upgrade takes place - Avi's controller will serve up a "Sorry Page" indicating that it's not available yet - which is pretty neat.

When complete, you should see this:

Avi Clouds

Clouds are Avi's construct for deployment points - and we'll start with the more traditional one here - vCenter. Most of this has already been configured as part of the wizard above. Several things need to be finished for this to run well, however:

Service Engine Group - here we customize service engine settings
IPAM - Push IP address, get a cookie

SE Group Changes are executed under Infrastructure -> SE Groups. Here I want to constrain the deployment to specific datastores and clusters.

IPAM is located in two places, Templates -> Profiles -> IPAM/DNS Profiles (bindable profile):

Ranges are configured under Networks. If you configure a write-access cloud, it'll scan all of the port groups and used IP ranges for you. IP ranges and Subnets will still need to be configured and/or confirmed:

Note: This IPAM profile does need to be added to the applicable cloud to leverage auto-allocate functions with vIPs.

Avi Service Engines

Now that the setup work is done, we can fire up the SE deployments by configuring a vIP. By default, Avi will conserve resources by deploying the minimum SEs required to get the job done - if there's no vIP, this means none. It takes some getting used to!

Once the vIP is applied, you should see some deployment jobs in vCenter:

Service engines take a while to deploy - don't get too obsessive if the deployment lags. There doesn't appear to be a whole lot of logging to indicate deployment stages, so the only option here is to wait it out. If a service engine doesn't deploy quite right, delete it. This is not the type of application we just hack until it works - I did notice that it occasionally will deploy with vNICs incorrectly configured.

From here, we can verify that all service engines are deployed. The health score will climb up over time if the deployment is successful.

Now we can build stuff!

Leveraging Hyperglass and NSX-T!

Sun, 09 May 2021 16:42:00 -0800

For this example deployment, I'll be using my NSX-T Lab as the fabric, VyOS for the Overloaded Router role, and trying out hyperglass:

Installation (VyOS)

I already have a base image for VyOS with its management VRF set up - and updating the base image prior to deployment is a breeze due to the vSphere 7 VM Template Check Out Feature.

In this case, I'll deploy to an NSX-T External Port and peer up, with fully implemented ingress filtering:

Export Filters - Permit all prefixes:

Import Filters - don't trust any prefixes from this router:

Set in the correct directions:

Configure the BGP Neighbors:

From here, we build the VNF, by adding the following configuration:

 1protocols {  
 2    bgp 64932 {  
 3        address-family {  
 4            ipv4-unicast {  
 5                maximum-paths {  
 6                    ebgp 4  
 7                }  
 8            }  
 9            ipv6-unicast {  
10                maximum-paths {  
11                    ebgp 4  
12                }  
13            }  
14        }  
15        neighbor 10.7.2.1 {  
16            remote-as 64902  
17        }  
18        neighbor 10.7.2.2 {  
19            remote-as 64902  
20        }  
21        neighbor x:x:x:dea::1 {  
22            address-family {  
23                ipv6-unicast {  
24                }  
25            }  
26            remote-as 64902  
27        }  
28        neighbor x:x:x:dea::2 {  
29            address-family {  
30                ipv6-unicast {  
31                }  
32            }  
33            remote-as 64902  
34        }  
35        timers {  
36            holdtime 12  
37            keepalive 4  
38        }  
39    }  
40}

Then, let's verify that BGP is working:

 1  
 2vyos@vyos-lg-01:~$ show ip bgp summary  
 3  
 4IPv4 Unicast Summary:  
 5BGP router identifier 10.7.2.254, local AS number 64932 vrf-id 0  
 6BGP table version 156  
 7RIB entries 75, using 14 KiB of memory  
 8Peers 4, using 85 KiB of memory  
 9  
10Neighbor             V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt  
1110.7.2.1             4      64902       278       272        0    0    0 00:11:31           40       42  
1210.7.2.2             4      64902        16        13        0    0    0 00:00:16           39       42  
13x:x:x:dea::1 		 4      64902       234       264        0    0    0 00:11:43 NoNeg  
14x:x:x:dea::2 		 4      64902       283       368        0    0    0 00:11:43 NoNeg  
15  
16Total number of neighbors 4

The VNF is configured! Now, we'll follow the application maintainer's instructions for installation: https://hyperglass.io/docs/getting-started

The documentation for install is pretty good - but some customization is still required. I built the following configuration files out - hyperglass leverages YAML as a configuration file format, examples are here. I did make some changes:

Some combination of VyOS 1.4, MP-BGP, and/or VRF-lite changed the syntax for the BGP views around. Setting a commands file fixes this.
VyOS driver is appending a host mask (/32, /128) on routes with no prefix specified.
NB: I reached out to the maintainer (Matt Love) and he informed me that this was configurable per-VRF using the force-cidr option.

This particular tool has been extremely useful to me, as NSX-T still lacks comprehensive BGP visibility without CLI access - and even if it didn't, this will provide consumers an easy way to validate that prefixes have propagated, and where.

PSA: PAN-OS Drops BGP peers with an invalid NLRI / Always filter inbound prefixes from Avi Vantage

Sun, 02 May 2021 21:31:00 -0800

If Avi Vantage IPAM cannot allocate an address for a new vIP, it will advertise an all-zeros host address - 0.0.0.0/32:

This will cause Palo Alto PAN-OS to restart a peer - even if it is not the immediate downstream prefix. Palo Alto uses routed as their dynamic routing engine - so this is probably default behavior inherited from there:

 1**** EXCEPTION   0x4103 - 57   (0000) **** I:008e7cd1 F:00000004  
 2qbmlpar2.c 1352 :at 20:54:21, 2 May 2021 (1822572648 ms)  
 3UPDATE message contains NLRI of 0.0.0.0.  
 4  
 5**** PROBLEM     0x4102 - 46   (0000) **** I:008e7cd1 F:00000004  
 6qbnmmsg.c 1074 :at 20:54:21, 2 May 2021 (1822572648 ms)  
 7NM has received an UPDATE message that failed to parse.  
 8Entity index               = 1  
 9Local address              = 10.6.64.9  
10Local port                 = 0  
11Remote address             = 10.6.64.12  
12Remote port                = 0  
13Scope ID                   = 0  
14  
15**** EXCEPTION   0x4102 - 71   (0000) **** I:008e7cd1 F:00000020  
16qbnmsnd2.c 167 :at 20:54:21, 2 May 2021 (1822572648 ms)  
17A NOTIFICATION message is being sent to a neighbor due to an unexpected  
18problem.  
19NM entity index       = 1  
20Local address         = 10.6.64.9  
21Local port            = 0  
22Remote address        = 10.6.64.12  
23Remote port           = 0  
24Scope ID              = 0  
25Remote AS number      = 64905  
26Remote BGP ID         = 0X0A06400C  
27Error code            = UPDATE Message Error (3)  
28Error subcode         = Invalid Network Field (10)

This could cause a network outage for all subtending networks on this peer. Consider this a friendly reminder to always leverage route filtering between autonomous systems!

Unfortunately, strict import filters on PAN-OS did not resolve this issue.

NSX-T Edge Transport Node Packet Captures

Sun, 02 May 2021 17:36:00 -0800

NSX-T Edge Transport Node Packet Captures

NSX-T Edge nodes have a rudimentary packet capture tool built in to the box. It is important to have a built-in tool here, as GENEVE encapsulation will wrap just about everything coming out of a transport node.

NSX-T's CLI guide indicates the method for packet captures - from here we can break it down to a few steps:

Find the VRF you want to capture from
Find the interface in that VRF you want to capture from
Capture from this interface!

1get logical-routers  
2vrf {{ desired VRF }}  
3get interfaces  
4set capture session 0 interface {{ interface-id }} direction dual  
5set capture session 0 file example.pcap

The result will be placed in:

1/var/vmware/nsx/file-store/

I do have some notes to be aware of here:

Be careful with packet captures! This is on an all-CPU router - so isolating the device before capturing packets is a wise choice. We can do that with NSX-T, we just need to remember to.
It's possible to use tcpdump-based packet filters instead of a wholesale capture - just replace the final line with a command similar to this:

1set capture session 0 file example.pcap expression port 179

PAN-OS IPv6 Error: bgp peer local address 0:0:0:0:0:0:0:0 does not belong to interface

Sun, 11 Apr 2021 09:25:00 -0800

When encountering this error, please ensure that "Enable IPv6" is set under interfaces:

Hope this helps! Happy IPv6ing!

VMware NSX Advanced Load Balancer - Overview

Sat, 03 Apr 2021 09:03:00 -0800

Load Balancing is Important

Load balancing is an important aspect of network mobility.

How is a network useful if you can't move around within it?

Cellular networks lose their appeal if you drop connectivity every time you roam between towers
Wi-Fi networks are designed to facilitate smaller-scale movements. Imagine if you had to sit still for your Wi-Fi to work

Network Movements also facilitate migrations between services - as a consumer of a network service, frequent cutovers occur without your knowledge:

Infrastructure upgrades: Firewalls, routers, switches constantly need to be bumped up to higher speeds, and feeds
Preventing outages: Network "Maintenance Mode"

As computer networks get more complex - SDN is important for the orchestration of these changes or "movements". A distributed, off-box, dedicated management and control plane is essential to tracking "customers" in a scalable fashion - but load balancing is special here.

Most of our consumed services today leverage load balancers to "symmetrify" network traffic to accommodate nodes that do not support them. This can solve a lot of problems large enterprises have:

Need to scale firewalls past 2?
Need to scale firewalls in any public cloud?
Imperfect link balancing with ECMP hashing?
Want to prefer an ISP over another, but use both?

These problems are all solvable by the right load balancer platform - but are infrastructure specific. Load balancers often solve application-specific problems, including:

HTTP Transforms
TLS Quality Enforcement / Consolidated Stack
"Diet" Acceleration, e.g. HTTP Compression

Stateless apps work perfectly without some form of load balancer/ingress controller but still benefit greatly from a discrete point to ingest data as well.

NSX Advanced Load Balancer Differentiating Points

Avi Networks was founded in 2012 with the goal of providing a software load balancer designed from the ground up to leverage Software-Defined Networking (SDN) capabilities. Every aspect of the platform's design appears to eschew this - the company clearly wanted to perform a totally new platform without any need for maintaining legacy platforms. In 2019, VMware acquired Avi Networks and is rebranding the platform to "NSX Advanced Load Balancer".

Here are some clear differentiating points I have found with the platform so far:

Enterprise (Web) Oriented - Some load balancing platforms, like Kemp Technologies and Loadbalancer.org focus on clear, common enterprise needs and executing as effectively as possible; instead of "boiling the ocean" with a more feature-complete platform. If this is you as a customer, you can expect significant cost and quality improvements due to this more narrow focus - but Service Providers and specialty customers may be turned off by this.
This product is designed for self-service, with robust management plane multi-tenancy
This is a VMware product, so Avi is diving head-first into providing high-quality Kubernetes support
Offloaded Control Plane: So far, this is a big one for me personally. I'm continually amazed as to how much rich data can be extracted simply by offloading telemetry processing to a controller. Logging and Analytics do not impact data plane performance and have minimal impact on sizing/costs due to per Service Engine licensing
Software-only Kitchen Sink: Few load balancing platforms can support all clouds, KVM, K8s, Cisco ACI, Mesosphere, Acropolis, and OpenStack with direct support. Usually, the best we can hope for with a KVM install is an ISO and a prayer. This is refreshing.
Support for dynamic routing: The vast majority of load balancers on the market don't natively support this, and specific implementations like anycast or multi-site load balancing stand to benefit from this particular feature.
Global Server Load Balancing (GSLB) allows an engineer to control which site traffic may route to with anycast DNS. This provides them the ability to perform application-level capacity management with multiple sites in one solution.

Design Elements

Controller

This is Avi's brain and the primary reason for using a platform like Vantage - the control and management planes are, by default, managed by an offboard controller. The following functions are available here, with no performance penalty to the data plane:

Central Configuration Management, all locations, all the time.
Configure BGP once
Configure routes once
Configure vIPs once
Configure hardening (logging, TLS settings, passwords) once
Monitoring of vIPs, if a service is down relocate it
Software Lifecycle Management
IP Address Management
Periodic monitoring for common issues
Per Virtual Service extensive Analytics (Avi Enterprise only). They are running ElasticSearch on-box to achieve this, it's pretty neat.

NB: Avi Release 20.1.4 has <900 Debian packages (based on bullseye/sid), so they are running a little lean but could do more cleanup. 20.1.5 is down to 820 - so they are working on this.

Service Engine

Generally, these components do work. Structurally, these appliances are running Debian bullseye/sid with load balancer processes as Docker images. They're running the same edition of FRRouting as NSX-T - with the same approximate OS edition.

Service Engines do:

Report in to the AVI controller
Perform actual load balancing functions

NB: Avi Release 20.1.5 is much leaner than prior releases, and SEs typically have a much more compressed install base. 515 Debian packages here - almost in line with NSX-T 3.1.2!

IPv6

AVI Controller UI and vCenter/NSX-T Interaction have hard-coded IPv4 Constructs, 20.1.5 introduces preliminary support for IPv6, but VMware's NSBU is usually ahead of everyone else here. I'll be testing vCenter + IPv6 in a later post.
AVI Controllers appear to pick up an IPv6 address via SLAAC
This platform appears to have full data-plane support.

Deployment Methodology

Management/Control Plane

No orchestrator pre-sets will be used here - per the Avi NSX-T Integration Guide. The primary reason for my doing this is as a more thorough test of this platform - I'll be deploying 3 "Clouds":

Layer 2 Cloud (Typical A/P Load Balancer Deployment)
Layer 3 Cloud (MP-BGP Load Balancer Deployment)
NSX-T Cloud (NSX-T Integrated Deployment)

Avi Vantage designates any grouping of infrastructure presets as a "Cloud", which can have its own tenancy and routing table. This construct allows us to allocate multiple infrastructures to each administrative tenant or customer. This access is decoupled from "Tenant", which is the parent for RBAC.

Data Plane Topologies

The Avi Vantage VCF Design Guide 4.1 indicates that service engines should be attached to a tier-1 router as an overlay segment. The primary reason for this has to do with NSX-T and Avi's integration - in short, the Avi controller invokes the NSX-T API to add and advertise static routes to each service engine to handle dynamic advertisement.

Design Pattern: Looking Glasses

Mon, 22 Mar 2021 08:34:00 -0800

It's probably safe to say that service provider networking is pretty unique.

One particular design pattern - Looking Glasses - is extremely useful for complex dynamically routed networks.

I'd really like to shift the gatekeeping needle here - networks that are complex enough to benefit from a looking glass should move to:

100 Routing table entries globally
Some vague preference towards reliability
Dynamic Routing (BGP is preferred)

In any small to medium enterprise, I'd posit that the only thing truly preventing benefits, in this case, is the lack of dynamic routing adoption, primarily because pre-packaged offerings in this range don't have an "easy button" for implementing them. This lack of accessibility causes a real problem with SMB networking, as reliability features stay out of their reach.

Design Pattern: Looking Glass

A Network "Looking Glass" is a type of web server that responds to user requests, providing externalized (without userspace access to network equipment) to an authenticated or unauthenticated client. This allows clients to view BGP meta-data, routing tables to ensure outbound advertisements between Service Providers have propagated.

Here's my starting point for this design pattern.

History (non-inclusive)

Note: I don't have everything here. It seems most Looking Glasses were stood up silently by telecommunications companies. They're searchable, but I can't find any citable data on when they started out.

Looking Glass software has been publicly available since 2000, with the MRLG package reaching public availability
The University of Oregon set up a public route visibility project called Route Views to expand public visibility to Looking Glasses as early as 1999, with numerous citations in academic research driving the majority of network reliability knowledge we use today.
BGPView.io starts up in 2016
BGP Hijacking, whether malicious or not, is causing many more outages than it used to as the internet grows in scope.

Form

Least (Zero) Privilege Access to a network services routing table, searchable via API and/or GUI

Forces

Few Network Operating Systems (NOS) have a read-only permission set that also allows access to the RIB/FIB/Routing Table
Multi-tenancy should be addressed in a scalable, easy-to-manage manner
This shouldn't take the network down

Of these forces, #1 is probably the biggest. Since we cannot force all of the networking industry titans (yet) to provide a permission set that will facilitate this use - I'd propose the following approach:

In this solution, I'm proposing some additional safeguards/scale-guards to make sure that the approach will not be harmful to a "host" network. In addition to implementing the looking glass, I'd propose the deployment of a series of Virtual Network Functions (VNFs) scaled out with monitored routing tables. This is where the collectors would interact - if the physical network doesn't allow any inbound prefixes from the VNF, it's easy enough to build a solution to safely collect from it. There are tons of VNF options here - as we only need BGP capability and a collection method.

Unearned Uptime - Present and Future Design Patterns

Sat, 13 Mar 2021 21:31:00 -0900

After all that meatspace talk, let's look at a few technical solutions and why they might not meet business needs in a specific setting.

Shared Control Planes / Shared Failure Plane

Shared Control Plane design patterns are prolific within the networking industry - and there's a continuum. Generally, a control plane between devices should be designed with reliability in mind, but most shared control plane implementations tend to have "ease of administration" as intent instead of reliability. Here are some common examples.

Stacking

"Stacking" implementations represent an early industry pattern where (typically) campus deployments weren't entirely large enough to justify a chassis switch but still wanted enough lateral bandwidth to eliminate a worry point. Primary motivations for "stacking" were:

Single Point of Administration
Linear scale-out costs

Stacking was an artifact from when software like Ansible, Cisco DNA, ArubaOS-CX/NetEdit, etc. didn't exist from within the industry. Significant downsides exist to stacking software, including:

Tight coupling with software, often a total outage or a many-step ISSU upgrade path
Software problems take the whole stack down
Stacking cables are expensive and proprietary

Stacking is still a pretty good, viable technology for small to medium campus networks. One particular technology I have found interesting is Aruba's Spine and Leaf design, leveraging Aruba's mobility tunnel features to handle anything that needs to keep an IP address.

MC-LAG

Multi-Chassis LAG is a pretty contentious issue within the industry.

Note: In Service Provider applications, Layer 2 Loop Prevention is a foundational design pattern for delivering Metro Ethernet services by creating a loop-free single endpoint path. I'm not covering this design pattern, as it's a completely different subject. In this case, I'm illustrating Data Center/Private Cloud network design patterns, and then tangentially Campus from there.

MC-LAG as a design pattern isn't all that bad compared to some - however, some applications of MC-LAG in the data center turn out to be fairly problematic.

Modern Data Center Fabric Switching

Given the rise of Hyper-Converged Infrastructure - we're actually seeing data center hardware get used. Prior to this last generation (2012-onwards) just "being 10 Gig" was good enough for most use cases. Commodity server hardware wasn't powerful enough to really tax fabric oversubscribed switches.

...or was it? Anybody remember liking Cisco FEXes? TRILL? 802.3br?

Storage Area Networks (SAN) offloaded all compute storage traffic in many applications, and basically constituted an out-of-band fabric that was capable of 8-32Gbits/s.

The main problem here is Ethernet. Ethernet forwarding protocols aren't really capable of non-blocking redundant forwarding. This is because there is no routing protocol. Fiber Channel will use either IS-IS or SPF in most cases for this purpose, and hosts participate in this routing protocol.

The biggest change that this has - Fiber Channel can have two completely independent fabrics, devoid of interconnection. This allows an entire fabric to go completely offline with no issues.

MC-LAG goes in a completely different direction - forcing redundant Ethernet switches to share a failure plane. With Data Centers, the eventual goal for this design pattern is to move to this "share-nothing" approach, eventually resulting in EGP or IGP participation by all subtending devices in a fabric.

Now - we don't have that capability in most hypervisors today. Cumulus does have a Host Routing Implementation, but most common hypervisors have yet to adopt this approach. VMware, Amazon, Microsoft, and Cumulus all contribute to a common routing code base (FRRouting) and are using it to varying extents within their networks to prevent this "Layer 2 Absenteeism" from becoming a workload problem. Of these solutions - VMware's NSX-T is probably the most prolific solution if you're not a hyperscaler that can develop your own hypervisor / NOS combination like Amazon/Microsoft: https://nsx.techzone.vmware.com/

Closing Notes

Like it or not, these examples are perfectly viable design patterns when used properly. Given industry trends and some crippling deficiencies with Giant-Scale Ethernet Topologies in large-scale data center and campus networks, we as network designers must keep an eye to the future, and plan accordingly. In these examples, we examined (probably very for some) tightly coupled design patterns used in commodity networks, and where they commonly fail.

If you use these design patterns in production - I would strongly recommend asking yourself the following questions:

What's the impact of a software upgrade, worst-case?
What happens if a loop is introduced?
What's the plan for removing that solution in a way that is not business invasive?
What if your end-users scale beyond the intended throughput/device count you anticipated when performing that design exercise?

Hopefully, this explains some of the why behind existing trends. We're moving to a common goal - an automatable, reliable, vendor-independent fabric for interconnection of network devices using common protocols - and nearly all of the weirdness around this can be placed at the networking industry's feet - We treat BGP as this "protocol of the elites" instead of teaching people how to use EGPs. We (the networking industry) need to do more work to become more accessible to adjacent industries - They'll be needing us really soon if they don't already.

Unearned Uptime: Letting Old Ideas Go

Sat, 13 Mar 2021 18:18:00 -0900

We don't always earn reliability with the systems we deploy, design, and maintain

Infrastructure reliability is a pretty prickly subject for the community - we as engineers and designers tend to anthropomorphize, attach, and associate personal convictions with what we maintain. It's a natural pattern, but it inflicts a certain level of self-harm when we fail to improve upon the platforms that serve as the backbone to those we support.

There are two major problems I perceive with regards to translating unearned uptime to reliability

History
Ego
Architecture (later post)

Throughout this article, I'll cover these problems and then transition into common examples of "unearned uptime" in the industry. These are not "networking" issues - it's an infrastructure issue. We have the same problems with most civil structures, interchanges, runways, etc.

The idea that we didn't earn reliability delivered to the business is one thing that we as infrastructure engineers and designers aren't particularly comfortable with.

History

It doesn't have a problem! It's been working fine for years!

Credit: Marc Olivier-Jodoin

Infrastructure needs routine replacement to function correctly

Consumers rarely notice issues with infrastructure until they've gotten to be truly problematic. An easy example of this is asphalt concrete (or bitumen, depending on where you live).

The material itself is relatively simple, rock aggregate + oil - but it's pretty magical in terms of usefulness. Asphalt itself functions as a temporary adhesive, bonding to automotive tires and making roads really safe by shortening stopping distances. The composite material is also flexible, allowing the ground below it to shift to an extent - which means that places with more dynamic geology.

We don't really think about wear to this surface as consumers after it's been installed. Public works / Civil Engineers sure do, because it's their job, but think about it - if you drive your car over a residential street three times a day, that's probably over 4 metric tons of material that the road has to withstand in a day. This wear adds up! A typical residential (neighborhood) street will see over 15,000 metric tons of weight per year.

The sheer scale of road wear is utterly staggering. This GAO Report on Weight Enforcement illustrates how controlling wear (usage) is a method of conveying importance, but that doesn't really work all that well for us...

Practical IT Applications

When designing technology infrastructure, especially as a service provider, you want to encourage usage.

Usage drives bigger budgets and your salary! Ultimately, wear with tech infrastructure is going to be about the same regardless of load. Scarcity economics don't work particularly well in IT.

To solve the history problem, you want to convince business line owners to desire and delight in what you provide.

The antithesis to "customer delight" in this case is often this big guy: By User:MrChrome, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=33206669

Fun fact, the Cisco 6500 is a lot older than you'd think, entering service in 1999. For more: https://en.wikipedia.org/wiki/Catalyst_6500

Cisco 6500 series switches were simply too reliable. The Toyota Camry of switches, Cisco's 6500s lived everywhere, convincing executives that it was totally okay to skip infrastructure refreshes, much to the chagrin of Infrastructure Managers worldwide.

The Solution - Messaging

We shouldn't be waiting for stuff to fail to replace it - it's time to get uncomfortable and speak to consumers. Most humans are intelligent - let's help them understand why we care about 25/100 Gigabit connectivity, cut-through switching, 802.11ax in terms that are geared towards them.

Here are some pointers on where to start:

You're not replacing something because it was bad.
- A pretty easy pitfall for IT professionals - if you devalue "what came before" you devalue the role a replacement fills. It may be hard to do, but most things here were built for a reason - the intent behind the design is important for other reasons, but this negativity will affect anything you do after that.
Show how they can use it
- This might not make a lot of sense at the outset, but any trivial method for interaction will make a particular change feel more concrete. Some examples:
  - Add a Looking Glass view if it's a new network. Providing users a way to "peek inside" is a time-honored tradition with many industries.
  - Open some iPerf/Spirent servers for users to interact with, or other benchmarking
  - Functional demos like blocking internetbadguys.com
Share how it is made
- You never know, why not try?

Ego

This one's a bit harder - and I'm not trying to apply major negative connotations here. As engineers, we get pretty attached to our decisions, attributing significant personal effort to the products we purchase.

As an industry, IT professionals really need to re-align here. We consider vendor relationships allegiances and fundamentally attribute our own personal integrity. If I had my way, I'd stop hearing that someone's a "Cisco" or a "VMware" guy - we need to shift this focus back to consumers.

The biggest point for improvement here is also on the negativity front. Let's start by shifting from "this solution is bad" (devaluing your own work for no reason) to "This solution doesn't fit our needs, and this is why." The latter helps improve future results by getting the ball rolling on what criteria consumers value more.

After deploying quite a few solutions "cradle-to-grave," my personal approach here is to think of them like old cars, computers, stuff like that. I fondly remember riding around in my parents' 80's suburban, but we replaced it because it wasn't reliable enough for the weather we had to face in Rural Alaska, and it was too big.

Here are some examples of how I regard these older, later replaced solutions/products:

Cisco 6500s: Fantastically reliable, fantastic power bills, fantastic complexity to administer
Aruba 1xx series Access Points: Revolutionary access control, less than stellar radio performance
Palo Alto 2000/4000 series firewalls: Again, revolutionary approaches to network security, but not enough performance for modern businesses to function. Commit times improved greatly on later generations
TM-OS 11.x: Incredible documentation, incredible feature depth. If it's more modern than 2015, you're going to want more features

All of these served businesses well, then needed to be replaced. I see too many engineers beat themselves up when services eventually fell apart, and it's just not necessary.

9/10 NGINX Use Cases, URI and Host rewrites

Sun, 17 Jan 2021 10:29:00 -0900

NGINX Rewrite Directives, The 9/10 Solutions

When doing ADC/Load Balancer work, nearly all requests fit into two categories:

Please rewrite part of the URL/URI
Please change the host header for this reverse proxy

These are fairly simple to implement in NGINX, so I'm creating a couple of cheat-sheet code snippets here.

"Strip Part of the URL Out"

URI stripping is fairly common, and the primary motivation for this blog post. As enterprises move to Kubernetes, they're more likely to use proxy_pass directives (among other things) to multi-plex multiple discrete services into one endpoint.

With URI stripping, an engineer can set an arbitrary URI prefix and then remove it before the web application becomes aware. URI stripping is a useful function to stitch multiple web services together into one coherent endpoint for customer use.

NGINX comes to the rescue here, with a relatively simple solution:

location directive: Anchors the micro- or sub- service to an NGINX URI
rewrite directive: Rewrites the micro- or sub- service to a new directory, allowing for minimal backend modifications

The below example achieves this by rewriting the URI /build* to /, ensuring that the build service (Jenkins) doesn't need to be re-tooled to work behind a proxy:

1  location /builds/ {  
2    root /var/lib/jenkins/workspace/;  
3    rewrite ^/builds(.\*)$ $1 break;  
4    autoindex on;  
5  }

As you can see, this example is an obvious security risk, as the autoindex directive lets clients browse through the build service without authentication and potentially access secrets, and is intended as an illustration and not a direct recommendation for production practice. Here's a little bit more production-appropriate example providing Jenkins over TLS (source: https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-nginx/)

 1    server {  
 2        listen       443 ssl http2 default\_server;  
 3        listen       \[::\]:443 ssl http2 default\_server;  
 4        server\_name  cicd.lab.engyak.net;  
 5  
 6        ssl\_certificate "CERT;  
 7        ssl\_certificate\_key "KEY";  
 8        ssl\_session\_cache shared:SSL:1m;  
 9        ssl\_session\_timeout  10m;  
10        ssl\_protocols TLSv1.2 TLSv1.3;  
11        ssl\_ciphers ALL:!AES:!RC4:!SHA:!MD5;  
12        ssl\_prefer\_server\_ciphers on;  
13  
14        # Load configuration files for the default server block.  
15        include /etc/nginx/default.d/\*.conf;  
16  
17        location ~ "^/static/\[0\-9a-fA-F\]{8}\\/(.\*)$" {  
18            #rewrite all static files into requests to the root  
19            #E.g /static/12345678/css/something.css will become /css/something.css  
20            rewrite "^/static/\[0\-9a-fA-F\]{8}\\/(.\*)" /$1 last;  
21        }  
22  
23        location /userContent {  
24            \# have nginx handle all the static requests to userContent folder  
25            #note : This is the $JENKINS\_HOME dir  
26            root /var/lib/jenkins/;  
27            if (!-f $request\_filename){  
28            #this file does not exist, might be a directory or a /\*\*view\*\* url  
29            rewrite (.\*) /$1 last;  
30            break;  
31            }  
32            sendfile on;  
33        }  
34        location / {  
35                    sendfile off;  
36                    proxy\_pass http://jenkins/;  
37            \# Required for Jenkins websocket agents  
38            proxy\_set\_header   Connection        $connection\_upgrade;  
39            proxy\_set\_header   Upgrade           $http\_upgrade;  
40  
41            proxy\_set\_header   Host              $host;  
42            proxy\_set\_header   X-Real-IP         $remote\_addr;  
43            proxy\_set\_header   X-Forwarded-For   $proxy\_add\_x\_forwarded\_for;  
44            proxy\_set\_header   X-Forwarded-Proto $scheme;  
45            proxy\_max\_temp\_file\_size 0;  
46  
47            #this is the maximum upload size  
48            client\_max\_body\_size       10m;  
49            client\_body\_buffer\_size    128k;  
50  
51            proxy\_connect\_timeout      90;  
52            proxy\_send\_timeout         90;  
53            proxy\_read\_timeout         90;  
54            proxy\_buffering            off;  
55            proxy\_request\_buffering    off; \# Required for HTTP CLI commands  
56            proxy\_set\_header Connection ""; \# Clear for keepalive  
57        }  
58        error\_page 404 /404.html;  
59            location = /40x.html {  
60        }  
61  
62        error\_page 500 502 503 504 /50x.html;  
63            location = /50x.html {  
64        }  
65    }

Set Host Headers

This is quite a bit easier, using the proxy_set_header directive:

1  location /builds/ {  
2    proxy\_pass http://localhost:8080;  
3    proxy\_set\_header Host cicd.lab.engyak.net  
4    rewrite ^/fabric-builds(.\*)$ $1 break;  
5  }

NSX-T Transitive Networking

Sun, 03 Jan 2021 14:41:00 -0900

One major advantage to NSX-T is that Edge Transport Nodes (ETNs) are transitive

Transitivity (Wikipedia) (Consortium GARR) is an extremely important concept in network science, and in computer networking.

In simple terms, a network node (any speaker capable of transmitting or receiving on a network) can have the following transitivity patterns:

Transitive: Most network equipment fit in this category. The primary purpose of these devices is to allow traffic to flow through them and to occasionally offer services over-the-top.
- Examples:
  - Switches
  - Routers
  - Firewalls
  - Load Balancers
  - Service Meshes
  - Any Linux host with ip_forward set
  - Mobile devices with tethering
Non-Transitive: Most servers, client devices fit in this category. These nodes are typically either offering services over a network or consuming them (Usually both). In nearly all cases, this is a deliberate choice by the system designer for loop prevention purposes.
- Note: It's completely possible to participate in a routing protocol while being non-transitive.
- Examples:
  - VMware vSphere Standard Switch && vSphere Distributed Switch (no Spanning-Tree participation)
  - Amazon vPC
  - Azure VNet
  - Any Linux host with ip_forward disabled
  - Nearly any server, workstation, mobile device
Anti-Transitive: This is a bit of a special use case, where traffic is transitive but only in specific use cases. Anti-Transitive network nodes have some form of control in place to prevent transit in specific scenarios but allowing it in others. The most common scenario is when an enterprise has multiple service providers - where the enterprise doesn't want to pay for traffic going between those two carriers.
- Examples:
  - Amazon Transit Gateway
  - Any BGP Router with import/export filters

vSphere Switch Transitive Networking Design

To fully understand VMware's approach, it is important to first understand earlier approaches to network virtualization. vSphere switches are a bit of a misnomer, as you don't actually switch at any given point. Instead, vSphere switches leverage a "Layer 2 Proxy" of sorts, where NIC-accelerated software replaces ASIC flow-based transitive switching.

This approach offers incredible flexibility, but is theoretically slower than software switching; to preserve this capability VMware noticed early on that loop prevention would become an issue. Pre-empting this problem, making the platform completely non-transitive to ensure that this flexibility will be more readily adopted.

Note: VMware's design choices here contained the direct intent to simplify the execution and management of virtualized networking. This choice made computer networking simple enough for most typical VI administrators to perform, but more of the advanced features (QoS, teaming configurations) require more direct involvement from network engineers to execute well. Generally speaking, the lack of need for direct networking intervention for a VSS/vDS to work has led to a negative trend with the VI administrator community. Co-operation between VI administration and networking teams often suffer due to this lack of synchronization, and with it systems performance as well.

NSX-T Transitive Networking Design

NSX-T is highly prescriptive in terms of topology. VMware has known for years that a highly controlled design for transitive networking will provide stability to the networks it may participate in - just look at the maturity/popularity of vDS vs Nexus 1000v.

NSX-T does depend on VDS for Layer 2 forwarding (as we've established, not really switching), but does follow the same general principles for design.

To be stable, you have to sacrifice flexibility. This is for your own protection. These choices are artificial design limitations, intentionally placed for easy network virtualization deployment.

VMware NSX-T Tier-0 logical routers have to be transitive to perform their main goal, transporting overlay traffic to underlay network nodes. Every time a network node becomes transitive in this way, specific design decisions must be made to ensure that anti-transitive measures are appropriately used to achieve network stability.

NSX-T Tier-1 Distributed routers are completely nontransitive, and NSX-T Tier-1 Service Routers have severely limited transitive capabilities. I have diagrammed this interaction as non-transitive because the Tier-1 services provided are technically owned by that logical router.

Applications for Transitive Tier-0 Routers

Given how tightly controlled transit is with NSX-T, the only place we can perform these tasks is via the Tier-0 Logical Router. Let's see if it'll let us transit networks originated from a foreign device, shall we?

Hypothesis

NSX-T Tier-0 Logical Routers are capable as transit providers, and the only constructs preventing transit are open standards (BGP import/export filters)

Unit Test

Peer with vCLOS network via (transiting) NSX-T Tier-0 Logical Router:

Let's build it, starting with the vn-segments:

Then, configuring Tier-0 External Interfaces:

Ensure that we're re-distributing External Interface Subnets:

Ensure that the additional prefixes are being advertised. Note: This is a pretty big gripe of mine with the NSX GUI - we really ought to be able to drill down further here...

Configure BGP Peering to the VyOS vCLOS Network:

We're good to go on the NSX Side. In theory, this should provide a transitive peering, as BGP learned routes are not Re-Distributed but learned.

(The other side is VyOS, configured in the pipeline method outlined in a previous post. This pipeline delivery method is really growing on me)

We can verify that prefixes are propagating transitively via the NSX-T Tier-0 in both protocol stacks by checking in on the spines that previously had no default route:

 1vyos@vyos-s1.engyak.net:~$ show ip route  
 2Codes: K - kernel route, C - connected, S - static, R - RIP,  
 3       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,  
 4       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,  
 5       F - PBR, f - OpenFabric,  
 6       > - selected route, * - FIB route, q - queued, r - rejected, b - backup  
 7  
 8B>* 0.0.0.0/0 [20/0] via 10.6.194.1, eth1, weight 1, 00:15:20  
 9B>* 10.0.0.0/8 [20/0] via 10.6.194.1, eth1, weight 1, 00:15:20  
10vyos@vyos-s1.engyak.net:~$ show ipv6 route  
11Codes: K - kernel route, C - connected, S - static, R - RIpng,  
12       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,  
13       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,  
14       f - OpenFabric,  
15       > - selected route, * - FIB route, q - queued, r - rejected, b - backup  
16  
17B>* ::/0 [20/0] via fe80::250:56ff:febc:b05, eth1, weight 1, 00:15:25

Now, to test whether or not packets actually forward:

1vyos@vyos-s0.engyak.net:~$ ping 1.1.1.1  
2PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.  
364 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=49.7 ms  
464 bytes from 1.1.1.1: icmp_seq=2 ttl=53 time=48.10 ms  
564 bytes from 1.1.1.1: icmp_seq=3 ttl=53 time=45.9 ms  
664 bytes from 1.1.1.1: icmp_seq=4 ttl=53 time=45.0 ms

Looks like Tier-0 Logical Routers are transitive! This can have a lot of future implications - because NSX-T can become a launchpad for all sorts of virtualized networking. Some easy examples:

Tier-0 Aggregation: Like with aggregation-access topologies within the data center and campus, this is a way to manage BGP peer/linkage count at scale, allowing for thousands of Tier-0 Logical Routers per fabric switch.
Load Balancers: This shifts the peering relationship for load balancers/ADC platforms from a direct physical peering downward, making those workloads portable (if virtualized)
Firewalls: This provides Cloud Service Providers (CSP) the ability to provide customers a completely virtual, completely customer-owned private network, and the ability to share common services like internet connectivity.
NFVi: There are plenty of features that can leverage this flexibly in the NFV realm, as any given Enterprise VNF and Service Provider VNF can run BGP. Imagine running a Wireless LAN Controller and injecting a customer's WLAN prefixes into their MPLS cloud - or even better, their cellular clients.

Why Automate? Using Pipelines to Develop and Manage Network Configurations

Thu, 31 Dec 2020 22:26:00 -0900

Continuous Delivery: No Rest for the Wicked

Now that we have:

A method to generate Desired State Configurations, by defining Declaratively what the device config should be, and combining it with what a device config should have
A method to apply configurations automatically, without PuTTY Copy-Pasting

We now can achieve Infrastructure As Code, where we can take a few artifacts from source control and turn them into a live, viable network device.

This is handy, but what about maintaining it? CI/CD Pipelines

In simplest terms, CI/CD tools provide an automated way to "do a thing" to make it pretty easy to perform repetitive tasks. For this example, I'll be using Jenkins CI, but the steps we'll be performing are pretty simple.

Pipelines aren't the only things that a CI tool can do, but there are some pretty big differences between a traditional pipeline and managing a network - for example, there's no code to compile. Instead, it's best to map out the steps that we want a CI tool to perform. Jenkins has a project type - Freestyle that lends itself well to applications like this, but it can also get fairly messy/disorganized.

A more comprehensive definition of a pipeline (from Red Hat) is here: https://redhat.com/en/topics/devops/what-cicd-pipeline

Installing Tools

In this case, I am leveraging a purpose-build CentOS host with Ansible, Jenkins, Jinja, and Python3 installed. Since this prerequisite list is fairly short, it should lend itself rather well to containerization.

Network infrastructure tends to have inbound access restrictions that most container platforms cannot meet in an auditable, secure method. This capability can be provided with VMware NSX-T or with Project Calico, but these capabilities are pretty advanced. I'd consider containerization an option for those willing to take it on in this case, and am keeping this guide as agnostic as possible.

Perhaps later I'll build on this and provide a dockerfile. Starring the repository will probably be the best way to keep track!

Executing Continuous Integration / Continuous Delivery

Let's start with the specifications for what we want to do. This doesn't need to be excessively convoluted.

The CI Tool should simply execute code, minimally. If we resort to a ton of shell scripting here, it won't be managed by source control and cannot easily be updated.
The CI Tool is responsible for:
- Execution of written code
- Logging
- Notification
- Testing of written code
- Scoring of results to assess code viability / production readiness

Steps:

Fetch code from GitHub. Execute every five minutes, if a new code commit is available.
Lint (syntax validate) all code.
Compile Network Configurations, and apply to network infrastructure
Test
Notify of build success

I have added an example CI Project file to this repository. https://github.com/ngschmidt/vyos-vclos It does not contain testing or validating steps yet, as those are considerably more complex - writing a parsable logger will take quite a bit more time than I feel an individual post is worth.

The CI Project

We're not asking much of Jenkins CI in this case, so you can easily replicate this configuration by:

Setting a Git repository to clone from (Under Source Code Management)
Setting the Build Trigger to Poll SCM (H/5 * * * *)
Execute the playbooks (provided in the GitHub repository). Instead of executing each individual pipeline, I elected to make a main.yml playbook that contains all steps, so that the control aspects of this remain centralized in the Git repository.
Automated Evaluation: I provided a yamllint example, eventually this should be tallying the results of each automated test and scoring it.

People want new stuff, and some of it might be new networking Features

Now that we have an easy way of keeping all of our networking gear (2-N nodes) managed and in baseline with the same level of effort, it's pretty straightforward to automatically roll out Features.

Features in this case don't need to be a large, earth-shaking new capability in more traditional software development parlance. Instead, let's consider a Feature something smaller:

A Feature should be something a consumer wants (DevOps term would be to delight users)
A Feature should be a notable change to an information system
A Feature should be maintainable or maintainability. A system's infrastructure administrator/engineer/architect is a consumer as well, and that person's needs have value, too!

Some Examples of Network Features:

Wireless AP-to-AP Roaming: Users like having connectivity stay as they move about. This can vary from 802.11i in Personal Mode, to 802.11i in Enterprise mode with 802.11k/r/v implemented to be truly seamless.
- If this were a CI Project:
  - Minimum Viable Product would be defined. If the security teams are okay with WPA2-PSK, then that would be it. If not, the roaming capability would be at ~6 seconds, with lots of room for improvement.
  - Roll out 802.11k reports for better AP association decisions
  - Roll out 802.11v for better notifications around Power Saving
  - Roll out 802.11r or OKC for secure hand-off
  - No Rest for the Wicked: Do it all again with WPA3!
VPN Capability
- If this were a CI Project:
  - MVP: IPSec-based VPN with RADIUS authentication
  - TLS Fallback for low-MTU networks or PMTUD
  - Improved authentication mechanisms, like PKI or SAML
  - Client Posture Assessment

In the world of continuous delivery, these can be done out of order, or to a roadmap. When you're done with a capability, deliver it instead of waiting for the next major code drop.

I'm a network guy, what's a code drop?

Honestly, infrastructure teams never really followed more traditional software development approaches - Continuous Delivery is a better fit, because of our key problems:

Change
Loops caused by changes

There's no true hand-off from development to operations, just the people who run the network, and those who don't. We are afflicted by an industry of either change fear or CAB purgatory where once something is built, it can no longer be improved. This builds up a lot of indebtedness that is rarely fixed by anything short of a forklift. Ideally, we can leverage CI tools in this way:

Clean Slate: Delete all workspace files
Write Feature Code
Build configurations
Apply configurations to test nodes
Validate (manually or automatically, or both) that the change did what it was supposed to, and that it worked
If it fails, go back to step #1
Stage Feature release, do paperwork, etc.
Release Feature to all applicable managed nodes
Work on the next Feature

I have attached a Jenkins Project that performs most of these tasks here. There are some caveats to this method that I'll cover below.

This should result in much higher quality work being released, and in the networking world, reliability is king. This is the key to becoming free of CAB Purgatory in large organizations.

A Day in the life of a Feature

Since the majority of the muscle work with Jenkins has already been programmed, we simply need to focus on the source code (device configuration), and work from there:

Create a new git branch. This can be achieved with git checkout or via your SCM GUI.
Write code for the git branch. Ideally, you'd create a new project for this step against that specific branch, but there is no "production environment" to speak of in my home lab.
Commit code. Again, small steps are still the best approach. The biggest change here is to periodically check in on your pipeline to see if anything breaks. This gives you the "fix or backpedal" opportunity at all times, and makes it easy to spot any breakage.
Submit a git pull request: This is an opportunity for the team to review your results, so be sure to include some form of linkage to your CI testing/execution data to better make your case.
Merge code. This will automatically roll to production at the next available window, and is your release lever.

Example 1: Fix an issue where BGP NLRIs are not being imported due to no policy

Pull Request #1

For this, we ran into a particularly odd behavior change - VyOS was somewhat recently rebased from Quagga to FRR, which picked up the following behavior: http://docs.frrouting.org/en/latest/bgp.html#require-policy-on-ebgp

 1Require policy on EBGP  
 2[no] bgp ebgp-requires-policy  
 3This command requires incoming and outgoing filters to be applied for eBGP sessions. Without the incoming filter, no routes will be accepted. Without the outgoing filter, no routes will be announced.  
 4  
 5This is enabled by default.  
 6  
 7When the incoming or outgoing filter is missing you will see “(Policy)” sign under show bgp summary:  
 8  
 9exit1# show bgp summary  
10  
11IPv4 Unicast Summary:  
12BGP router identifier 10.10.10.1, local AS number 65001 vrf-id 0  
13BGP table version 4  
14RIB entries 7, using 1344 bytes of memory  
15Peers 2, using 43 KiB of memory  
16  
17Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt  
18192.168.0.2     4      65002         8        10        0    0    0 00:03:09            5 (Policy)  
19fe80:1::2222    4      65002         9        11        0    0    0 00:03:09     (Policy) (Policy)

This was preventing BGP route propagation, and was a result of an upstream change. In Software Development, this is called a "breaking change" because it implements major functional changes that will have potentially negative effects unless action is taken.

To mitigate this, we can develop a solution iteratively, using our lab environment, and test, re-test, and then test again until we get the desired result. 24 Commits later, I'm satisfied with the result. Once a solution is sound (passes automated testing) it is best practice to submit a solution for peer review. Git calls this action a pull request. Here's the one for this change:

Example 2: Roll out IPv6 Dynamic Routing

Pull Request #2

Like with the previous pull request, this particular implementation isn't huge.

By code volume, this was about 200 lines, but the real difference here is in the multiplier. From those 284 lines:

100 lines DRY (Don't Repeat Yourself) highly repetitive code (template)
57 are documentation
38 lines DRY highly repetitive code (variables)

The history of this pull request is publicly available. I made a few mistakes, and then caught them with automated testing, as everyone can see.

About two-thirds of the way through this I realized I was rolling out IPv6 will a pull request. Neat.

Conclusions

This generates quite a bit of code, repeatably and reliably.

Value in Volume

All in all, we're generating 1,020 lines of configuration with 833 lines of code. The ratio becomes more favorable for a developer in terms of sheer work more homogenous your environment or custom configurations are. If you're only evaluating saved time:

2 Devices may feel dubious
3 Devices will show real value in saved time
4+ the benefits become insane

Value in Consistency

The real value here is consistent configurations. Using traditional methods I'd normally have a ton of frustration trying to configure things consistently, un-doing and re-doing copy-paste errors, and re-testing. If you configure both sides with Jinja2, they'll match exactly and peer up, every time

Value in Documentation

This is the part where I truly value this approach. If an engineer or architect designs variable definitions well, the end result summarily defines the device. This can be attached in-line or as meta-data to a diagram, or easily verified against a diagram to ensure things are consistent. The few issues I had were quickly resolvable by comparing YAML to a diagram. I'm probably going to use this method to generate diagrams as well.

Downsides

I trivialized the network driver aspect of this work. The one I chose, vyos.vyos.vyos_config, is not idempotent and was causing serious issues as a result (BGP neighbors dropping constantly as I re-applied the configuration). Off-the-shelf network drivers are perfectly well suited for prototyping, but substantial development is required to use them in production. This would take a full team to become reality, but a middle ground is readily achievable.

We call it Continuous Delivery for a reason.
Automate when it helps you.

We can use this guidance to come up with a plan, for example:

Milestone 1: Jinja2-fy your golden configurations, and stop manually generating them
Milestone 2: You have the config a device should have, gather_facts the current configuration, and generate a report to see if it's not compliant.
Milestone 3: Topical automation replaces manual remediation
Milestone 4: Fully mature NETCONF springs forth and saves the day!

People who are at #4 aren't better than people who have finished #1. Use what's useful.

Why Automate? Ansible Playbooks and Desired State for Network Operating Systems

Tue, 29 Dec 2020 12:37:00 -0900

Don't Reinvent the Wheel: Ansible Playbooks

Writing your own code isn't always the answer

Often, communities such as Python will contribute code of substantially higher quality than what you/I can create individually.

This is OK. In nearly every case, dyed-in-the-wool traditionalist programmers will consume "libraries" in their language of choice - it's only an outsider perspective that developers create everything they use.

In modern engineering, a true engineer or architect will often apply practices they studied in college to real-world situations instead of trying to create their own solutions. This doesn't discount creativity, nor does it discount those who are more pragmatically oriented. Without creativity, we have no way to improve engineering practice, and without pragmatism, we have seen some pretty serious loss of life: https://interestingengineering.com/23-engineering-disasters-of-all-time

...but you still have a lot of work to do

Adapting engineering practices, code from the internet, Googled Cisco example topologies as a matter of practice does take work. Do you trust all code from Stack Overflow? Cisco-answers.net (not a real website)?

You shouldn't, and modern engineering practice doesn't either. In nearly every case, the ability to apply engineering practice to a problem comes with years of training, millennia of past examples (failures and successes) as history for individual practice, ideally with similar applications. A good example of this is the study of brittle fractures where manipulating (maximizing) material hardness is no longer an automatic victory, but more of a serious safety risk.

We live in a simpler world of abstraction and pure mathematics, and behaviors are a lot more reliable - but they're not perfectly so. We as designers and implementers of computer solutions (Network, Systems, don't care) can learn from our more disciplined cousins. I'll write more on this later, but for now, let's simply at least agree to review every action critically.

Playbook Automation

Let's use the lens of an engineer evaluating a technical control here. Ansible is going to be my example here, as it's probably the most straightforward.

Supporting Files

While it is possible to run a standalone, self-supporting playbook, it's not generally recommended at scale. The first step towards leveraging this automation is by defining an inventory. As always, this is typically in YAML, so most of the effort goes into structuring your data as opposed to actual work.

Some recommendations:

Don't let names collide between production, lab, etc. We don't want to have a Wargames scenario in anybody's production network.
Make sure it makes sense. It's pretty easy to over/under-organize; think about the smallest elemental unit you may work on.
Leverage Source Control! Save a copy, keep your revision history. Even better, get peer reviews.
Remember, this can be edited later! This should continually improve.

Example (loosely based from https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html)

I'm using the project (virtualized Clos Topologies) as a prefix, and then organizing device types from there. Spines don't need VLANs, and will be route reflectors - which is enough to justify separation in this case.

Inventory

 1vyos_vclos_leafs:  
 2  hosts:  
 3    vclos_l0.engyak.net:  
 4      ansible_host: "1.1.1.1"  
 5    vclos_l1.engyak.net:  
 6      ansible_host: "1.1.1.2"  
 7  vars:  
 8    ansible_network_os: vyos.vyos.vyos  
 9    ansible_user: vyos  
10    ansible_connection: ansible.netcommon.network_cli  
11vyos_vclos_spines:  
12  hosts:  
13    vclos_s0.engyak.net:  
14      ansible_host: "1.1.1.3"  
15    vclos_s1.engyak.net:  
16      ansible_host: "1.1.1.4"  
17  vars:  
18    ansible_network_os: vyos.vyos.vyos  
19    ansible_user: vyos  
20    ansible_connection: ansible.netcommon.network_cli

Let's explain what I've done here. There are a few deviations from the typical. I'll try to explain them here:

YAML Inventory: This is just me, I prefer it over the INI format as a Linux guy. It also helps a lot with structured hierarchies, which I like as a network guy.
Variable declarations:
- Per Ansible's documentation on networking, we do know that there are a few things unique to network automation - namely the lack of on-board python. This means that the Ansible control node (the one EXECUTING the playbook) needs to know that it's doing all of the planning/thinking. For this to work, we need to make a few unique (but re-usable) declarations
  - ansible_network_os: More or less does exactly what it says. There's a built-in ansible interpreter for VyOS - but this is really only true for a handful of network distros. You can get more from Ansible Galaxy, but extensive testing should be applied.
    - ansible_connection: This is basically the "driver" for the CLI. You can use Paramiko or SSH as well. this is primarily governed by your Network OS.
    - ansible_user just instructs the control node on what username to attempt against the target host.

Outside of this, I have also set up SSH key authentication to all VyOS nodes. It's pretty easy: (VyOS Documentation)

1set system login user vyos authentication public-keys key1 key blahblahblah  
2set system login user vyos authentication public-keys key1 type ssh-rsa

The Playbooks

Values

Before designing a playbook, we do need to cover some of Ansible's key design values:

Idempotency: Run once, get the same result every time. If a change already has been made and is invasive, don't repeat it unless the state doesn't match.
Thin Veil of Abstraction: You should be aware of what is being implemented from a technical perspective, but not have to control every last aspect of it.
Be Declarative: Try to design from the abstract concept you want to implement, and fill in the technical details as needed, not the other way around.

Day 0, get the system online

In this example, we want to have four devices have some level of usable configuration, and we don't want to do lots of manual, error-prone editing to get there. We're going to adapt my base configuration for this purpose by re-tooling it to support Jinja deployments. At a high level, Jinja playbooks:

Load Variables: This will be a separate file, effectively designing the what of your deployment
Load Template, then translate variables: This will be executed by the template module

We'll keep this example pretty short - it's available in the linked repository, but we also want to leverage idemopotency for future changes. It doesn't leverage inventory, because it's creating base configurations to be applied by some other method.

Fun fact - this is the first stage to any Infrastructure-as-Code implementation. The created end results (*-compiled.conf) can be directly applied, or by using a "Day 2 Method".

Variables:

1---  
2global:  
3 hostname: 'vyos-router.engyak.net'  
4 domain: 'engyak.net'  
5 timezone: 'US/Alaska'

Execution (Playbook):

 1---  
 2- hosts: localhost  
 3 tasks:  
 4 - name: Import Vars...  
 5 include_vars:  
 6 file: vyos-base.yml  
 7 - name: Combine vyos...  
 8 template:   
 9 src: templates/vyos-base.j2  
10 dest: vyos-compiled.conf

Day 2, apply routine changes

In this example, we've already started the deployment, and have it up and running. We have some form of routine change to make, but we want it to be consistently applied, and idempotently. This will mean that the configuration change playbook shouldn't contain anything about the specific change in an ideal world with this method.

 1- hosts: vclos_l0.engyak.net  
 2  tasks:  
 3    - name: Apply on L0!  
 4      vyos.vyos.vyos_config:  
 5        src: 'vyos-l0-compiled.conf'  
 6        save: yes  
 7- hosts: vclos_l1.engyak.net  
 8  tasks:  
 9    - name: Apply on L1!  
10      vyos.vyos.vyos_config:  
11        src: 'vyos-l1-compiled.conf'  
12        save: yes  
13- hosts: vclos_s0.engyak.net  
14  tasks:  
15    - name: Apply on S0!  
16      vyos.vyos.vyos_config:  
17        src: 'vyos-s0-compiled.conf'  
18        save: yes  
19- hosts: vclos_s1.engyak.net  
20  tasks:  
21    - name: Apply on S1!  
22      vyos.vyos.vyos_config:  
23        src: 'vyos-s1-compiled.conf'  
24        save: yes

This will re-apply any changes that are staged via the base configuration and Jinja merge repeatedly if re-executed.

Note: This particular network driver is not idempotent. In production networks something like NAPALM/Nornir may be more appropriate. You can verify if a method is idempotent by repeatedly running the playbook - an expected result is changed=0.

 118:55:40 PLAY [vclos_l0.engyak.net] *****************************************************  
 218:55:40   
 318:55:40 TASK [Gathering Facts] *********************************************************  
 418:55:41 [WARNING]: Ignoring timeout(20) for vyos.vyos.vyos_facts  
 518:55:44 ok: [vclos_l0.engyak.net]  
 618:55:44   
 718:55:44 TASK [Apply on L0!] ************************************************************  
 818:55:49 changed: [vclos_l0.engyak.net]  
 918:55:49   
1018:55:49 PLAY [vclos_l1.engyak.net] *****************************************************  
1118:55:49   
1218:55:49 TASK [Gathering Facts] *********************************************************  
1318:55:49 [WARNING]: Ignoring timeout(20) for vyos.vyos.vyos_facts  
1418:55:53 ok: [vclos_l1.engyak.net]  
1518:55:53   
1618:55:53 TASK [Apply on L1!] ************************************************************  
1718:55:57 changed: [vclos_l1.engyak.net]  
1818:55:57   
1918:55:57 PLAY [vclos_s0.engyak.net] *****************************************************  
2018:55:57   
2118:55:57 TASK [Gathering Facts] *********************************************************  
2218:55:58 [WARNING]: Ignoring timeout(20) for vyos.vyos.vyos_facts  
2318:56:02 ok: [vclos_s0.engyak.net]  
2418:56:02   
2518:56:02 TASK [Apply on S0!] ************************************************************  
2618:56:06 changed: [vclos_s0.engyak.net]  
2718:56:06   
2818:56:06 PLAY [vclos_s1.engyak.net] *****************************************************  
2918:56:06   
3018:56:06 TASK [Gathering Facts] *********************************************************  
3118:56:06 [WARNING]: Ignoring timeout(20) for vyos.vyos.vyos_facts  
3218:56:10 ok: [vclos_s1.engyak.net]  
3318:56:10   
3418:56:10 TASK [Apply on S1!] ************************************************************  
3518:56:14 changed: [vclos_s1.engyak.net]  
3618:56:14   
3718:56:14 PLAY RECAP *********************************************************************  
3818:56:14 localhost                  : ok=12   changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0     
3918:56:14 vclos_l0.engyak.net        : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0     
4018:56:14 vclos_l1.engyak.net        : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0     
4118:56:14 vclos_s0.engyak.net        : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0     
4218:56:14 vclos_s1.engyak.net        : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The next step is important - automatically updating a network based on configuration changes! As always, my source code for executing this is here. Note that this is a moving project and will get updates with future posts.

Advantages for BGP in Virtualized Topologies

Mon, 21 Dec 2020 20:14:00 -0900

BGP Advantages in Virtualized Networking

BGP is more like an Application

BGP, by design, is a lot more capable than most typical routing protocols. Here are a few ways MP-BGP/BGPv4 is fundamentally unique:

Extensible: MP-BGP can store and run multiple address families (AFIs) and subsequent address families (SAFIs). When doing virtual networking, it's likely that we'll pick up a lot more AFIs for potential use cases in the future. Here's some that I'm aware of / might use in the future:
- AFI: IPv4
  - SAFI: Unicast
  - SAFI: Multicast
  - SAFI: Flow Spec (Firewalling)
- AFI: IPv6
  - SAFI: Unicast
  - SAFI: Multicast
  - SAFI: Flow Spec (Firewalling)
- AFI: L2VPN
  - SAFI: EVPN
- AFI: Link State (IGP Domain Traffic Engineering)
- https://www.iana.org/assignments/address-family-numbers/address-family-numbers.xhtml
Universal: Nearly all networking equipment can support BGP
Stable: No flooding, no large-scale recalculations with ephemeral routing entries. This is especially important if you're scaling network segments on a regular basis
It's designed for interlinking Autonomous Systems.
- Most Routing protocols do not support import/export filtering methods - it breaks loop prevention to use them. BGP is designed to use them right out of the gate.
- I'll repeat this one again - with OSPF/IS-IS/EIGRP, you have to either run 2 IGP instances and configure redistribution between them, or trust any prefix a virtual instance (NSX-V, NSX-T, Vyatta, etc) sends to your physical network!*
Portable: Since BGP runs on TCP (Unicast IPv4/IPv6) it'll run over just about anything, and get rid of static routes in a wide variety of use cases. This includes:
- IPSec Tunnels. You can keep the packet overhead GRE would normally consume. This is also the de-facto approach for interconnecting Public Clouds:
  - Amazon AWS: Direct BGP-over-IPSec is supported on their VPN appliance and on Transit Gateway (TGW) https://docs.aws.amazon.com/vpn/latest/s2svpn/your-cgw.html#CGRequirements
  - VMware VMC on AWS: I may be biased, but this follows a very similar method to TGW but is considerably easier to set up. Dead simple, up and running in minutes. https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws.networking-security/GUID-5AF45CE6-FA53-45C0-83E5-25F8E3A055E9.html
  - Azure VNet: Indirect BGP-over-IPSec, with Multi-hop set to 2. The trick here is to create a loopback on each side and set static routes, then to fire up peer relationship at that. This is pretty nerdy and overly elegant in my opinion, but it runs fairly well. https://docs.microsoft.com/en-us/azure/vpn-gateway/tutorial-site-to-site-portal
  - Site-to-Site VPNs between security appliances: 10/10 would recommend. Nearly every appliance vendor can do it instead of policy VPN nowadays - I do have a snippet https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws.networking-security/GUID-5AF45CE6-FA53-45C0-83E5-25F8E3A055E9.html
- MPLS: Pretty much any major enterprise that uses MPLS already does this. Enabling Layer 2 carrier services introduces a great deal of complexity and opportunities for failure, but even if you're consuming Layer 2 Services...
  - BUM: Broadcast, Unknown-Unicast, Multicast traffic types all introduce inefficient flooding and disproportionately tax those carrier networks you depend on for your business. Most carrier ethernet services do not have an explicit method for handling this type of traffic! (EVPN Excluded) https://en.wikipedia.org/wiki/Broadcast,_unknown-unicast_and_multicast_traffic. Since BGP is completely unicast, it's not as much of an issue here.
- SD-WAN: As probably expected, SD-WAN is highly dynamic, and typically runs better when you can make frequent, incremental changes to pathing (this may be less true in small-scale implementations where there's an SD-WAN appliance as a SPOF)
  - Velocloud: OSPF/BGP https://docs.vmware.com/en/VMware-SD-WAN-by-VeloCloud/3.3/velocloud-admin-guide-33/GUID-7A080D7A-C527-433C-96CA-534D1418A3E0.html
  - Cisco: OSPF/BGP https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/routing/vEdge-20-x/routing-book/m-unicast-routing.html
  - Aryaka: This one's interesting because they support only eBGP or RIPv2 https://www.aryaka.com/docs/smartmanage-ANAP-product-brief.pdf
- SASE: With all SASE offerings, static routing is officially off the table, because you can't automatically move between circuits. Most offerings will leverage BGP-over-IPSec (above included) and the ability to do the same over diverse connectivity. If you're looking at SASE, you want to seriously consider more BGP than people who don't.

Advantages Over Link-State Routing Protocols

I'll keep this short to avoid beating a dead horse too much.

There's no such thing as flooding. Network changes aren't that big of a deal anymore due to this - but in many cases, that's the entire point of dynamic routing.
Security appliances tend to either not support them or weren't designed with OSPF support in mind. Palo Alto's skunkworks division is building a separate route engine that will be BGP-only https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-new-features/networking-features/advanced-route-engine.html
There are ways to limit route flapping in BGP

The following advantages are specific to NSX-T/V or virtualized routers:

Link-state adjacencies don't change if a virtual system is down. If a hypervisor hosting a VM stays up and a VM is down, link-state doesn't change, so you're going to wait for the entire dead interval as an outage.
I'll repeat this again, with virtualized network functions Link-state adjacencies failover at their maximum dead interval, nullifying the primary advantage to these routing protocols!
Interlinks from a physical network must be specifically engineered to prevent non-determinism. If you're multi-homing a virtual router via the same Layer 2 domain, LSAs will not only flow between the physical network endpoints and your desired Virtual Network Function (VNF) but *between physical network devices.
- This can be designed around, but you lose the ability to scale multi-pathed machines easily and automatically.
- You can get the dynamic adjacency capability with the BGP Neighbor Range feature on nearly all datacenter network equipment today.

TL;DR, So What if you don't run BGP currently?

This is where most people get hung up - if a network doesn't currently use BGP, it'll potentially introduce problems by adding a new thing for engineers to maintain, major forklifts to pick up hardware support, and so on.

These are all very valid concerns. I'd recommend that instead of shutting down the argument, try some of these solutions on for size instead:

We Inflate eBGP's complexity because we've been conditioned to

Most of the complicated stuff is iBGP loop prevention or pro-grade tuning. Cisco education mechanisms have failed the community somewhat here and with IS-IS (you can only test on SO MUCH in the CCNA/CCNP!). These advanced capabilities are rarely necessary for most typical enterprise deployments. The typical enterprise BGP deployment responsibilities will consist of:

eBGP to a vendor or provider
Import/Export Filters (route-map, prefix-list)
TTL-Security / Authentication
Timers: Default dead interval is 180, which is pretty high. My rule of thumb is 30/90 for WAN, 4/12 for the datacenter. https://docs.vmware.com/en/VMware-Validated-Design/5.0/com.vmware.vvd.sddc-design.doc/GUID-46A773E1-38F7-4F14-B158-489BEB90F44E.html

This can be either really difficult and complex or really simple depending on needs. If it's not an enterprise-wide deployment of BGP (usually it won't be) just plan it out on paper before implementing - there will be learning experiences, accept that they'll happen, and maximize end results. You can't get this education without getting your hands dirty, so make sure it won't hurt the business / use a lab if you can

If you can't, contain the deployment: Set up a prefix for whatever workload is being used, and redistribute that instead of BGP until you've hit maturity. In many cases, it'll just stay there, and that's OK.

BGP to Security Appliances

This is probably where I'd start - it's got the highest value to effort ratio. Given your vendor choices, it's probably not that complicated and doesn't necessarily need to be redistributed to campus or other internet edge modules. For most enterprise deployments, this is totally cool. If you're me, you'll start getting annoyed by NLRIs not propagating across sites, which brings you to...

Run BGP on top of an IGP

This is actually how most Service Provider networks work! BGP isn't designed to synchronize - it doesn't modify any next-hop addresses for advertised prefixes and needs another routing protocol to do that. There are some applications where you can go all-BGP https://tools.ietf.org/html/rfc7938 but they're usually reserved for hyper-scaler applications or shops that already are very familiar with BGP. Physical network routes can continue to propagate in this scenario just like they always did, and you're using BGP for the virtual ones. The only redistribution required would be a zeroes/default route from your point of origin to keep things nice and intuitive.

Re-Distribution

This is pretty complicated unless you contain the use cases. In the two scenarios above, you're mostly off the hook on this one - at most, you'll be installing a default route.

Workloads that benefit from BGP competency in the enterprise

VMware NSX-T
VMware Cloud on AWS
Avi Networks Load Balancer
Amazon AWS
Project Calico (Kubernetes!)
Vyatta Vyos
F5 LTM
Microsoft Azure
All SD-WAN
All firewalls

Needless to say, if you're a shop that consumes more than vCenter and ESXi, you probably should be dipping your toes in the water. How far is up to you, but it cannot be avoided.

Some things to remember

If it's providing value, you're doing well.
If you don't know something, that's OK. We're in an ever-changing industry.

vCenter Upgrade Error: `Exception Occurred in install precheck phase`

Sun, 20 Dec 2020 21:33:00 -0900

Error presented by VAMI Interface

Caveat

This is definitely bypassing some form of pre-check, please contact VMware support if it's on a production system!

Troubleshooting

VCSA 7.0 has moved the upgrade process logging to a new location - the log itself is now at /storage/log/vmware/applmgmt/update_microservice.log (actual) or /var/log/vmware/applmgmt/update_microservice.log (symlink)

`update_microservice`

This appears to be a rough order of operations with this new update process:

Pre-Checks: First, the upgrade tries to identify the system being upgraded:

 1update_microservice::          precheckEventHandler: 148 -     INFO - Precheck event happens  
 2update_b2b::                      precheck: 709 -    DEBUG - Running update prechecks  
 3update_b2b::               b2bRequirements: 479 -    DEBUG - Running B2B Requirements hook and processing the results  
 4update_b2b::                _runScriptHook: 330 -    DEBUG - Running B2B script with hook CollectRequirementsHook  
 5update_b2b::                _runScriptHook: 339 -    DEBUG - update script output to file /var/log/vmware/applmgmt/upgrade_hook_CollectRequirementsHook  
 6extensions::                _findExtension:  83 -    DEBUG - Found script hook <module 'update_script' from '/storage/core/software-update/updates/7.0.1.00200/scripts/update_script.py'>:CollectRequirementsHook'  
 7update_utils::                     isGateway:  83 -    DEBUG - Not running on a VMC Gateway appliance.  
 8update_utils::                  isB2BUpgrade:  72 -    DEBUG - Bundle will execute upgrade: False  
 9update_script::           collectRequirements: 492 -    DEBUG - Checking verisons  
10update_script::           collectRequirements: 496 -    DEBUG - Source VCSA version = 7.0.1.00100  
11update_script::           collectRequirements: 500 -     INFO - Target VCSA version = 7.0.1.00200  
12update_utils::               getRPMBlacklist: 185 -    DEBUG - vCSA deployment Type: embedded  
13update_b2b::               b2bRequirements: 493 -    DEBUG - Getting packages excluding the ones in blacklist

From there, it picks up the scope for the upgrade, and verifies against common upgrade issues:

 1update_b2b::               b2bRequirements: 528 -    DEBUG - Calculated packages list   
 2update_b2b::                     checkDisk: 423 -    DEBUG - Checking for disk utilization  
 3update_b2b::                     checkDisk: 467 -    DEBUG - CheckDisk completed, returning with selected disk partition /storage/updatemgr  
 4update_b2b::                      precheck: 740 -    DEBUG - Estimating time to install..  
 5update_b2b::                 estimate_time: 679 -    DEBUG - Estimating time required for rpm-update, services start-stop and reboot time if its required  
 6update_b2b::                 estimate_time: 682 -    DEBUG - Calculating RPM installation time  
 7update_b2b::              rpm_install_time: 587 -    DEBUG - Reading all rpms present in rpm-manifest.json  
 8update_b2b::              rpm_install_time: 588 -    DEBUG - Estimating installation time for installed rpms and new rpms  
 9update_b2b::       get_installed_rpms_list: 564 -    DEBUG - Getting the list of installed RPMs along with the time of install  
10update_b2b::       get_installed_rpms_list: 578 -    DEBUG - Completed getting the list of rpms, returning with the list: <class 'list'>  
11update_b2b::              rpm_install_time: 610 -    DEBUG - Installation time estimated successfully, returning with time for installation 23  
12update_b2b::                 estimate_time: 684 -    DEBUG - Calculating time to start and stop services  
13update_b2b::        estimate_time_services: 620 -    DEBUG - Estimating time for services-start and services-stop  
14update_b2b::        estimate_time_services: 640 -    DEBUG - Completed estimating time for starting and stopping services, returning with the required time: 2  
15task_manager::                        update:  80 -    DEBUG - UpdateTask: status=SUCCEEDED, progress=100, message={'id': 'com.vmware.appliance.update.prechecks_task_ok', 'default_message': 'Prechecks completed', 'args': []}

In this case, everything looks good. I'm not really sure why it needs the SSO Administrator password, and there isn't much on-line about this. We're seeing three errors after we hit go time:

 1update_b2b::                   resumeStage:3431 -    DEBUG - 'download' phase is 100% completed. checkAllRpmsArePresent  
 2rpmfunctions::        checkAllRpmsArePresent: 308 -    ERROR - Empty Stage location passed. This cannot be empty.  
 3update_b2b::                   resumeStage:3497 -    ERROR - Exception in resume stage. Exception : {Package discrepency error, Cannot resume!}  
 4task_manager::                        update:  80 -    DEBUG - UpdateTask: status=FAILED, progress=0, message={'id': 'com.vmware.appliance.plain_message', 'default_message': '%s', 'args': ['Package discrepency error, Cannot resume!']}  
 5dbfunctions::                       execute:  81 -    DEBUG - Executing {SELECT CASE WHEN count(*) == 0 THEN 0 ELSE 1 END as status FROM progress WHERE _stagekey = 'patch-state' AND _message = 'Stage successful'}  
 6functions::              get_resume_state: 340 -    DEBUG - Resume needed in Stage phase  
 7update_b2b::           install_with_resume:2477 -    DEBUG - Installing version 7.0.1.00200  
 8update_functions::                  readJsonFile: 224 -    ERROR - Can't read JSON file /storage/core/software-update/stage/stageDir.json [Errno 2] No such file or directory: '/storage/core/software-update/stage/stageDir.json'  
 9task_manager::                        update:  80 -    DEBUG - UpdateTask: status=FAILED, progress=0, message={'id': 'com.vmware.appliance.not_staged', 'default_message': 'The update is not staged', 'args': []}  
10update_b2b::              installPrechecks:2146 -    DEBUG - Exception occurred while checking for discrepancies Update not staged  
11task_manager::                        update:  80 -    DEBUG - UpdateTask: status=RESUMABLE, progress=0, message={'id': 'com.vmware.appliance.plain_message', 'default_message': '%s', 'args': ['Exception occurred in install precheck phase']}

This is pretty odd, because it's indicating a "resumable error" despite the fact that it cannot resume until a file lock is removed. Here are the errors I see:

Empty Stage Location: Unsure what this means, given the context. Odds are the upgrade script cannot find out where to stage RPMs (Red Hat Package Manager).
Package discrepancy error: It could be relating to the above, or it could be a failed checksum. No other logging is generated by the agent to indicate what's wrong.
Can't read JSON file /storage/core/software-update/stage/stageDir.json: This one's more actionable! It looks like there's no directory by this name.

Easter Egg: statsmoitor probably should be statsmonitor

Remediation

Allow the update to resume

VAMI saves the installation state as a file in /etc/applmgmt/appliance/software_update_state.conf:

1{  
2    "state": "INSTALL_FAILED",  
3    "version": "7.0.1.00200",  
4    "latest_query_time": "2020-12-21T00:19:32Z",  
5    "operation_id": "/storage/core/software-update/install_operation"  
6}

VAMI will be stuck in a loop until you remove this file as root:

1rm -rf /etc/applmgmt/appliance/software_update_state.conf

This will not necessarily resolve the issue that caused the failure, however, more work still needs to be done.

Install via ISO

EDIT: The update ISO can be found at: https://my.vmware.com/group/vmware/patch#search

We're going to try a fallback method, attaching the upgrade ISO. The following snippet is from the vSphere UI, modifying vCenter's VM Hardware:

From there, simply click "Check CD-ROM" and it will immediately appear.

This time, we know what directories to search, so I'm going to watch the logs:

1tail -f  /var/log/vmware/applmgmt/update_microservice.log | grep -i err

Attempt via Command-line with ISO

VMware documents the following method to update via the command line https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-8466F019-C57C-4344-9E15-8CFF74A6E4C2.html

Stage Packages

We're going to try and clear the (empty) workspace and try fresh, auto-accepting EULAs:

 1Command> software-packages unstage  
 2Command> software-packages stage --iso --acceptEulas  
 3 [2020-12-20T17:49:54.355] : ISO mounted successfully  
 4 [2020-12-20T17:49:54.355] : UpdateInfo: Using product version 7.0.1.00100 and build 17004997  
 5 [2020-12-20T17:49:55.355] : Target VCSA version = 7.0.1.00200  
 6 [2020-12-20 17:49:55,169] : Running requirements script.....  
 7 [2020-12-20T17:50:12.355] : Evaluating packages to stage...  
 8 [2020-12-20T17:50:12.355] : Verifying staging area  
 9 [2020-12-20T17:50:12.355] : ISO unmounted successfully  
10 [2020-12-20T17:50:12.355] : Staging process completed successfully  
11 [2020-12-20T17:50:12.355] : Answers for following questions have to be provided to install phase:  
12        Question:  
13                ID: vmdir.password  
14                Text: Single Sign-On administrator password  
15                Description: For the first instance of the identity domain, this is the password given to the Administrator account.  Otherwise, this is the password of the Administrator account of the replication partner.  
16                Allowed values:  
17                Default value:  
18  
19 [2020-12-20T17:50:12.355] : Execute software-packages validate to validate your input

Let's take a look at the update:

 1Command> software-packages list --staged  
 2[2020-12-20T17:52:00.355] :  
 3 category: Bugfix  
 4 kb: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-70u1c-release-notes.html  
 5 leaf_services: ['vmware-pod', 'vsphere-ui', 'wcp']  
 6 vendor: VMware, Inc.  
 7 name: VC-7.0U1c  
 8 tags: []  
 9 version_supported: []  
10    size in MB: 5107  
11 releasedate: December 17, 2020  
12 executeurl: https://my.vmware.com/group/vmware/get-download?downloadGroup=VC70U1C  
13 version: 7.0.1.00200  
14 updateversion: True  
15 allowedSourceVersions: [7.0.0.0,]  
16 buildnumber: 17327517  
17 rebootrequired: False  
18 productname: VMware vCenter Server  
19 type: Update  
20 summary: {'id': 'patch.summary', 'translatable': 'In-place upgrade for vCenter appliances.', 'localized': 'In-place upgrade for vCenter appliances.'}  
21 severity: Critical  
22 TPP_ISO: False  
23 thirdPartyInstallation: False  
24 timeToInstall: 0  
25 requiredDiskSpace: {'/storage/core': 6.286324043273925, '/storage/seat': 228.3861328125}  
26 eulaAcceptTime: 2020-12-20 17:50:12 AKST

Let's run it!

 1Command> software-packages install --staged  
 2 [2020-12-20T17:53:52.355] : For the first instance of the identity domain, this is the password given to the Administrator account.  Otherwise, this is the password of the Administrator account of the replication partner.  
 3Enter Single Sign-On administrator password:  
 4  
 5 [2020-12-20T17:54:02.355] : Validating software update payload  
 6 [2020-12-20T17:54:02.355] : UpdateInfo: Using product version 7.0.1.00100 and build 17004997  
 7 [2020-12-20 17:54:02,095] : Running validate script.....  
 8 [2020-12-20T17:54:09.355] : Validation successful  
 9 [2020-12-20 17:54:09,125] : Copying software packages  [2020-12-20T17:54:09.355] : ISO mounted successfully  
10166/166  
11 [2020-12-20T17:57:31.355] : ISO unmounted successfully  
12 [2020-12-20 17:57:31,238] : Running system-prepare script.....  
13 [2020-12-20 17:57:40,289] : Running test transaction ....  
14 [2020-12-20 17:57:54,344] : Running prepatch script.....  
15 [2020-12-20 18:01:22,731] : Upgrading software packages ....  
16 [2020-12-20T18:07:39.355] : Setting appliance version to 7.0.1.00200 build 17327517  
17 [2020-12-20 18:07:39,538] : Running patch script.  
18....  
19 [2020-12-20 18:28:42,743] : Starting all services ....  
20 [2020-12-20T18:28:46.355] : Services started.  
21 [2020-12-20T18:28:46.355] : Installation process completed successfully  
22 [2020-12-20T18:28:46.355] : The following warnings have been found:  
23['\tWarning: \n\t\tsummary: Failed to start all services, will retry operation.\n']  
24Command> shutdown reboot -r "patch reboot"

Looks like the manual install worked for me - 7.0 U1c

TL;DR

1rm -rf /etc/applmgmt/application/software_update_state  
2grep -i error /var/log/vmware/applmgmt/update_microservice.log  
3exit  
4software-packages unstage  
5software-packages stage --iso --acceptEulas  
6software-packages list --staged  
7software-packages install --staged  
8shutdown reboot -r "patch reboot"

Using VM Templates and NSX-T for Repeatable Virtual Network Deployments

Sat, 03 Oct 2020 16:46:00 -0800

So far, we've provided the infrastructure for continuous delivery / continuous integration, but it's been for those other guys.

Is that odd?

Let's try using the principles provided for more infrastructure-oriented reasons. Let's build a network lab using NSX-T.

First, we need some form of a mutable router. Normally, that'd be whatever flavor's "in production," but the specific implementation doesn't really matter.

First, we need to outline what basic functionality would need to be in place for this basic image to work:

Management Plane isolation: Build a separate "routing table," or VRF for the first applied interface.
Automatic connectivity. We should have some way to automatically get network connectivity separate from the "data plane," and perform configuration loading, command invocations, and software lifecycle management.
Enable inbound management protocols.

I have built a light configuration to do that here.

Once operational, we will want a good process to keep software up-to-date. Once established with this basic configuration, it'll be possible to SSH into this device and run the update process. Here's how:

 1vyos@vyos:~$ add system image https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso vrf mgmt  
 2Trying to fetch ISO file from https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso  
 3  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current  
 4                                 Dload  Upload   Total   Spent    Left  Speed  
 5100  309M  100  309M    0     0  1424k      0  0:03:42  0:03:42 --:--:-- 1551k  
 6ISO download succeeded.  
 7Checking for digital signature file...  
 8  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current  
 9                                 Dload  Upload   Total   Spent    Left  Speed  
10  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  
11curl: (22) The requested URL returned error: 404 Not Found  
12Unable to fetch digital signature file.  
13Do you want to continue without signature check? (yes/no) [yes] yes  
14Checking MD5 checksums of files on the ISO image...OK.  
15Done!  
16What would you like to name this image? [1.3-rolling-202010020117]:  
17OK.  This image will be named: 1.3-rolling-202010020117  
18Installing "1.3-rolling-202010020117" image.  
19Copying new release files...  
20Would you like to save the current configuration  
21directory and config file? (Yes/No) [Yes]: Yes  
22Copying current configuration...  
23Would you like to save the SSH host keys from your  
24current configuration? (Yes/No) [Yes]:  
25Copying SSH keys...  
26Running post-install script...  
27Setting up grub configuration...  
28Done.  
29vyos@vyos:~$ show system image  
30The system currently has the following image(s) installed:  
31  
32   1: 1.3-rolling-202010020117 (default boot)  
33   2: 1.3-rolling-202009200118  
34vyos@vyos:~$ reboot  
35Are you sure you want to reboot this system? [y/N] y  
36  
37...  
38  
39vyos@vyos:~$ show system image  
40The system currently has the following image(s) installed:  
41  
42   1: 1.3-rolling-202010020117 (default boot) (running image)  
43   2: 1.3-rolling-202009200118  
44  
45vyos@vyos:~$ delete system image  
46Possible completions:  
47  Enter       Execute the current command  
48  1.3-rolling-202009200118  
49                Name of image image to delete  
50  1.3-rolling-202010020117  
51  
52vyos@vyos:~$ delete system image 1.3-rolling-202009200118  
53Are you sure you want to delete the  
54"1.3-rolling-202009200118" image? (Yes/No) [No]: Yes  
55Deleting the "1.3-rolling-202009200118" image...  
56Done

Ta-da! new version! We cleaned up the old image for disk space compaction as well.

Our virtual router is built - let's shut it down, and then convert it to a template:

Ready to go!

Why Automate? Writing a self-testing Python class for REST or XML API invocation

Sun, 16 Aug 2020 17:05:00 -0800

So far, most API invocations, at least in terms of what you need to do, are pretty simple to execute.

Then again, just about every other administrative function on a computer is, as well. For example:

Clicking a button
Typing in a command or variable

Interacting with a programmable interface is as simple as any other interaction with a computer.

The primary goal with an API is not to simply replace any of those functions normally performed by a user. Using a programmable interface effectively skips most of the rigmarole performed by a skilled administrator, like:

Ensuring the change is correct
Ensuring the change is appropriate
Ensuring that the change won't have unexpected impacts

As an example, when you enter a vehicle to back it out of a driveway, you achieve these goals:

Correct: You ensure that you are entering the right vehicle, in the right driveway, and will head in the right direction.
Appropriate: You typically do not perform this action if you have no need, but you also don't use someone else's vehicle without permission, drive inappropriate speeds, or take unnecessary steps that could endanger life
Performs as expected: People are generally more unpredictable, so the analogy falls apart here. But generally, people get where they intend to go while driving.

While most people don't always fully realize that they're performing these steps, each is typically present. We see many instances in the industry where engineers are considered "unreliable". In my experience, these individuals just aren't aware of those steps, and simply need to make it a conscious effort.

This has to be a fully conscious effort when developing software or automating changes. While a programmable interface does not perform these things automatically, we can do them ourselves relatively easily, given the right tools.

Let's cover this in micro first - and cover the concept of unit testing.

Unit testing is based upon the principle, that, for every individual thing you can do programmatically, you should at least test once.

The website Software Testing Fundamentals actually covers unit testing itself much more thoroughly than I will here, as this is geared more towards immediate practical applications for people who don't exclusively write code for a living.

This is step one to ensuring that programmatic changes are correct, appropriate, and won't have unintended side effects or at least ensuring your infrastructure won't end up on r/softwaregore

For this to work, every single software function executed must be proven just like any other mathematical formula. Typically, the easiest way to do this from a pure mathematics standpoint is by trying the formula in reverse.

I'll be honest, this doesn't scale particularly well when dealing with infrastructure programmability. We used to joke in college that physicists and mathematicians would start with "assuming a cow is a sphere at absolute zero in a vacuum," but we didn't really understand that yet. The joke was probably inherited and re-used, where:

We, as engineers designing infrastructure, have limited time and resources to tackle the fractal complexity of what we consider the "real-world."

Infrastructure designers and maintainers live somewhere between the two, where software is based on mathematics but is slowly approaching the fractal complexity of the "real world."

So, we rip off what other engineering disciplines have done for millennia, component testing.

Typically, engineers test a component based on results, or by breaking a component. Some examples of where these approaches are practical are:

Mathematical proofs and sanity checks: Generally, if you ask for a fraction, you want a fraction. If you ask for a boolean, you want true or false. If you ask for a routing table, you probably don't want a VPN client table.
Simulations: Run the code against simulated production systems, remembering that machines don't really mind ugly levels of repetition. Sample sizes of less than 100% on individual tests are impossible in the real world, so test coverage stays in the low percentages and is later found statistically representative. We're not really burdened by this here!
Fuzzing: Intentionally feed garbage input, give a piece of software

Third-party tools like pipelines can cover automated test EXECUTION, but before we cover that, we need to cover how to test, and better yet, how to bake testing in so that it doesn't take much effort.

If you already have a library that you're re-using to execute changes, you're handing off responsibility for mathematical proofs, but as the person executing a change, you still have operational responsibility for any unintended effects. So you treat this as an engineer, and move forward with simulations and fuzzing.

Let's start by creating a Python class. PEP 8 - the style guide for writing python code, has a lot to say about names. I'll call this one IronStrataReliquary, for the following reasons:

CapWords: This is just what PEP 8 agrees is convention compliant.
Obvious:
Iron is a common prefix for Palo Alto coding projects - it's a portmanteau derived from a common acronym for Palo Alto Networks (PAN).
Strata is the currently rebranded signifier for Palo Alto's NGFW or "Core" Product line. this delineates from Cortex or Prisma.
We love things in threes. Reliquary is a thing that holds relics - I picked this because the word "toolbox" was too derivative
Unique: We want to package this class as an installable, and if the name conflicts with existing software, it's typically because of a class name.

From here, we structure the class by illustrating a rough outline for what the class should contain:

Initialization: This is not a C Constructor. This is effectively a script or function to bootstrap an object. In our case - __init__ contains or initial connection to a Strata appliance, and will prepare it for immediate use.
Variables: I am storing API XML responses against variable names in a table:
Name: What you'd find it by, annotated with the version first tested against
XML Query string: This is you asking the API for something
XML Response string: This is what a normal response should contain, in some form. See how easy this is?
HTTP Errors: Just a quick one - I didn't create it. I added in HTTP errors that an NGFW can throw as well.
API GET/POST functions. Feed this XML, they'll send it to an NGFW.
Data conversion: Interpret HTTP errors, convert XML to JSON, etc.

All I have to do, once done, is write an exceedingly simple script to test this out:

Since the array in question already stored expected responses, I'm able to apply a for loop and just iterate through all of the provided XML Queries and responses to test the code with nearly full coverage. After I've finished the rest of the PEP 8 / Code conformance, the last remaining work is to:

Explore the API and add more variables/responses
Export strata_bibliotheca to a JSON file for easy management outside of the Python class.

Why Automate, Part 2: RESTFul APIs and why they aren't as hard as you think

Sun, 24 May 2020 09:05:00 -0800

Let's be realistic about the API craze - it seems everything has one, and everybody is talking about API consumption in their environment as if they've invented fire.

Here are a few things to know about APIs that could have been communicated better:

Writing code to consume an API is easy. Most of the time, a cURL command will do what you need. To top it off, most platforms have a Swagger UI, or even better, an API Sandbox to guide you through it.
You have to write code to consume an API. Most of the time, you're simply buying a product that does this for you. For example, with ArubaOS all management plane traffic uses PAPI to communicate, and you just interact with the controller. Even better, platforms like Ansible and Hashi's Terraform make it as easy as defining what you want in a YAML file.
APIs need to be secured. As a security practitioner, this one is pretty scary. Think of an API as your SSH connection, but with less baked-in security controls, because the industry hasn't hardened m(any) of them yet. API proxies are really useful here because you can limit what permissions any given client can have.
APIs are useful in ways that the CLI isn't. There are features and advantages to performing work via any API - one of which is platform abstraction. You can easily write code to make changes to a Juniper switch as a Cisco guy, just by learning the automation constructs!
If you're sick of PuTTY/(insert SSH client here)'s bulk copy issues, the API is for you. Even if you don't want to regularly use an API for most things, bulk changes are typically authenticated and validated and will tell you where any breakage is. Next time you install a few hundred static routes, import multi-line ACL, try it. How do you validate that those changes went in today? Have you ever had issues with just one missing line when doing those bulk imports?

Let's try and consume an API with base code - just to see how easy it really is.

First, let's try something easy, adding a few hundred static routes to an NX-OS device. The main reason why I'm using NX-OS here is that the platform includes an "API Sandbox" by default, which should be disabled in production environments:

1no nxapi sandbox

That being said, we're using a lab, and it's stitched together via NSX-T. We can firewall, IDS, etc. the management and data plane of any simulated network asset, and connect them as arbitrary topologies to fit our needs really easily. These workloads (virtual routers & switches) should be ephemeral, so it should be OK for now. Later I'll go into automatically securing and loading base configurations.

Let's get started! Here's the NX-API Sandbox:

I generated an IP list of /32s starting from 1.1.1.1/32 up to 1.1.2.44/32 as null routes, with individual tags, and applied it accordingly. Then I set the format to JSON, mode to cli_conf, and set the error action to "rollback on error". this would convert everything into a common language, and roll back a change if there are problems.

Generated code is here.

First, we check the routing table beforehand:

1sho ip ro  
2IP Route Table for VRF "default"  
3'*' denotes best ucast next-hop  
4'**' denotes best mcast next-hop  
5'[x/y]' denotes [preference/metric]  
6'%<string>' in via output denotes VRF <string>

Then we run it:

1python3 nxapi-add-bulk-routes.py

And then we verify.

 1show ip ro  
 2IP Route Table for VRF "default"  
 3'*' denotes best ucast next-hop  
 4'**' denotes best mcast next-hop  
 5'[x/y]' denotes [preference/metric]  
 6'%<string>' in via output denotes VRF <string>  
 7  
 81.1.1.1/32, ubest/mbest: 1/0  
 9*via Null0, [222/0], 00:00:15, static, tag 1111  
101.1.1.2/32, ubest/mbest: 1/0  
11*via Null0, [222/0], 00:00:15, static, tag 1112  
121.1.1.3/32, ubest/mbest: 1/0  
13*via Null0, [222/0], 00:00:15, static, tag 1113  
141.1.1.4/32, ubest/mbest: 1/0  
15*via Null0, [222/0], 00:00:15, static, tag 1114  
161.1.1.5/32, ubest/mbest: 1/0  
17*via Null0, [222/0], 00:00:15, static, tag 1115  
181.1.1.6/32, ubest/mbest: 1/0  
19*via Null0, [222/0], 00:00:15, static, tag 1116  
201.1.1.7/32, ubest/mbest: 1/0  
21*via Null0, [222/0], 00:00:14, static, tag 1117  
221.1.1.8/32, ubest/mbest: 1/0  
23*via Null0, [222/0], 00:00:14, static, tag 1118

We can also roll back (script in GitHub):

1python3 nxapi-rollback-bulk-routes.py

And verify:

1show ip ro  
2IP Route Table for VRF "default"  
3'*' denotes best ucast next-hop  
4'**' denotes best mcast next-hop  
5'[x/y]' denotes [preference/metric]  
6'%<string>' in via output denotes VRF <string>

Just to be clear, this is a starting point. There is no error handling, no automatic validation, no secure storage of credentials. It's fantastic that Cisco and other vendors provide this, but there are quite a few things that should be improved with just a tiny bit of coding time:
- User-friendly formatting of the payload. You'll want to prettify the payload blob, so that it's easier to peer review.

- try-catch statements: You want to, at a minimum, get a 200 OK or 400 Failure of some kind, and report it to the executor of your script. This is pretty easy to capture.

- Automatic change validation: In this example, capturing the routing table after the fact could also be generated automatically by the sandbox, and would make for the perfect validation step. Be creative!

- Test Test Test: These API calls go by pretty quickly, and you don't have the typical MOP approach where constant validation is taking place. Get a lab, and thoroughly test your automation before using it on a live network.

My examples of automation implementations are here.

Cisco's library of NX-OS examples are here.

IPv6 Sage Certification with NSX-T, Part 2

Sun, 15 Mar 2020 08:51:00 -0800

To get past the first major test (Explorer), you simply need to access a page over IPv6, and pass a quiz. To do this, spin up a desktop VM on your dual-stack vn-segment and navigate to https://ipv6.he.net/certification/

To get past your next phase (Enthusiast) you do have to spend some money - purchase a domain (the cheaper, the better) and link it to he.net's name servers. Jacob Salmela has a pretty good step-by-step on this: (https://jacobsalmela.com/2013/10/30/ipv6-certification-walkthrough-enthusiast-level-hurricane-electric-part-3/)

From here, you should be able to get through it via trial and error. I recommend just spinning up a linux VM on that vn-segment and toying around with it, e.g. installing apache, postfix, etc.

One thing worth noting is that the last few phases (Professional on up) have automated tests that may need to be manually restarted by HE to work. If you get really stuck, you can ask them at ipv6@he.net.

IPv6 Up and Running - Dual-Stack connectivity with NSX-T

Sun, 15 Mar 2020 08:37:00 -0800

The next step is to get IPv6 up and running with NSX-T!

This should be pretty short - as with existing deployments of NSX-T, most of the difficult work is already completed. Here are a few preparatory steps to be performed before getting started:

Ensure MP-BGP is on and that the data center fabric is running the ipv6-unicast address-family.
Ensure the same on NSX-T manager by navigating to Advanced Networking & Security -> Networking -> Routers -> Global Config:

Now, let's review feature support (up to date as of NSX-T 2.5), as it's not really in the NSX-T documents. More detail can be found here

Routing
- IPv6 Unicast AFI
- eBGP and iBGP
- ECMP
- BGP Route Aggregation, Redistribution, tuning
Dataplane forwarding
- Route Advertisements
- Neighbor Discovery
- Duplicate Address detection
- DHCPv6 helper
Security
- Full Layer 4 firewalling
- IP Discovery/Security, e.g. IP spoofing prevention, DHCPv6 spoofing prevention

We're pretty much covered on the data plane portion, with one notable exception - IPv6 load balancing is not supported. Other things that are not supported include:

IPv6 native underlay: VTEPs, Controller-to-host communication is IPv4 only. I'd expect this to be resolved relatively soon...
NSX Manager cannot have an IPv6 address, nor can it cluster via IPv6
vCenter and ESXi still does not fully support IPv6. Additionally, with the deprecation of the FLEX UI, the experimental feature that allowed you to try is no longer exposed via any GUI.
Versions of vRA prior to 8.0 don't appear to support IPv6 autoconfiguration, so it may be a while before you can automatically invoke these features.

Now that I've been a total buzzkill on feature support (VMWare historically hasn't been great on this front), let's get to configuring!

First, let's configure an IPv6 address on our Tier-0 routers:

Add BGP Peers:

Note that you already have Tier-0 to Tier-1 automatically set up - click "View More" under router links, and you'll see it's using the prefix fcc4::, which is currently reserved by RFC4193 for Unique local connectivity. Props to VMWare for following spec!

There actually isn't much else to do here - you're done. You can add IPv6 subnets and profiles to segments really easily:

And that's it! Interestingly enough, you can run IPv6 only on NSX-T vn-segments as well - just create a new external interface, attach it to the VyOS VM via a vn-segment, and peer BGP.

IPv6 Sage Certification with NSX-T, Part 1: Requesting an extended prefix

Sat, 15 Feb 2020 15:14:00 -0900

As is probably obvious from the sidebar, I'm pretty enthusiastic about IPv6 - for quite a few reasons, not least of which is implementing a new Layer 3 protocol after guys like Vint Cerf already did most of the cool stuff.

However, I didn't want to simply complete this task - most people complete all of these tasks without properly implementing IPv6 - no routing, network configuration is required if you simply install a tunnel client on your computer and work from there.

So instead, let's introduce a lot of complexity and make it easier for the testing to fail.

First things first, since we have a whole network in play instead of a single Layer 2 domain, we need to request a bigger prefix. Since you can't (shouldn't) chop up a /64 for end devices, let's start with establishing a larger prefix. HE.net's tunnelbroker site lets us one-click request a /48:

So I'd recommend doing that - and from there we'd want to modify the tunnel created in my previous blog post, and chopping it up as you see fit.

I already have a dual-stack Clos fabric in my lab, so establishing tunneled connectivity here was trivial - standing up a VyOS virtual router (config here) and peering BGP with the fabric. This is pretty much the upside to Clos fabrics - you have flexibility in spades.

Why Automate, Part 1: Network Config Templating in Jinja2

Sat, 01 Feb 2020 09:21:00 -0900

Let's answer the big question: "What's the answer to the ultimate question of life, the universe, and everything?"

Kidding, it's easier to cover the question: "Why automate?"

So let's get started! Here I'm going to start a few easy and quick ways to benefit from automation, with a slight networking bias...

File Templating

Have you ever deployed a single-config device (doesn't have to be a router or switch) and encountered copy-paste errors, adding old VLAN names, from some master config (ideally) or other devices (not ideally)?
As it turns out, so many developers ran into this issue that they created a parsing language specifically for purposes like this - Jinja2.

When visiting their website, the API documentation can be a bit overwhelming. There are many features for single-file templating, but if your goal is to cookie-cutter generate device configurations, you don't need to learn all that much of it, as Ansible takes care of the vast majority of the coding required. That's right - no coding required.

The Basics

Jinja2 file templates emphasize the use of variables, and escape them with double curly brackets, for example:

1hostname {{ hostname }}

As a language, it also supports a hierarchy of variables:

1hostname {{ global.hostname }}-{{ global.switchid }}

This is pretty simple, right? The first step I'd recommend here is to go through any configuration standards you have and highlight all of the variables in it.

Now to add a little bit more difficulty - it's time to define the variables in a document, to eventually combine together with the Jinja template we're creating. This is incredibly difficult to do in a vacuum, as you need a good way to name/organize the variables. So let's take that highlighted document, and start attaching names / organizing them at the same time. I'd recommend using a text editor that supports multi-document editing, putting your variable list on one side, and your Jinja template on the other. Here's how I did it in Visual Studio Code:

As you can see, on the left I have used YAML to define attributes of a leaf switch, while adding the names into the template itself. I'll keep this brief, as there's one important aspect to automation here:

YOU are automating YOUR OWN, EXISTING, expertise on a platform. This is not replacing YOU, nor is it making YOUR SKILLS IRRELEVANT. Those skills are still absolutely necessary. YOU will still have to hand-configure and explore equipment like you always have. The biggest change YOU will see is that you'll have more time to test configurations and making them more reliable, instead of performing some of the more boring tasks like editing text files.
For this reason, I'm not going to get very prescriptive on the what or the how from here, as this is an exploration exercise that will vary greatly based on the use case. Here are some quick guidelines while trying it out:

Keep it organized! The Jinja document's supporting YAML file is there for YOU to read. Make it easy to do so.
If you think you'll need it, add it. You man not have a use case for making MTU a variable currently, but it's seeing widespread adoption in the data center and campus networks - if you think you may change it someday in the future, add it into the documents.
Use with extreme prejudice against your configuration templates!

Now that the vast majority of the work here is done, let's focus on the no-code way to combine these files. For this, all you need is python and Ansible, and pretty much any version works. To achieve this, Ansible has a pre-installed module called template.

 1---  
 2- hosts: localhost  
 3  tasks:  
 4    - name: Import Vars...  
 5      include_vars:  
 6        file: example-ios-switch-dictionary.yml  
 7    - name: Combine IOS Stackable Leaf...  
 8      template:   
 9        src: templates/example-ios-stackable-leaf.j2  
10        dest: example-ios-stackable-leaf.conf

..and that's it. Run it with the command ansible-playbook, and it will create a new file. Unfortunately, this requires one playbook per configuration, as the include_vars module doesn't unload anything from the YAML file.

Usage At Scale

This method scales extremely well - I have provided an example on Github (https://github.com/ngschmidt/j2-config-examples which leaves some standardized framework for keeping things organized, like using roles per device configuration, so it should be pretty easy to fork and expand to encompass multiple switches and multiple configuration standards, all in one repo.

In the real world, I use several Git repositories - the sheer quantity of templates and roles just gets out of control otherwise, and collaboration like using Git Pull Requests for continuous review and improvement (It's amazing what you can do with the saved time!) is much easier with that separation.

I've also generated an entire datacenter fabric configuration in seconds this way. Once you get your repositories organized, that's not even that big of a deal.

Demystifying CI/CD and Automation in General

You're already using automation. If you use Pull Requests to improve templates, you're simply formalizing previous practices you already did, but you also (probably) accidentally did CI/CD and network automation here.

A lot of DevOps gurus tend to treat automation work like it's the technological equivalent to inventing the wheel, and a lot of that is more to advance and protect the profession, and less a play to establish dominance / a place of power. Unfortunately, this tends to create a bit of a rift between them and the people they are there to help, but I've never seen that be intentional with DevOps engineers. They're developers, just like other ones, with a fiery burning passion for reducing boring, repetitive tasks for you, and making sure that the methods to do so are well-organized, and want to share those experiences. You don't need to give them a hug, but ask how they do stuff, it's probably the quickest way for you and them to learn something.

Securing Dual-Stack (IPv4,IPv6) Endpoints with NSX-T

Sun, 29 Dec 2019 08:39:00 -0900

I have mentioned in a previous blog post that I'm not using any ACLs on my tunnel broker VM

This is usually pretty bad, but again, we can get those protections outside of the VM - I'm using this to prove out how NSX-T can provide utility in this situation.

Solution Overview

VyOS is a fantastic platform, with a ton of rich, extensive features that can empower any network engineer to achieve greater outcomes. There's a lot of good stuff - here I'm using it as a tunnel broker, but we also have these other features:

Manageability

Configuration versioning: Any network platform with in-built configuration versioning (and its cousin, the wonderful "commit review" capability) gets a favorable vote in my book
API/CLI: The two have feature parity. It's source control friendly, as I have already shown
IPv6: You do not need an IPv4 management plane for this platform to work

Functionality

All routing protocols except IS-IS
All VPN functionality except VPNv4 (although EdgeOS, Ubiquiti's fork, has that. It shouldn't take long). This includes WireGuard and OpenVPN, and SIT as I used in this previous example
Full IPv6 support, including DHCPv6, RA, SLAAC, OSPFv3, MP-BGP, etc. The only thing missing is 6to4 for completely native IPv6 deployments

It'd be fair to say that VyOS is a fantastically capable router, which like Cisco ISR or any other traditional router, does have some downsides.

What's Missing - or What Could Be Easier

Just as a caveat, I do think we'll see this a lot with virtualized routing and switching.

VyOS has always had a bit of a problem with firewalling. I've been using it since it was simply Vyatta, prior to Brocade's acquisition, and the primary focus of the platform has always been high-quality routing and switching. Functions like NAT and firewalling are disabled by default and have an extremely obtuse, Layer-4 centric interface for creating new rules. This gets messy pretty quickly, as the rules themselves consume significant configuration space and have to be carefully stacked to apply correctly. This interface is manageable but becomes difficult at scale.

Of course, if it was my entire job to manage firewall policies, I'd automate baseline generation and change modifications, the platform is pretty friendly for that. This may not necessarily be maintainable if it's not placed in an area easily discoverable by other engineers, and definitely doesn't resemble the "single pane of glass" I'd rather have when running a network.

What I'd like to see is a way to intuitively and centrally implement a set of firewall security policies against this device, in a way that can be centrally audited, managed, and maintained. Keep in mind - the auditing aspect is critically important, as any security control that isn't periodically reviewed may not necessarily be effective.

Fortunately, VMWare's NSX (or as it was previously known, vShield) has been doing this for quite some time. There are some advantages to this:

Distributed Firewall enforces traffic at the VM's NIC, but is not controlled by the VM. This means that you don't have to automatically trust the workload to secure it.
VM Guest Firewalling CPU/NIC costs don't impact the guest's allocation. This blade has two edges:
VM Guests don't need firewall resources factored into their workload, as it's not their problem. This allows for easy onboarding, as the application you're protecting doesn't have to be refactored.
VM Hosts need CPU to be over-provisioned, as this will be taken out of the host resources at a high priority. This being said, if you're going down the full VMWare Cloud Foundations / Software Defined Data Center (VCF/SDDC) it is important to re-think host overhead, as other components such as vSAN, HA do the same thing!

Securing Workloads

First - we need to ensure that the IPv6 tunnel endpoint VM is on a machine that is eligible for Distributed Firewalling. From the NSX-T homepage, click on the VM Inventory:

Then we select the IPv6 tunnel VM:

From here, let's verify those tags, as we'll be using that in our security policies:

We also need to add some IP Sets - this is the NSX-T construct that handles non-VM or non-Container addressing for external entities. Technically, East-West Firewalling shouldn't always be used for this, but IPv6 tunnel brokering is an edge case: (IP Sets guide here)

From here, you want to add the IP Sets to a group via tag membership - a topic I will cover later as it's vitally important to get right with NSX-T:

We also want to do the same with our virtual machines:

We're all set to start applying policies to it! Navigate over to Security -> East-West Firewalling -> Distributed Firewall:

Add these policies. I have obfuscated my actual addresses under groups for privacy reasons.

That's about it! If you want to add more tunnel nodes, you'd simply apply the tag to any relevant VM with NSX Manager, and all policies are automatically inherited.

Some Recommendations

If you haven't deployed a micro-segmentation platform, the #1 thing to remember is that distributed firewalling, because it captures all lateral traffic, generates a TON of logs, all of which happens to be invaluable troubleshooting data. I'd recommend rolling out vRealize Log Insight + Network Insight (vRLI/vRNI) to help here, but ELK stack will probably work just fine in a pinch.
Have a tag plan! Retroactive refactoring of tags is a pretty miserable task, so try and get it at least well organized the first time.
Have a naming convention for all of the objects listed above! I'll write a skeleton later on and place on this blog, along with tagging strategies.
Make sure to set "Applied to" whenever possible, as this will prevent your changes from negatively affecting other data center tenants.
Try to use North-South firewalling (tier-0 and tier-1 edges ONLY) for traffic that leaves the data center. East-West wasn't really designed for that.
Try to use North-South firewalling, period. If a data center tenant (or their workload) is not globally trusted, assign that entity its own tier-1, making it really easy to wall off from the rest of the network. This is probably the easiest thing to do in NSX-T, and generates the most value!

IPv6 Up and Running - Address Planning Basics and using a Tunnel Broker

Sat, 23 Nov 2019 07:49:00 -0900

First things first - let's cover some IPv6 basics.

What's Different

Many aspects of IPv6 is actually much easier than most people would expect - since there's such a large addressing space, entire fields of work with IPv6 go away.

Custom CIDR / Subnetting

Remember how you had to do binary math, and use your crystal ball to guess how many hosts will be on any given subnet? Well, if you use CIDR masks from /29 to /19 for individual subnets, that will be replaced with a /64.

A great deal of functionality breaks if you use a subnet mask longer than /64 for generic devices - such as RA/DHCP. When setting up a network for any host-facing network, you need to remember only four masks:

/64: Use this everywhere
/126: Use like a /30, but ONLY when interconnecting network devices. You're not saving space by trying to use this for hosts.
/127: Use like a /31, but with even more flakey vendor support. This is more space efficient, but you need to verify that ALL of your equipment supports it, or deal with a really fragmented point-to-point prefix.
/128: Loopbacks

NAT

You don't need it, because it's IPv4 duct tape. Prepare yourself for a simpler life without it.

Private Addressing

IPv6 does take a different approach here - there are TWO "private" allocations:

Link-local addressing (fe80::/10): This addressing allocation is used on a per-segment basis, and pretty much just exists so that every IPv6 speaker will always have an IP address, allowing routing protocols to work on unnumbered interfaces, for example.
ULA (fc00::/7) Unique local addresses are on the should not be routed list, and should not be used, generally speaking. You have to use NAT Prefix translation to be globally routable, a feature that isn't well supported. I use this in my spine-and-leaf fabric examples to avoid revealing my publicly allocated prefix, and only in my lab.

Instead, IPv6 architecture focuses on the inverse - allocating prefixes you CAN use. Right now the planet (e.g. Earth, not kidding) has the Global (hehehe) allocation of 2::/3. All IPv6 prefixes are allocated out of this block by providers, using large allocations to ensure easy summarization.

DHCP

DHCPv6 is not mandatory, as SLAAC/RA Configuration can provide any client device with the default gateway and DNS servers. For enterprise applications, however, it is recommended to use DHCPv6 so you don't unintentionally disclose any information encoded into your IP by SLAAC, and so that your ARP tables aren't murdered by SLAAC privacy extensions. More here.

DNS

DNS actually isn't all that different anymore, but still deserves mention for a few reasons.

The first reason why I think it deserves mention is because, as an application, its IPv6 journey was extremely well designed.

IPv6 Constructs are available, regardless of which "stack" you're running: Global DNS Servers have a new (ish) record type, AAAA, that indicates that IPv6 is available for any service, and any DNS server should serve AAAA records, even if solicited on IPv6. This is useful in situations where your DNS server may have additional attack surface over IPv6, like Microsoft's Active Directory servers. It also helps make your migration strategy a bit smoother, as you implement the IPv6 stack progressively throughout your network.

Second, if you don't have AAAA resolving, IPv6 won't do much for you.

IPv6 Address Planning

IPv6 address planning is fundamentally different for the reasons listed above, but I do have some general guidelines that help establish a good starting point:

/48 and /56 are good site prefixes: Since we are using 8x the space in our FIB for each route, allocate a /48 or /56 depending on size per site, but don't do anything weird like allocating a /63 or a /62 to save space. Keep your sites consistent. A /56 is the IPv6 equivalent of a /16 in IPv4 - you'll almost always be right allocating at this length.
Allocate the last 2 /64s in your prefix for point-to-point prefixes and loopbacks, respectively. It just keeps address fragmentation less messy, and you can summarize the /64s at your backbone to ensure that traceroute "just works".
You have lots of space, leave gaps between sites. If you get a /48, you have 255 sites to play with. You can block out entire regions, sites, in a myriad of ways to help your routing table "make sense".

Here's how I did it (/48 allocated to me, prefix is masked):

ffff:ffff:ffff:ffff::/64: Loopbacks
ffff:ffff:ffff:fffd::/64: Point-to-point links
ffff:ffff:ffff:e::/49: Allocated to NSX-T, because I don't have multiple sites in my lab. Don't do this in the real world, this is for various (messy) experiments with address summarization.
ffff:ffff:ffff:b::/49: Allocated to the underlay fabric. See above.
ffff:ffff:ffff:a::/64: Home campus network. This is where Pinterest, and other meatspace activities live.

I'm actually not using much else - I'm allocating large because IPv6 Address shortening makes it easier to type (P.S. IPv4 Address shortening works too, but there are fewer opportunities. Try and ping 1.1) and allocating properly would look like:

ffff:ffff:ffff::/56 for Site A (Maybe a headquarters location?)
ffff:ffff:ffff:001::/56 for Site B (Satellite office near HQ?)
ffff:ffff:ffff:008::/56 for Site C (in another geographic region or state?)
ffff:ffff:ffff:1::/56 for Site D (HQ in another country?)

Hopefully this is helpful - when in doubt, whiteboard it out.

Well that's nice, but I'd like to actually do something!

Let's go through the process of selecting a tunnel broker (this assumes you do not have native IPv6 connectivity, because this would already be done):

Step 1: Search and select the best tunnel broker for you. Since I'm in the United States, I selected Hurricane Electric. I am biased by their educational outreach and certification program. I cannot recommend enough taking a crack at their Sage certification.

Step 2: Sign up using the links provided in the cheat sheet. If possible, ask for a /48 for maximum productivity.

Step 3: Establish a tunnel - I have provided a VyOS template here, but a great deal of networking equipment supports SIT tunneling, so it's not particularly difficult to set up. Keep in mind that there's no firewall enabled here, I wouldn't recommend the same approach, but I'm doing that elsewhere.

Step 4: Start experimenting!

Anycast Stateless Services with NSX-T, Implementation

Sat, 26 Oct 2019 15:01:00 -0800

First off, let's cover what's been built so far:

To set up an anycast vIP in NSX-T after standing up your base infrastructure (already depicted and configured), all you have to do is stand up a load balanced vIP at multiple sites. NSX-T takes care of the rest. Here's how:
Create a new load balancing pool.

Create a new load balancer:

Create a new virtual server:

If your Tier-1 gateways have the following configured, you should see a new /32 in your routing table:

Repeat the process for creating a new load balancer and virtual server on your second Tier-1 interface, pinned to a completely separate Tier-0. If multipath is enabled, you should see entries like this in your routing table:

It really is that easy. This process can be repeated for load balancers, and (when eventually supported) multisite network segments.

A few caveats:

State isn't carried through: if you're using a stateful service, use your routing protocols (AS-PATH is an easy one) to ensure that devices consistently forward to the same load balancer
Anycast isn't load balancing: This is easy here, as NSX-T can do both. This won't protect your servers from overload unless you use one.
Use the same server pool: It was (hopefully) apparent that I used the same pool everywhere. Try to keep regional configurations consistent, to ensure that new additions aren't missed for a pool. Server pools should be configured on a per region or per transport zone basis.

Some additional light reading on anycast implementations:

Cloudflare's Anycast

Google Public DNS

F5 BIG-IP DNS

Anycast Stateless Services with NSX-T, the Theory

Sat, 19 Oct 2019 22:07:00 -0800

Before getting started, let's cover what different IP message types exist in a brief summary, coupled with a "day in the life of a datagram" as it were.

Unicast:

One source, one well-defined destination. Most network traffic falls into this category.

Mayfly perspective:
Source device originates packet, and fires it to whatever route (yes, hosts, VMs and containers can have a routing table) matches based on the destination.
The destination router, if reachable, forwards the packet, and decrements the time-to-live (TTL) field by 1. Rinse and repeat until the destination is reached. Note: the TTL field is 8 bits, so if a message needs over 255 hops, it won't make it. (we're looking at YOU, Mars!) Pretty boring, but boring is good.

Multicast:

One source, many specific destinations. This has a moderate gain in efficiency over bandwidth constrained links when routed.

In most cases, if a group pruning protocol, e.g. IGMP, MLD, is not running, multicast traffic "floods" and distributes all messages across all ports. The most common application for multicast is as a discovery or routing protocol.

Mayfly perspective:
Source device originates packet and the next layer 2 device replicates the packet to all multicast destinations (if IGMP/MLD is not doing its job, this becomes a flood, and forwards on all ports, which removes the forwarding efficiency) and then stops.
If multicast routing is enabled, traffic will forward just like it did with unicast, and have a moderate increase in efficiency. This is at the expense of traffic control. Since all multicast traffic is inherently stateless, there's no way to manage bandwidth consumption, fully eliminating the efficiency gain in many cases. If you're running routed multicast, I'd highly recommend using BGP to prune the multicast table... to help with some of this.

Broadcast:

One source, ALL destinations. This is usually the least efficient traffic type and is part of why most networks don't have one all-encompassing VLAN, but instead use a number of subnetworks. With some exceptions, this traffic type is exclusively for when a source doesn't know how to get to a destination, e.g. ARP.

Mayfly Perspective:
Source device originates packet and the next layer 2 device floods on all ports but the origin (unless it's a hub). This traffic is subsequently dropped by all layer 3 forwarding devices unless a broadcast helper address is configured.

Anycast:

Unicast with a twist. Addresses (or networks) are advertised by multiple nodes, all capable of providing a service, enabling an end device to speak to the nearest available node.

Mayfly Perspective:
Source device originates packet and forwards on the appropriate interface leverages whatever routing metrics will choose. Next Layer 3 device will forward traffic to the available node with the most favorable routing protocol metric.

There's a lot to unpack here. Let's focus on the main points re: Anycast:

It DOES forward to the nearest available node, and if configured correctly, will use less reachable nodes as a backup.
It DOES NOT load balance traffic in any meaningful way.
It DOES NOT retain state

This is a pretty big deal-breaker, but let's keep in mind that we have more tools - these incapabilities are completely achievable. The only things you need to provide to make a anycast service are:

A load balancer
A load balancer that provides stateful services, or one that will synchronize state.
A load balancer

NSX-T conveniently provides the above with fully integrated routing and switching (We set up BGP, the routing protocol of the internet before), and adds micro-segmentation firewalling to boot. I'll cover more of that on the next post.

Before we go much further, this is a critically important that we understand something very fundamental.

ALL OVERLAY NETWORKING WITH NSX-T, SHOULD BE ANYCAST BY DESIGN.

I know it sounds dramatic, but VMWare's concept of a "transport zone" seems to imply that universal reachability via a PORTABLE SUBNET is the primary goal. In NSX-V, this was described as a Universal Distributed Logical Router (UDLR), and does not appear to be fully implemented in NSX-T. As a network designer, we should plan for universal reachability leveraging the Anycast model, e.g. "Will the nearest NSX-T Edge please stand up" wherever possible.

Hopefully, it is clear by now, but Anycast isn't a specific IP message type, but instead a design for network reachability. It's commonly Unicast, but can be multicast if an implementation is carefully designed. The core principle for Anycast is to provide the shortest path to an asset, to the best knowledge of the network routing protocol.

More on the practical side of this post, but common Anycast applications include:

DNS
Application load balancers
Content Delivery Networks (CDNs)

Coming soon - how to do this with NSX-T!

BGP Graceful Restart, some inter-platform oddities, what to do with it

Sat, 12 Oct 2019 09:19:00 -0800

Since most of NSX-T runs in a firewall mode of sorts, it's probably worthwhile to discuss on of the less well-known routing protocol features - Graceful Restart.

As published for BGP, IETF RFC 4724 outlines a mechanism for "preserving forwarding traffic during a BGP restart." This definition may be a little misleading, but that's mostly because of HOW the industry is leveraging Graceful Restart. Here are a few of the "normal use-cases" for BGP GR:

Cisco Non-Stop Forwarding and other similar technologies:
Cisco has developed another standard - NSF - that applies industry-generic methods for executing a BGP restart with forwarding continuity, with a twist. In many cases, multi-supervisor redundancy is a popular way of keeping your high-availability numbers up, with either a chassis switch running multiple supervisor modules or multiple devices bonded into a virtual chassis. In theory, these implementations get better availability numbers because they'll keep the main IP address reachable during software upgrades or system failures.
In my experience, this is great in campus applications, where client devices don't really have any routing/switching capability (like a cell phone) and where availability expectations are somewhat low (99%-99.99% uptime). However, in higher availability situations or ones running extensive routing protocol functionality, this appears to fall apart somewhat, where the caveats start to break the paradigm:

ISSU caveats: You have to CONSTANTLY upgrade your routers because ISSU is typically only supported across 1 or 2 minor releases. If you have a "cold" cutover, i.e. with a major version upgrade, you'll see a pretty extensive outage (5-30 minutes long depending on hardware)
Older implementations of a multi-supervisor chassis tend to have configuration sync issues, you need to CONSTANTLY test your failover capability (I mean, you should do that anyway...)

Just my 2 cents. But here's where Graceful Restart does its job: During a supervisor failover, the IP address of the routing protocol speaker is shared between supervisors, so when establishing a routing protocol adjacency, the speakers negotiate GR capability, along with tunable timers. Since the IP doesn't change, the greatest availability action would be to continue forwarding to a "dead" address until the adjacency is established, ensuring sub-second availability for a dynamic routing protocol speaker (except in the case of updating your gear...)
Firewalls:
Most firewall implementations are either Active-Active or Active-Standby, with shared IP addresses and session state tables. Well-designed firewall platforms use a generic method for sharing the state table, which includes (ideally) the session table, routing table, etc. ensuring that mismatched software versions do not introduce a disproportionate outage. The primary downside to this approach is that you don't have a good way to test your forwarding path (beyond Layer 2) so you should TEST OFTEN.

Now let's cover where you should NOT use Graceful Restart:
Any situation where the routing protocol speaker does not have a backup supervisor or any state mechanism. Easy, right?

NOPE. You have to enable Graceful Restart on speakers that have an adjacent firewall (or NSX-T Tier-0 gateway) to support the downstream failover.

RFC 4724 outlines two modes for Graceful Restart: Capable and Aware. Intuitively, GR Capable speakers should be stateful network devices, such as multi-supervisor chassis, firewalls, or NSX-T edges, and GR Aware devices should be stateless network devices, such as layer 3 switches.
The catch, however, is that not all devices support GR Awareness mode. For example, it IS supported in IOS 12, but provides caveats on what hardware has this capability.

So why does this matter? Well, Cisco illustrated it well in this NANOG presentation by stating that if an NSF-Capable advertising device fails, but there is no backup device sharing that same IP address, all traffic is dropped until the GR timers expire. Ouch. This is especially bad given some defaults:

RFC 8538 Recommendation: 180 seconds
Palo Alto: 120 seconds
Cisco: 240-300 seconds
VMWare NSX-T: 600 seconds?!?!?!?

Now that's pretty weird. If we fetch from VMWare's VVD 5.0.1, it says the following:

NSXT-VISDN-038 Do not enable Graceful Restart between BGP neighbors. Avoids loss of traffic. Graceful Restart maintains the forwarding table which in turn will forward packets to a down neighbor even after the BGP timers have expired causing loss of traffic.

Coupled with the recommendation for Tier-0 to be active-active (remember, as I stated before, stateless devices do NOT need GR):

Oddly, it did not warn me about needing to restart the session. Let's find out why:

 1bgp-rrc-l0#show ip bgp summary  
 2BGP router identifier 10.6.0.0, local AS number 65000  
 3BGP table version is 84, main routing table version 84  
 47 network entries using 819 bytes of memory  
 511 path entries using 572 bytes of memory  
 614/6 BGP path/bestpath attribute entries using 1960 bytes of memory  
 72 BGP AS-PATH entries using 48 bytes of memory  
 80 BGP route-map cache entries using 0 bytes of memory  
 90 BGP filter-list cache entries using 0 bytes of memory  
10BGP using 3399 total bytes of memory  
11BGP activity 102/93 prefixes, 264/247 paths, scan interval 60 secs  
12  
13Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
1410.6.0.240      4 65000  143031  142962       84    0    0 14w1d           2  
1510.6.0.241      4 65000  143036  142962       84    0    0 14w1d           1  
1610.6.99.1       4 64900  330104  280526       84    0    0 1d17h           1  
1710.6.200.2      4 65001  178250  174230       84    0    0 1w0d            3  
18FD00:6::240     4 65000  310833  578924       84    0    0 14w1d           0  
19FD00:6::241     4 65000  301493  578924       84    0    0 14w1d           1

Note that for GR to be modified, the BGP session must re-start, so if this was a production environment with equipment that supports GR (*sigh*) you would want to get into the leaf switch and perform a hard restart of the BGP peering.

VMWare's VVD recommendation here is pretty sound, as with most devices the GR checkbox is a global one, so you'd want to buffer between GR/Non-GR with a dedicated router (it's just a VM in NSX's case!), keeping in mind most leaf switches will have GR enabled by default.

Oddly enough, Cisco's Nexus 9000 platform (flagship datacenter switches) default to graceful restart capable. My recommendations (to pile on with the VVD) on this platform would be to:

Set BGP timers to 4/12
Set GR timers to 120/120 or lower (they're fast switches, so I chose 30/30)
Under BGP, configure graceful-restart-helper to make the device GR-Aware instead of GR-Capable

Obviously, the VVD will adequately protect your infrastructure to issues like this, but I think it's unlikely you'll have NSX-T as the only firewall in your entire datacenter.

NSX-T 2.5 Getting Started, Part 2 - Service Configuration!

Sat, 05 Oct 2019 07:52:00 -0800

Now that the primary infrastructure components for NSX-T are in place, it is now possible to build-out the actual functions that NSX-T is designed to provide.

A friendly suggestion, make sure your Fabric is healthy before doing this:

NSX-T differs from NSX-V quite a bit here. Irregular topologies between edge routers aren't supported, and you have to design any virtual network deployments in a two-tier topology that somewhat resembles Cisco's Aggregation-Access model, but in REVERSE.

The top tier of this network, or as VMWare calls it in their design guide, Tier-0, the primary function provided by logical routers in this layer are simply route aggregation devices, performing tasks such as:

Firewalling
Dynamic Routing to Physical Network
Route Summarization
ECMP

The second logical tier, Tier-1 is automatically and dynamically connected to Tier-0 routers via /31s generated from a prefix of your choosing. This logical router will experience a much higher frequency of change, performing tasks like:

Layer 2 segment termination/default gateway
Load Balancing
Firewalling
VPN Termination
NAT
Policy-Based Forwarding

Before implementing said network design, I prefer to write out a network diagram.

Let's start with configuring the Tier-0 gateway:

We'll configure the Tier-0 router to redistribute pretty much everything:

Configure the uplink interface:

Oddly enough, we have spotted a new addition with 2.5 in the wild - the automatic inclusion of prefix-lists!

We also want to configure route summarization, as the switches in my lab are pretty ancient (WS-3560-24TS-E). I'd recommend doing this anyway in production, as it will limit the impact of widespread changes. To pull that off, you *should* reserve the following prefixes, even if they seem excessive:

A /16 for Virtual Network Services per transport zone
A /16 for NSX-T Internals, allocating /19s to each tier-0 cluster, as outlined in our diagram.

I did so below, and it makes route aggregation or summarization EASY.

Now, we configure BGP Neighbors:

At this point, we want to save and test the configuration. It'll take a while for NSX-T to provision the services listed here, but once it's up, you'll see:

Check for advertised routes. Only routes that exist are aggregated, so you should only see 10.8.0.0/16:

As a downside, I have prefix-filtering to prevent my lab from stomping on the vital pinterest and netfix network, so I had to add the new prefixes to that:

That was quite a journey! Fortunately, Tier-1 gateway configuration is MUCH simpler, initially. Most of the work performed on a Tier-1 Gateway is Day 1/Day 2, where you add/remove network entities as you need them:

Let's add a segment to test advertisements. I STRONGLY RECOMMEND WRITING A NAMING CONVENTION HERE. This is one big difference between NSX-V and NSX-T, where you don't have this massive UUID in the port group obfuscating what you have. Name this something obvious and readable, your future self will thank you.

Hey look, new routes!

As I previously mentioned, these segments, once provisioned, are just available as port-groups for consumption by other VMs on any NSX prepared host:

Next, we'll configure NSX-T to make waffles!

NSX-T 2.5 Getting Started, Part 1

Sun, 29 Sep 2019 10:46:00 -0800

Since NSX-T 2.5 just came out, it's about time to do a full rebuild and getting started guide. NSX-T differs greatly from NSX-V in that the initial setup is quite a bit more complicated and doesn't have many guardrails or direct paths to initial set-up.

We'll be skipping the appliance deployment, because if you have troubles deploying an OVA this will probably be too difficult.

First off, we'll be using our applied Clos fabric for this, and we won't be multihoming these devices as of yet, as this post will be pretty lengthy as it is. Diagram is here:

With that in mind, the first step to configuring virtualized routing & switching for NSX-T is in the vCenter GUI. In this lab, I have two hosts in two separate clusters -

Payload: Virtual Tunnel Endpoints (VTEPs) exist primarily on the host, and are leveraged as port-groups for guest network connectivity
Management/Edge: No host VTEPs currently exist, as they are not required for the management VMs, nor for the Edge Appliances (Primary difference coming from NSX-V!)

Coming from the vCenter UI, it looks like this:

The NSX-T Edge Appliances need to ingest underlay networks via 802.1Q tags, instead of as individual port groups. Fortunately, vSphere has been able to do this for quite some time, so we use the lesser-known "VLAN trunking"

From here, it's time to outline our Edge Design - BEFORE anything is built.

We'll use this as a guide throughout the configuration process., First, we set transport zones and device profiles:

We create the underlay (VLAN) transport zone to ensure that virtualized traffic can exit to the "real network":

We create the overlay network where the GENEVE VN-Segments will live next:

Then we configure the Layer 2 uplink profiles. Note: specifically configuring the Active uplink to FP-ETH0 is REQUIRED. The NSX Edges will not function without this, and NSX-T will never tell you why.

And the VTEP profiles. Note that this portion uses the name allocated in the transport node profile.

Finally, the host transport profiles. Here we set a profile that will use a single uplink for the N-VDS, add transport zones, etc. Note that the physical NIC name on the left needs to exactly match the physical NIC identifier in ESXi.

Now, we can finally start configuring transport nodes. Note that since we deployed profiles prior to this, there's not a whole lot to do as far as roll-out is concerned.

Ensure the edge appliance is ready:

Configure the edge cluster:

Now we're ready to configure routing and switching functionality. This can go several different ways, as VMWare has provided additional capabilities with regards to configuring NSX-T assets - declarative configuration methods. We'll cover that in detail, along with how to use it, in the next post!

Spine and Leaf Networks, an Outline

Sat, 21 Sep 2019 09:08:00 -0800

With the previous post, I covered traditional data center networking, and some of the reliability compromises made to accommodate typical workloads. Here I'll be outlining my take as a series of blog posts on the next iteration of data center network design, Spine and Leaf:
Lab Diagram

What came before
Introduction: What Layer 3 Leaf Spine is and isn't, Dispelling Myths
Topology Diagram
Practical Applications with IGPs
RIP
OSPF
Practical application with EGP only
eBGP
Practical applications with IGPs and EGP
The IP Portability / Layer 2 problem
What I've found while practically applying this, tips and tricks
VMWare NSX-V
VMWare NSX-T
Cisco NX-OS & Automation

Note: This page may change based on content. I'm going to try and keep things generic up until the very end.

NSX-T Datacenter 2.5 Upgrade Process and Preview

Sat, 21 Sep 2019 09:07:00 -0800

Now that NSX-T Datacenter 2.5 is downloadable, it's time to try this out in my home lab.

First things first, if you log in more than 90 days out, you'll be locked out of the appliance completely. If you make any changes the normal linux way (passwd and chage) the appliance will automatically revert it in about a minute. Since this is a home lab, VMWare has added the capability to set a higher maximum age here. In production, use Active Directory or another LDAP source to prevent yourself from losing NSX-T.

Downloading the upgrade bundle took quite a while - it seems that VMWare is having troubles with hosting capacity. I'm guessing that it's going to be a popular release!

We start these in the usual way, by uploading the upgrade bundle (.mub file):

Then we hit the upgrade button, and it prompts you step-by-step throughout the process. No progress bars, though!

Once the upgrade coordinator is set up, it's time to run pre-checks. This one warns you about the messaging port change.

It's time to start upgrading stuff!

As with other 2.4 releases, you don't have to use maintenance mode (expect an outage if you do that).

The management node will be unreachable unless you roll-out a cluster. If you see an "appliance unhealthy" or "Error Status 101" message, this simply means the appliance isn't ready yet.

Post-upgrade, we can see BGP statuses as promised:

Capacity Management is there:

Unfortunately, I see BGP status but no route table in the GUI. The documentation will probably direct things somewhat, but I do not see the full capabilities there.

NSX-T Datacenter 2.5 Released!

Sat, 21 Sep 2019 07:15:00 -0800

As of 19 September 2019, NSX-T 2.5 has been officially released and is available for download!

It's been a bit since the announcement, so let's cover some of the new capabilities of interest with NSX-T 2.5. This is a summary of what I found interesting, the complete release notes are here

NSX Intelligence

VMWare will be introducing a new paid service to analyze traffic handled by distributed firewalling, to allow infrastructure administrators to map out service applications, ports, and policies to better secure their east-west network environment. It will also provide the capability that NSX-V has natively, Application Rule Manager.

Testing and Troubleshooting

VMWare added a ton of good stuff here, some of which seems a little late...

Layer 2 MTU/VLAN Checking

This one has been a big pain point for NSX administrators everywhere, especially if they don't also control the route-switch infrastructure. Prior to this, NSX-T had tunnel status (which would alarm if no VMs in a port group were on a host, causing a LOT of noise) and NSX-V had nothing.

Layer 3

We get BGP routing information from the API and GUI for the first time!

New Capabilities

IPv6

We pick up SLAAC, Router advertisements allowing for automatic IP configuration. Ideally, this would not be something we really need - but I'm sure there's a use case somewhere.

Firewalling and Security

NSX-T now supports configuration management as well, with config drafts!
NSX Cloud is beginning to support native constructs in public cloud for security enforcement. This is a pretty big deal for hybrid cloud shops that won't have to use an agent to enforce consistent multi-cloud security!
VMWare has introduced Layer 7 (App-ID) support for gateways and is beginning to introduce FQDN filtering as a precursor to URL filtering.
VMWare has also added Identity-based firewalling.
Elliptic Curve Cryptography over IPSEC is now available
Preset compliance suites for VPNs are also available

Other

Load Balancing GUI Improvements - We'll see the simplified GUI in a bit.
SNMPv3 Polling is supported on all appliances
The NSX-V to NSX-T migration tool has unlisted improvements
NSX Manager to Edge communication is changing ports - from 1234 to 5671. This could potentially break connectivity during an upgrade. Port 1235 does still need to be open.

Next, let's try it out!

Spine and Leaf Practical Applications, The IP Portability Problem

Sat, 29 Jun 2019 09:48:00 -0800

So far, this is all great, but it IS missing something

Networks are useless unless you do something with it. In most cases, a device (in this case, we'll use a server) needs to connect via redundant links to a common Layer 2 segment.

Why is this?

Well, most servers are incapable of dynamic routing. Instead, the server (which is a perfectly capable router as far as forwarding is concerned) simply has a static route (default gateway) that is used for all Layer 3 forwarding. This is not really a deal breaker for Clos fabrics - there are a few ways to solve this problem - and several of them intermix really well:

The VMware Way

This is probably the most achievable. It's not really a Clos fabric, due to some deficiencies (ESXi doesn't do BGP yet) that will probably be resolved at some point, but it is close enough to achieve our goals. Let's review those

Make change frictionless and low risk so network changes can be on-demand (The Change Problem)
Ensure that the network utilizes all links, with consistent forwarding (The Loop Problem)

The primary value proposition with Layer 3 Leaf Spine (a Clos implementation) is to leverage a consistent 3-stage (leaf, spine, leaf) forwarding topology where all links have the same exact latency and link speed. This, along with some other features (ECMP support being the big one) allows for N-scaling leaf-to-leaf communications - you can have 1,2,..64 spines in a network.

Cisco really pushed this to the limit, publishing a paper on a reference implementation where leveraging 16+ spines actually saved money versus using QSFP+ capable devices. The conclusion is somewhat dated due to QSFP28 coming out and being more affordable, but the takeaway should be the same - BGP/IS-IS are built to facilitate tens of thousands of network nodes in irregular topologies. Datacenter networks with hundreds of switches don't really hold a candle to that, but we can use this overkill to our advantage.

VMWare is also now on board with this topology, because they're starting to solve the routing problem with NSX. The currently published reference architecture (VMWare Validated Design 5.0 at the time of this post) featured a new compromise with version 4.0 - linking ToR switches into pairs more like a traditional switch deployment, with VLANs subtending the leafs to provide server reachability.

There's a problem here - how do you get virtual machines to keep their IP address when moving between ToR pairs?

This is where NSX comes in. NSX-V/T both provide overlay networking, where dynamically pinned tunnel adapters (like GRE on ubersteroids) manage membership for virtual network segments inside of an encapsulation method (VXLAN/GENEVE) providing a fully virtualized Layer 2 segment, portable anywhere. This ensures that the only thing that isn't portable is the servers, which is good enough for now.

VMWare's approach isn't "pure" (whatever that means) but when revisiting our goals here (to provide an ultra-stable, change-friendly datacenter network), it does meet our needs and provides a demarcation point where the changes are substantially simpler as far as the datacenter fabric is concerned. If NSX breaks on a host, you may lose part of a vn-segment or a few VMs at worst. The fabric failing is far more disastrous.

Pros:

Change risk is low due to distributing the work
Highly flexible
NSX-T can run on things that aren't ESXi

Con:

You're going to need to blur the line between "Network" and "Systems". I've seen a pattern in many prod environments - where an organization's networking team will manage the vSphere Distributed/Standard Switches to ensure that switch and host are well-integrated. If this model is not one that is organizationally feasible, you'll have a difficult time with NSX. Even if it isn't, your Network/Systems team must cross-train.

The Cisco / Big Switch Way

Another option is to fully offload the responsibility of overlay networking onto the datacenter fabric, maintaining a "pure" Clos topology, and handling the ToR bonding in software with the same overlay technology. I'm keeping this at a high level because, honestly, I haven't worked anywhere large enough to benefit yet.

Pros:

Works on just about anything
Usually comes with an automated lifecycle management and provisioning platform
You can keep your network and systems teams separated

Cons:

If it's not a use case your vendor anticipated, you don't get the flexibility you need
Vendor lock-in is basically guaranteed
Doesn't run on generic hardware
$$$

The Mad Scientist Way

...just install a routing package on every virtual machine, docker host, or virtualization host. Run DHCP for the initial address issuance, and then run OSPF or BGP with a dynamic range, and then advertise a loopback address for the service you're offering.

It's not actually all that hard. If you have a linux OS, you just need to install a software package to run a routing service. If you're a systems guy, this is easier than setting up a LEMP stack! Here are some examples of open-source, publically available software that will perform this task:

Quagga (All RPs, including IS-IS)
BIRD (OSPF/BGP/RIP, includes MPLS)
OpenBGPD (BGP)

My new favorite is Free Range Routing. It's under the hood of VMWare's new version of NSX (NSX-T) Cumulus Networks, and a ton of other stuff. It's the most feature complete of this list, performing tasks that you'd normally pay far more for (Cisco still has BGP as an add-on license).

One neat thing this can provide is the concept of anycast network services. For a stateless service like DHCP, DNS, etc it is possible to leverage one of these daemons to advertise a common address. Instead of searching for the correct server or assembling a shortlist of DNS services, clients can simply ask for the nearest DNS server - this is how many exascale DNS implementations work, like:

1.1.1.1 (CloudFlare)
8.8.8.8 (Google)
9.9.9.9 (Quad9)

The downside to this approach is that you have no formal support whatsoever - which is a pretty big con. The good news is that there are some commercially viable host-routing products out there like Cumulus' host pack (white paper here) Eventually, products like this will be run as a plug-in on common hypervisors.

Conclusion

Why and where should I try this?

Let's keep this simple - applied Clos datacenter fabrics will require some level of solution design - it cannot simply be forklifted in the datacenter, but many products are available today that will solve the near-term issues. Few of these implementations are perfect, so design for iterative improvement, leave extra ports at the datacenter perimeter for new versions, etc.

What routing protocol, technology should I use?

Use what you know. The lab examples I provided in this block were manufactured in 2002. If it's layer 3 and familiar, use it. We only have one hard requirement - fast Layer 3 switching.

With routing protocols, use what you know - if a protocol is unfamiliar, you'll have a difficult time supporting it. There's nothing wrong with running OSPF (or even RIP!) for these purposes. My personal favorite is actually running two - either IS-IS or OSPF combined with BGP - but this is driven by a few requirements I have for the future:

NSX-T only supports BGP
BGP is the way to go for highly scalable deployments
Any carrier network engineer will feel at home using it

This concludes this part on Clos networking. Later, I might even apply it!

Spine and Leaf Practical Applications, EGP and IGP combined!

Sat, 22 Jun 2019 14:06:00 -0800

So far, all examples to date have been extremely simple, and stand rather well on their own (OSPF Clos would work just FINE in a campus network if you have cheap L3) but may not effectively address some more advanced use cases.

In short, we're about to enter niche territory.

Generally speaking, BGP is regarded as the most advanced networking topic out there, until it isn't. Most of the complexity lies in iBGP (same Autonomous System everywhere), because BGP's primary loop prevention mechanism is AS-Path (count on the number of AS this route transits).

To successfully implement iBGP (a staple of every carrier, so totally do-able), a network engineer must choose one of the following paths (non-inclusive list):

Fully mesh all BGP speakers with each other (factorial scaling)
Implement a Route Reflector: https://en.wikipedia.org/wiki/Route_reflector

In this case, the more scalable option is to opt for a route reflector, but it isn't that easy.

iBGP, when compared to eBGP:

Doesn't care about hop count to a peer speaker. As long as the route reflector is less than 255 hops away, there is no issue
Usually doesn't care about resolving paths to peer speakers - that's a problem for something else.

To provide a good example of an expandable, scalable fabric that can offer eBGP as a service to subtending network devices, we will implement IS-IS as the intra-fabric routing protocol, and then leverage iBGP with the spine switches as route reflectors.

First things first, diagram is here: (YAML)

First, we'd configure the spines. Note that future releases (my home lab is rockin' IOS 12.2.55, too old for this) the BGP Dynamic Neighbor that comes with more modern network operating systems is really useful.

Note: route reflector client status is configured on the SERVER side:

 1router bgp 65000  
 2 bgp log-neighbor-changes  
 3 neighbor 10.6.0.0 remote-as 65000  
 4 neighbor 10.6.0.0 update-source L0  
 5 neighbor 10.6.0.0 route-reflector-client  
 6 neighbor 10.6.0.1 remote-as 65000  
 7 neighbor 10.6.0.1 update-source L0  
 8 neighbor 10.6.0.1 route-reflector-client  
 9 neighbor FD00:6::0 remote-as 65000  
10 neighbor FD00:6::0 update-source L0  
11 neighbor FD00:6::0 route-reflector-client  
12 neighbor FD00:6::1 remote-as 65000  
13 neighbor FD00:6::1 update-source L0  
14 neighbor FD00:6::1 route-reflector-client  
15 maximum-paths 2  
16 !  
17 address-family ipv4  
18  neighbor 10.6.0.0 activate  
19  neighbor 10.6.0.1 activate  
20  maximum-paths 2  
21  no auto-summary  
22  network 10.6.0.240 mask 255.255.255.254  
23  network 10.6.240.0 mask 255.255.255.254  
24  network 10.6.240.2 mask 255.255.255.254  
25 exit-address-family  
26 !  
27 address-family ipv6  
28  neighbor FD00:6::0 activate  
29  neighbor FD00:6::1 activate  
30  network FD00:6::240/127  
31  network FD00:6:240::/126  
32  network FD00:6:240::4/126  
33  no synchronization  
34  maximum-paths 2  
35 exit-address-family  
36!

And then the leaf configuration:

 1  
 2router bgp 65000  
 3 bgp log-neighbor-changes  
 4 neighbor 10.6.0.240 remote-as 65000  
 5 neighbor 10.6.0.240 update-source L0  
 6 neighbor 10.6.0.241 remote-as 65000  
 7 neighbor 10.6.0.241 update-source L0  
 8 neighbor FD00:6::240 remote-as 65000  
 9 neighbor FD00:6::240 update-source L0  
10 neighbor FD00:6::241 remote-as 65000  
11 neighbor FD00:6::241 update-source L0  
12 maximum-paths 2  
13 !  
14 address-family ipv4  
15  neighbor 10.6.0.240 activate  
16  neighbor 10.6.0.241 activate  
17  maximum-paths 2  
18  no auto-summary  
19  network 10.6.0.240 mask 255.255.255.254  
20  network 10.6.240.0 mask 255.255.255.254  
21  network 10.6.240.2 mask 255.255.255.254  
22 exit-address-family  
23 !  
24 address-family ipv6  
25  neighbor FD00:6::240 activate  
26  neighbor FD00:6::241 activate  
27  network FD00:6::240/127  
28  network FD00:6:240::/126  
29  network FD00:6:240::4/126  
30  no synchronization  
31  maximum-paths 2  
32 exit-address-family  
33!

Note that no BGP peers are up yet - and BGP knows what the problem is, too!

 1bgp-rr0-s0#show ip bgp  
 206:12:15: %SYS-5-CONFIG_I: Configured from console by consoleneigh  
 3BGP neighbor is 10.6.0.0,  remote AS 65000, internal link  
 4  BGP version 4, remote router ID 0.0.0.0  
 5  BGP state = Active  
 6  Last read 00:03:02, last write 00:03:02, hold time is 180, keepalive interval is 60 seconds  
 7  Message statistics:  
 8    InQ depth is 0  
 9    OutQ depth is 0  
10                         Sent       Rcvd  
11    Opens:                  0          0  
12    Notifications:          0          0  
13    Updates:                0          0  
14    Keepalives:             0          0  
15    Route Refresh:          0          0  
16    Total:                  0          0  
17  Default minimum time between advertisement runs is 0 seconds  
18  
19 For address family: IPv4 Unicast  
20  BGP table version 1, neighbor version 0/0  
21  Output queue size : 0  
22  Index 1, Offset 0, Mask 0x2  
23  Route-Reflector Client  
24  1 update-group member  
25                                 Sent       Rcvd  
26  Prefix activity:               ----       ----  
27    Prefixes Current:               0          0  
28    Prefixes Total:                 0          0  
29    Implicit Withdraw:              0          0  
30    Explicit Withdraw:              0          0  
31    Used as bestpath:             n/a          0  
32    Used as multipath:            n/a          0  
33  
34                                   Outbound    Inbound  
35  Local Policy Denied Prefixes:    --------    -------  
36    Total:                                0          0  
37  Number of NLRIs in the update sent: max 0, min 0  
38  
39  Address tracking is enabled, the RIB does not have a route to 10.6.0.0  
40  Address tracking requires at least a /0 route to the peer  
41  Connections established 0; dropped 0  
42  Last reset never  
43  Transport(tcp) path-mtu-discovery is enabled  
44  No active TCP connection

Note how it says the RIB does not have a route to 10.6.0.0 - that's because iBGP doesn't resolve next-hops for us. Let's fix it by rolling out an Interior Gateway Protocol (IGP) to support iBGP here. I'm using IS-IS for a few reasons - namely:

Like BGP, one routing protocol for both IPv4 and IPv6
Selective flooding with ISPF
I'm too hip for OSPF now

 1router isis CLOS-1  
 2 net 42.0000.0000.0000.0240.00  
 3 is-type level-2-only  
 4 ispf level-2  
 5 log-adjacency-changes  
 6!  
 7interface   
 8ip router isis CLOS-1  
 9interface Loopback0  
10ip router isis CLOS-1

This is applied to every router, while changing the net-ID for each device.It's fun watching the adjacencies pop up, so I'll add that here too.

1  
2*Mar  1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0240 (FastEthernet0/22) Up, new adjacency  
3*Mar  1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0241 (FastEthernet0/23) Up, new adjacency   
4*Mar  1 06:31:41.565: %BGP-5-ADJCHANGE: neighbor 10.6.0.240 Up  
5*Mar  1 06:31:47.169: %BGP-5-ADJCHANGE: neighbor 10.6.0.241 Up

Note how BGP pops up immediately after IS-IS resolves the next-hop for the loopback in this case. Sadly, it doesn't look like my ancient lab switches support IS-IS for IPv6 - so I'll add OSPFv3 for the next topic- actually using Clos in a datacenter network

 1bgp-rr0-s0#show ip bgp sum  
 2BGP router identifier 10.6.0.240, local AS number 65000  
 3BGP table version is 3, main routing table version 3  
 42 network entries using 234 bytes of memory  
 54 path entries using 208 bytes of memory  
 63/1 BGP path/bestpath attribute entries using 420 bytes of memory  
 70 BGP route-map cache entries using 0 bytes of memory  
 80 BGP filter-list cache entries using 0 bytes of memory  
 9BGP using 862 total bytes of memory  
10BGP activity 4/0 prefixes, 6/0 paths, scan interval 60 secs  
11  
12Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
1310.6.0.0        4 65000       8       7        3    0    0 00:03:03        1  
1410.6.0.1        4 65000       6       5        3    0    0 00:01:51        1  
15FD00:6::        4 65000       0       0        0    0    0 never    Active  
16FD00:6::1       4 65000       0       0        0    0    0 never    Active

Configurations generated by this lab, if you want to replicate are here.

Spine and Leaf Practical Applications, eBGP

Sat, 08 Jun 2019 10:50:00 -0800

Overview

First off, here's the reference diagram (YAML):

Assumptions about difficulty

Most people I've met outside of the carrier space are pretty intimidated by BGP, as it is truly impressive in scope. Here, we're going to break-out BGP usage into two categories:

iBGP: This is where all nodes have the same Autonomous system number. A great deal of complexity exists with this deployment model, because BGP's primary loop prevention mechanism is a string with all of the autonomous system numbers to that route, counting each entry as a "hop" as it were.
eBGP: Every single device has its own ASN. Loops are easy to prevent by simply reading the AS-Path.

eBGP is not very difficult to learn.

This is worthwhile, because BGP has a pretty substantial strength within data center networks, and that is an emphasis on reliability.

I'm not going to be doing a deep-dive on BGP here - but can recommend some truly excellent resources on the subject:

BGP: Building Reliable Networks with the Border Gateway Protocol, by Iljitsch van Beijnum
Pretty much anything written by Narbik Kocharians

How is BGP different from IGPs like OSPF, EIGRP?

First, we must examine some key differences between BGP and IGPs:

IGPs are multicast-based, and dynamically generate peers. BGP is TCP-based and needs statically defined peers (note: you can define a dynamic range, which in a future example will be truly valuable)
EIGRP has one area, OSPF generally supports up to 16 without getting specific hardware. BGP supports 65,536 with 2-byte ASNs, or 4,294,967,295
IGPs are designed to trust their routing protocol peers to prevent loops, while BGP is designed to control route advertisement
IGPs (other than IS-IS, of course) only support IP-based address families, while MP-BGP can support any number of units defined as "Network Layer Reachability Information," making it extensible in numerous ways like EVPN or Segment Routing, or even MPLS. The key thematic point here is that BGP behaves more like a distributed database than a routing protocol would normally.
IGPs value fast reconvergence, while BGP values reliable reconvergence. It's slow moving, but is extremely change-friendly.

Applying Concepts

In a controlled environment, like a Clos fabric, eBGP is pretty easy to setup, troubleshoot, and maintain. So let's get started!

First, we configure the spines with the appropriate AS and neighbors. It looks like there's a lot going on here, but that's simply because we're running two address-families: IPv4 and IPv6:

 1bgp-as65000-s0#conf t  
 2Enter configuration commands, one per line.  End with CNTL/Z.  
 3bgp-as65000-s0(config)#  
 4router bgp 65000  
 5 bgp log-neighbor-changes  
 6 neighbor 10.6.240.1 remote-as 64900  
 7 neighbor 10.6.240.1 update-source FastEthernet0/24  
 8 neighbor 10.6.240.3 remote-as 64901  
 9 neighbor 10.6.240.3 update-source FastEthernet0/22  
10 neighbor FD00:6:240::2 remote-as 64900  
11 neighbor FD00:6:240::2 update-source FastEthernet0/24  
12 neighbor FD00:6:240::6 remote-as 64901  
13 neighbor FD00:6:240::6 update-source FastEthernet0/22  
14 maximum-paths 2  
15 !  
16 address-family ipv4  
17  neighbor 10.6.240.1 activate  
18  neighbor 10.6.240.3 activate  
19  no neighbor FD00:6:240::2 activate  
20  no neighbor FD00:6:240::6 activate  
21  maximum-paths 2  
22  no auto-summary  
23  no synchronization  
24 exit-address-family  
25 !  
26 address-family ipv6  
27  neighbor FD00:6:240::2 activate  
28  neighbor FD00:6:240::6 activate  
29 exit-address-family  
30  
31bgp-as65001-s1#conf t  
32Enter configuration commands, one per line.  End with CNTL/Z.  
33bgp-as65001-s1(config)#  
34router bgp 65001  
35 bgp log-neighbor-changes  
36 neighbor 10.6.241.1 remote-as 64900  
37 neighbor 10.6.241.1 update-source FastEthernet0/21  
38 neighbor 10.6.241.3 remote-as 64901  
39 neighbor 10.6.241.3 update-source FastEthernet0/23  
40 neighbor FD00:6:241::2 remote-as 64900  
41 neighbor FD00:6:241::2 update-source FastEthernet0/21  
42 neighbor FD00:6:241::6 remote-as 64901  
43 neighbor FD00:6:241::6 update-source FastEthernet0/23  
44 maximum-paths 2  
45 !  
46 address-family ipv4  
47  neighbor 10.6.241.1 activate  
48  neighbor 10.6.241.3 activate  
49  no neighbor FD00:6:241::2 activate  
50  no neighbor FD00:6:241::6 activate  
51  maximum-paths 2  
52  no auto-summary  
53  no synchronization  
54 exit-address-family  
55 !  
56 address-family ipv6  
57  neighbor FD00:6:241::2 activate  
58  neighbor FD00:6:241::6 activate  
59 exit-address-family

And then the leafs:

 1  
 2bgp-as64900-l0#conf t  
 3Enter configuration commands, one per line.  End with CNTL/Z.  
 4bgp-as64900-l0(config)#  
 5router bgp 64900  
 6 bgp log-neighbor-changes  
 7 neighbor 10.6.240.0 remote-as 65000  
 8 neighbor 10.6.240.0 update-source FastEthernet1/0/24  
 9 neighbor 10.6.241.0 remote-as 65001  
10 neighbor 10.6.241.0 update-source FastEthernet1/0/21  
11 neighbor FD00:6:240::1 remote-as 65000  
12 neighbor FD00:6:240::1 update-source FastEthernet1/0/24  
13 neighbor FD00:6:241::1 remote-as 65001  
14 neighbor FD00:6:241::1 update-source FastEthernet1/0/21  
15 maximum-paths 2  
16 !  
17 address-family ipv4  
18  neighbor 10.6.240.0 activate  
19  neighbor 10.6.241.0 activate  
20  no neighbor FD00:6:240::1 activate  
21  no neighbor FD00:6:241::1 activate  
22  maximum-paths 2  
23  no auto-summary  
24  no synchronization  
25 exit-address-family  
26 !  
27 address-family ipv6  
28  neighbor FD00:6:240::1 activate  
29  neighbor FD00:6:241::1 activate  
30 exit-address-family  
31  
32bgp-as64901-l1#conf t  
33Enter configuration commands, one per line.  End with CNTL/Z.  
34bgp-as64901-l1(config)#  
35router bgp 64901  
36 bgp log-neighbor-changes  
37 neighbor 10.6.240.2 remote-as 65000  
38 neighbor 10.6.240.2 update-source FastEthernet0/22  
39 neighbor 10.6.241.2 remote-as 65001  
40 neighbor 10.6.241.2 update-source FastEthernet0/23  
41 neighbor FD00:6:240::5 remote-as 65000  
42 neighbor FD00:6:240::5 update-source FastEthernet0/22  
43 neighbor FD00:6:241::5 remote-as 65001  
44 neighbor FD00:6:241::5 update-source FastEthernet0/23  
45 maximum-paths 2  
46 !  
47 address-family ipv4  
48  neighbor 10.6.240.2 activate  
49  neighbor 10.6.241.2 activate  
50  no neighbor FD00:6:240::5 activate  
51  no neighbor FD00:6:241::5 activate  
52  maximum-paths 2  
53  no auto-summary  
54  no synchronization  
55 exit-address-family  
56 !  
57 address-family ipv6  
58  neighbor FD00:6:240::5 activate  
59  neighbor FD00:6:241::5 activate  
60 exit-address-family

We can now verify that all peers are up with both stacks:

 1bgp-as65000-s0#show ip bgp sum  
 2BGP router identifier 10.6.0.240, local AS number 65000  
 3BGP table version is 1, main routing table version 1  
 4  
 5Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
 610.6.240.1      4 64900      23      23        1    0    0 00:20:39        0  
 710.6.240.3      4 64901      19      18        1    0    0 00:17:04        0  
 8bgp-as65000-s0#show bgp ipv6 unicast summary  
 9BGP router identifier 10.6.0.240, local AS number 65000  
10BGP table version is 1, main routing table version 1  
11  
12Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
13FD00:6:240::2   4 64900      13      12        1    0    0 00:10:17        0  
14FD00:6:240::6   4 64901       9       9        1    0    0 00:06:30        0

We do still have a problem - there are no prefixes received! Let's fix that by adding network statements to all relevant devices. In the demo equipment, the network statement must be an exact match to advertise.
Network statements are not required for interfaces, as in this case, multicast is not used for peer discovery:

1bgp-as64900-l0(config)#router bgp 64900  
2bgp-as64900-l0(config-router)#address-family ipv4  
3bgp-as64900-l0(config-router-af)#network 10.6.0.0 mask 255.255.255.255

After this is completed, we'll see more routes - note that the above step must be repeated on the spines for all applicable networks, to ensure end to end reachability. This hardware does not appear to support ECMP for IPv6.

 1bgp-as64900-l0#show ip bgp sum  
 2BGP router identifier 10.6.0.0, local AS number 64900  
 3BGP table version is 13, main routing table version 13  
 48 network entries using 936 bytes of memory  
 59 path entries using 468 bytes of memory  
 68/4 BGP path/bestpath attribute entries using 1120 bytes of memory  
 76 BGP AS-PATH entries using 144 bytes of memory  
 80 BGP route-map cache entries using 0 bytes of memory  
 90 BGP filter-list cache entries using 0 bytes of memory  
10BGP using 2668 total bytes of memory  
11BGP activity 16/0 prefixes, 22/1 paths, scan interval 60 secs  
12  
13Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
1410.6.240.0      4 65000      47      47       13    0    0 00:40:51        4  
1510.6.241.0      4 65001      47      45       13    0    0 00:40:03        4  
16  
17bgp-as64900-l0#show bgp ipv6 unicast summary  
18BGP router identifier 10.6.0.0, local AS number 64900  
19BGP table version is 10, main routing table version 10  
208 network entries using 1128 bytes of memory  
2112 path entries using 912 bytes of memory  
228/4 BGP path/bestpath attribute entries using 1120 bytes of memory  
236 BGP AS-PATH entries using 144 bytes of memory  
240 BGP route-map cache entries using 0 bytes of memory  
250 BGP filter-list cache entries using 0 bytes of memory  
26BGP using 3304 total bytes of memory  
27BGP activity 16/0 prefixes, 22/1 paths, scan interval 60 secs  
28  
29Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
30FD00:6:240::1   4 65000      44      43       10    0    0 00:37:56        5  
31FD00:6:241::1   4 65001      43      43       10    0    0 00:37:18        6  
32  
33  
34bgp-as64900-l0#show ipv6 ro  
35IPv6 Routing Table - Default - 11 entries  
36Codes: C - Connected, L - Local, S - Static, U - Per-user Static route  
37       B - BGP, R - RIP, D - EIGRP, EX - EIGRP external  
38       ND - Neighbor Discovery  
39       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2  
40       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2  
41LC  FD00:6::/128 [0/0]  
42     via Loopback0, receive  
43B   FD00:6::1/128 [20/0]  
44     via FE80::216:C8FF:FE04:4742, FastEthernet1/0/24  
45B   FD00:6::240/128 [20/0]  
46     via FE80::216:C8FF:FE04:4742, FastEthernet1/0/24  
47B   FD00:6::241/128 [20/0]  
48     via FE80::223:4FF:FE42:F3C1, FastEthernet1/0/21  
49C   FD00:6:240::/126 [0/0]  
50     via FastEthernet1/0/24, directly connected  
51L   FD00:6:240::2/128 [0/0]  
52     via FastEthernet1/0/24, receive  
53B   FD00:6:240::4/126 [20/0]  
54     via FE80::216:C8FF:FE04:4742, FastEthernet1/0/24  
55C   FD00:6:241::/126 [0/0]  
56     via FastEthernet1/0/21, directly connected  
57L   FD00:6:241::2/128 [0/0]  
58     via FastEthernet1/0/21, receive  
59B   FD00:6:241::4/126 [20/0]  
60     via FE80::223:4FF:FE42:F3C1, FastEthernet1/0/21  
61L   FF00::/8 [0/0]  
62     via Null0, receive  
63bgp-as64900-l0#show ip ro  
64Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP  
65       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area  
66       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2  
67       E1 - OSPF external type 1, E2 - OSPF external type 2  
68       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2  
69       ia - IS-IS inter area, * - candidate default, U - per-user static route  
70       o - ODR, P - periodic downloaded static route  
71  
72Gateway of last resort is not set  
73  
74     10.0.0.0/8 is variably subnetted, 8 subnets, 2 masks  
75C       10.6.0.0/32 is directly connected, Loopback0  
76B       10.6.0.1/32 [20/0] via 10.6.240.0, 00:12:49  
77C       10.6.240.0/31 is directly connected, FastEthernet1/0/24  
78B       10.6.0.240/32 [20/0] via 10.6.240.0, 00:10:03  
79C       10.6.241.0/31 is directly connected, FastEthernet1/0/21  
80B       10.6.0.241/32 [20/0] via 10.6.241.0, 00:07:40  
81B       10.6.240.2/31 [20/0] via 10.6.240.0, 00:08:47  
82B       10.6.241.2/31 [20/0] via 10.6.241.0, 00:07:40

I have posted the base configs here.

Switching Lab Topology Diagram

Sun, 02 Jun 2019 11:35:00 -0800

Here's the example topology used in the Spine-and-Leaf labs: (YAML)

Spine and Leaf Practical Applications, OSPF

Sun, 02 Jun 2019 10:51:00 -0800

As covered in the previous post, base configuration of a spine-and-leaf fabric is actually pretty simple. This will be pretty short, but we'll cover the conversion of the previously built fabric to OSPF.

Here's the updated diagram: (YAML). As we move to a more full-fledged implementation, we'll do dual-stack.

The cleanup for this is as follows:

1no router rip

From here, we can configure the router statements on all devices. It can be the same for all, because of the summarization performed while planning out the network.

1router ospf 1  
2 ispf  
3 log-adjacency-changes  
4 nsf cisco  
5 network 10.6.0.0 0.0.0.255 area 0  
6 network 10.6.240.0 0.0.1.255 area 0

In a production environment you should add passive-interface default on the leafs if the ToR does not peer dynamic routing with anything sub-tending it.

Unsurprisingly, this just works. Now, to setup IPv6!

1ospf-s0#conf t  
2Enter configuration commands, one per line.  End with CNTL/Z.  
3ospf-s0(config)#ipv6?  
4% Unrecognized command

Well, it looks like IPv6 is not available until IOS 12.2.55. Let's use this network to upgrade it, by hooking up a TFTP server to leaf-1:

1  
2interface FastEthernet0/14  
3 no switchport  
4 ip address 10.66.0.1 255.255.255.0  
5!  
6router ospf 1  
7 network 10.66.0.1 0.0.0.0 area 0

We test reachability from the other leaf - this is a fully layer 3 switched path:

1ospf-l0#ping 10.66.0.180  
2  
3Type escape sequence to abort.  
4Sending 5, 100-byte ICMP Echos to 10.66.0.180, timeout is 2 seconds:  
5!!!!!  
6Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

And then we copy it over TFTP:

1ospf-s1#copy tftp flash:  
2Address or name of remote host []? 10.66.0.180  
3Source filename []? c3560-ipservicesk9-mz.122-55.SE6.bin  
4Destination filename [c3560-ipservicesk9-mz.122-55.SE6.bin]?  
5Accessing tftp://10.66.0.180/c3560-ipservicesk9-mz.122-55.SE6.bin...  
6Loading c3560-ipservicesk9-mz.
7[OK - 12752912 bytes]  
8  
912752912 bytes copied in 201.133 secs (63405 bytes/sec)

Note that this, while a practical application, is still non-redundant.

UPGRADING INTENSIFIES
Now to implement IPv6 as follows: (YAML)

Note: We used ; instead of : due to a feature issue with drawthe.net. We're using /126 prefixes because this is on older equipment, which may not support /127 prefixes reliably.
On all devices, we need to enable ipv6 routing / OSPFv3:

1ipv6 unicast-routing  
2ipv6 router ospf 2  
3 log-adjacency-changes

We then configure each device:

 1ospf-l0# configure terminal  
 2interface Loopback0  
 3 ip address 10.6.0.0 255.255.255.255  
 4 ipv6 address FD00:6::/128  
 5 ipv6 ospf 2 area 0  
 6interface FastEthernet1/0/21  
 7 no switchport  
 8 ip address 10.6.241.1 255.255.255.254  
 9 ipv6 address FD00:6:241::2/126  
10 ipv6 enable  
11 ipv6 ospf 2 area 0  
12interface FastEthernet1/0/24  
13 no switchport  
14 ip address 10.6.240.1 255.255.255.254  
15 ipv6 address FD00:6:240::2/126  
16 ipv6 enable  
17 ipv6 ospf 2 area 0  
18  
19ospf-l1# configure terminal  
20interface Loopback0  
21 ip address 10.6.0.1 255.255.255.255  
22 ipv6 address FD00:6::1/128  
23 ipv6 ospf 2 area 0  
24interface FastEthernet0/22  
25 no switchport  
26 ip address 10.6.240.3 255.255.255.254  
27 ipv6 address FD00:6:240::6/126  
28 ipv6 enable  
29 ipv6 ospf 2 area 0  
30interface FastEthernet0/23  
31 no switchport  
32 ip address 10.6.241.3 255.255.255.254  
33 ipv6 address FD00:6:241::6/126  
34 ipv6 enable  
35 ipv6 ospf 2 area 0  
36  
37ospf-s0# configure terminal  
38interface Loopback0  
39 ip address 10.6.0.240 255.255.255.255  
40 ipv6 address FD00:6::240/128  
41 ipv6 ospf 2 area 0  
42interface FastEthernet0/22  
43 no switchport  
44 ip address 10.6.240.2 255.255.255.254  
45 ipv6 address FD00:6:241::1/126  
46 ipv6 enable  
47 ipv6 ospf 2 area 0  
48interface FastEthernet0/24  
49 no switchport  
50 ip address 10.6.240.0 255.255.255.254  
51 ipv6 address FD00:6:240::1/126  
52 ipv6 enable  
53 ipv6 ospf 2 area 0  
54  
55ospf-s1# configure terminal  
56interface Loopback0  
57 ip address 10.6.0.241 255.255.255.255  
58 ipv6 address FD00:6::241/128  
59 ipv6 ospf 2 area 0  
60interface FastEthernet0/21  
61 no switchport  
62 ip address 10.6.241.0 255.255.255.254  
63 ipv6 address FD00:6:241::1/126  
64 ipv6 enable  
65 ipv6 ospf 2 area 0  
66interface FastEthernet0/23  
67 no switchport  
68 ip address 10.6.241.2 255.255.255.254  
69 ipv6 address FD00:6:241::5/126  
70 ipv6 enable  
71 ipv6 ospf 2 area 0

From here, we test by initiating traffic from a subtending network on Leaf-1 to Leaf-0, and checking the routing tables:

 1ospf-l1#ping ipv6 fd00:6::  
 2  
 3Type escape sequence to abort.  
 4Sending 5, 100-byte ICMP Echos to FD00:6::, timeout is 2 seconds:  
 5!!!!!  
 6Success rate is 100 percent (5/5), round-trip min/avg/max = 0/1/8 ms  
 7ospf-l1#show ip route  
 8Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP  
 9       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area  
10       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2  
11       E1 - OSPF external type 1, E2 - OSPF external type 2  
12       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2  
13       ia - IS-IS inter area, * - candidate default, U - per-user static route  
14       o - ODR, P - periodic downloaded static route  
15  
16Gateway of last resort is not set  
17  
18     10.0.0.0/8 is variably subnetted, 8 subnets, 2 masks  
19O       10.6.0.0/32 [110/3] via 10.6.241.2, 00:03:58, FastEthernet0/23  
20                    [110/3] via 10.6.240.2, 00:03:58, FastEthernet0/22  
21C       10.6.0.1/32 is directly connected, Loopback0  
22O       10.6.240.0/31 [110/2] via 10.6.240.2, 00:03:58, FastEthernet0/22  
23O       10.6.0.240/32 [110/2] via 10.6.240.2, 00:03:58, FastEthernet0/22  
24O       10.6.241.0/31 [110/2] via 10.6.241.2, 00:03:58, FastEthernet0/23  
25O       10.6.0.241/32 [110/2] via 10.6.241.2, 00:03:58, FastEthernet0/23  
26C       10.6.240.2/31 is directly connected, FastEthernet0/22  
27C       10.6.241.2/31 is directly connected, FastEthernet0/23  
28ospf-l1#show ipv6 route  
29IPv6 Routing Table - Default - 11 entries  
30Codes: C - Connected, L - Local, S - Static, U - Per-user Static route  
31       B - BGP, R - RIP, D - EIGRP, EX - EIGRP external  
32       ND - Neighbor Discovery  
33       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2  
34       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2  
35O   FD00:6::/128 [110/2]  
36     via FE80::216:C8FF:FE04:4741, FastEthernet0/22  
37     via FE80::223:4FF:FE42:F3C2, FastEthernet0/23  
38LC  FD00:6::1/128 [0/0]  
39     via Loopback0, receive  
40O   FD00:6::240/128 [110/1]  
41     via FE80::216:C8FF:FE04:4741, FastEthernet0/22  
42O   FD00:6::241/128 [110/1]  
43     via FE80::223:4FF:FE42:F3C2, FastEthernet0/23  
44O   FD00:6:240::/126 [110/2]  
45     via FE80::216:C8FF:FE04:4741, FastEthernet0/22  
46C   FD00:6:240::4/126 [0/0]  
47     via FastEthernet0/22, directly connected  
48L   FD00:6:240::6/128 [0/0]  
49     via FastEthernet0/22, receive  
50O   FD00:6:241::/126 [110/1]  
51     via FastEthernet0/22, directly connected  
52C   FD00:6:241::4/126 [0/0]  
53     via FastEthernet0/23, directly connected  
54L   FD00:6:241::6/128 [0/0]  
55     via FastEthernet0/23, receive  
56L   FF00::/8 [0/0]  
57     via Null0, receive

Note: technically we don't have to number the leaf-spine-leaf links in IPv6 with OSPFv3/RIP-ng/EIGRP, but that is a personal preference of mine to keep it consistent with future designs, and to allow for ease of troubleshooting.

As always, example configurations are here.

Spine and Leaf Practical Applications, RIPv2

Sat, 01 Jun 2019 16:38:00 -0800

This is only slightly trolling, but is primarily to outline the topological simplicity of Spine-and-Leaf networking, in a way that is suspiciously similar to Cisco classes.

First things first, here's the diagram. This is performed using a set of four Cat3560s, enterprise licensed and wired in a redundant square topology to simulate a wide variety of topologies with minimal modification. At some point I'll post this setup as well, it was recommended in the book CCIE Routing and Switching v5.1, Bridging the Gap Between CCNP and CCIE
YAML Link

So this is actually pretty simple - as everything shouldbe Layer 3. We begin by configuring the Spines:

 1hostname rip-s0  
 2interface Loopback0  
 3 ip address 10.6.0.240 255.255.255.255  
 4interface FastEthernet0/22  
 5 no switchport  
 6 ip address 10.6.240.2 255.255.255.254  
 7interface FastEthernet0/24  
 8 no switchport  
 9 ip address 10.6.240.0 255.255.255.254  
10  
11hostname rip-s1  
12interface Loopback0  
13 ip address 10.6.0.241 255.255.255.255  
14interface FastEthernet0/21  
15 no switchport  
16 ip address 10.6.241.0 255.255.255.254  
17interface FastEthernet0/23  
18 no switchport  
19 ip address 10.6.241.2 255.255.255.254

Some explanation here:

We're using /31s to save address space as leaf-spine-leaf links are numerous and chew through address space like no tomorrow. If you'd like to know more about /31 usage, it's here.
I focused on IP Address Management (IPAM) before the actual network design, assigning pre-planned prefixes. In this example, each switch has a virtual number, making it easy to pre-provision and organize network topologies for scale. Remember, this is all to handle frequent loop-free changes at scale - this is important!
- S0: 240 (10.6.240.x/31, 10.6.0.240)
- S1: 241 (10.6.241.x/31, 10.6.0.241)
- L0: 0 (10.6.0.0)
- L1: 1 (10.6.0.1)
No switchport forces ports into Layer 3 mode.

And then the Leafs:

 1hostname rip-l0  
 2interface Loopback0  
 3 ip address 10.6.0.0 255.255.255.255  
 4interface FastEthernet1/0/21  
 5 no switchport  
 6 ip address 10.6.241.1 255.255.255.254  
 7interface FastEthernet1/0/24  
 8 no switchport  
 9 ip address 10.6.240.1 255.255.255.254  
10  
11hostname rip-l1  
12 interface Loopback0  
13 ip address 10.6.0.1 255.255.255.255  
14interface FastEthernet0/22  
15 no switchport  
16 ip address 10.6.240.3 255.255.255.254  
17interface FastEthernet0/23  
18 no switchport  
19 ip address 10.6.241.3 255.255.255.254

Normally, you'd add interconnection on these devices, but loopbacks suffice for this example.
This doesn't support routing but is a functional base configuration - so let's turn on routing (all switches):

1ip routing  
2router rip  
3 version 2  
4 network 10.0.0.0  
5 no auto-summary

Poof! It's working!

 1rip-l0#show ip route  
 2Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP  
 3   D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area  
 4   N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2  
 5   E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP  
 6   i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2  
 7   ia - IS-IS inter area, * - candidate default, U - per-user static route  
 8   o - ODR, P - periodic downloaded static route  
 9Gateway of last resort is not set  
10  10.0.0.0/8 is variably subnetted, 8 subnets, 2 masks  
11C   10.6.0.0/32 is directly connected, Loopback0  
12R   10.6.0.1/32 [120/2] via 10.6.241.0, 00:00:06, FastEthernet1/0/21  
13          [120/2] via 10.6.240.0, 00:00:06, FastEthernet1/0/24  
14C   10.6.240.0/31 is directly connected, FastEthernet1/0/24  
15R   10.6.0.240/32 [120/1] via 10.6.240.0, 00:00:12, FastEthernet1/0/24  
16C   10.6.241.0/31 is directly connected, FastEthernet1/0/21  
17R   10.6.0.241/32 [120/1] via 10.6.241.0, 00:00:17, FastEthernet1/0/21  
18R   10.6.240.2/31 [120/1] via 10.6.240.0, 00:00:13, FastEthernet1/0/24  
19R   10.6.241.2/31 [120/1] via 10.6.241.0, 00:00:17, FastEthernet1/0/21

Oddly enough, RIPv2 isn't supposed to support ECMP, but appears to be doing so here.
Hopefully, this was enlightening - because in this case, this topology is incredibly simple when involving an IGP. There are a few downsides to RIP deployed in this manner:

It's chatty and floods all the time, so all changes (network additions) will cause a reconvergence.
Link-state failure won't trigger a path re-route
It's RIP.

Configurations generated are here, for anyone who would want to experiment with them.

Spine and Leaf Networks, a Topology

Mon, 27 May 2019 10:28:00 -0800

Before we get too far - this is going to be a simple topology, relatively speaking. I'll be putting the switch configs on my GitHub account as well when I get to that point. The diagram is as follows:

This was drawn programmatically with drawthe.net. File is here

Spine and Leaf Networks, an Introduction

Sun, 21 Apr 2019 14:54:00 -0800

The two biggest problems a network engineer will face are changes and loops

In our Cisco exams, an overwhelming majority of the educational content is on methods of preventing loops, and for good reason. Unfortunately, the methods we use to prevent loops are highly complex - and if we partially understand these concepts, the risk is then transferred to changes, where we modify loop prevention mechanisms for a variety of reasons in the datacenter:

Adding new servers
Adding new networks to accommodate workloads
Adding new hardware to accommodate the addition of new servers and workloads
Adding new interconnects because the previous operations were so successful that new sites are now required

I'm sure many have said, "Make it Layer 3!" as if that's some form of easy fix that will magically remote datacenter reliability issues. The reality is that improperly Layer 3 networks can be just as unstable as Layer 2 ones, if not more so. To make it worse, you may not be able to accommodate your workload needs, causing the business to fail.

First, let's cover what network designers mean by Layer 2 or 3, as it violates the OSI model. In short, it is a reference to port configuration and whether or not Layer 2 loop prevention mechanisms are in play.

Layer 2

Layer 2 network provisioning is probably the easiest to configure, and the least scalable. Most typical systems administrators won't have any issues deploying a workable small-scale Layer 2 network on their own - and probably have experience doing so.

Layer 2 network configuration involves the creation of a VLAN, which in turn instantiates a loop prevention process of some kind, like:

Per-VLAN Spanning Tree (PVST)
Per-VLAN Rapid Spanning Tree (PVRST)
Multiple Spanning Tree (MST) https://en.wikipedia.org/wiki/Multiple_Spanning_Tree_Protocol
FabricPath https://www.cisco.com/c/en/us/solutions/data-center-virtualization/fabricpath/index.html
TRILL https://en.wikipedia.org/wiki/TRILL_(computing)

Oddly enough, TRILL can actually be configured to conform to a Spine and Leaf spec. I won't discuss that here - I'll get into why later.

Layer 3

Layer 3 network provisioning is much less flexible, but can also be much more stable and scalable. In this case, Layer 2 loop prevention may be in play, such as with SVIs, but is not the primary or mandatory source of loop prevention. Instead, routing protocols and potentially redistribution are used, each with their own hazards:

RIP and split horizon https://en.wikipedia.org/wiki/Split_horizon_route_advertisement
OSPF, EIGRP, and redistribution problems (entire books, but also https://www.cisco.com/c/en/us/support/docs/ip/enhanced-interior-gateway-routing-protocol-eigrp/8606-redist.html)
eBGP and AS-Path problems https://www.juniper.net/documentation/en_US/junos/topics/usage-guidelines/policy-prepending-as-numbers-to-bgp-as-paths.html
iBGP and everything

Again, this is all just to prevent loops. Most network designs do a good job of preventing loops in the ways listed above but at the cost of making change riskier by tightly coupling networks to specific devices. As we know, tight coupling is a big negative with high change frequency.

The goals

To design a highly reliable, highly mutable, and highly maintainable network, a network designer must meet the following goals:

Prevent loops reliably and automatically
Allow for frequent, preferably automatic additions and removals of new networks
Be easy to maintain, fix and troubleshoot
Do all of the above, but with a minimum number of changes, to a minimum number of devices

Enter Spine and Leaf

Introduced in the 1950s by Charles Clos (details here), Clos networking is a mathematical model for non-blocking multistage circuits. This is a lot to unpack:

Non-Blocking: Nearly all Layer 2 loop prevention mechanisms will prevent loops by refusing to forward on secondary or n-scale paths. Telecommunication companies don't really like this, as this reduces available bandwidth (and therefore revenue) by half. Non-blocking indicates that all available ports are able to forward at all available speeds.
Multistage: Nearly every datacenter network has more than 6 network ports. As a result, we need the ability to scale beyond a single integrated circuit or network device.

Today, we have a few more technological advances than in the 1950s. Most datacenter network switches leverage Clos topologies to reduce manufacturing costs, increase reliability, by providing more ports on a switch than a single ASIC can provide by aggregating 4,6, or 8 port ASICs onto a crossbar.

This begs the question, why not layer 1? Since the switch itself is Clos, why not just buy a big switch and call it a day? There are some upsides here:

One IP to administer, making change easier
Layer 1 topologies are pretty reliable, and break-fix actions are typically just reseating something
Layer 2/3 loop prevention isn't required

But when we think about it a second, the downsides are pretty big:

Unless you use technology such as VSS, VCS, you have no redundancy
You must always perfectly assess the correct port count for your data center on the first try, leading to massive amounts of waste
It completely violates rule 4, because you're either changing the entire network or not at all.

Layer 2 Leaf Spine suffers from more or less the same issue but removes the need to always be completely correct with port-count.

Layer 3 Leaf-Spine (L3LS from here on out) leverages only Layer 3 loop prevention mechanisms between network devices - while built in a non-blocking Clos pattern in a 3-stage topology:

Odd looking, isn't it? Where are the connections between Spines, or behind leafs?

With Clos Networking, crossbars/spines should not connect to each other - it violates rule #4, and leads to blocking circuitry.

Now - this obviously removes all IP portability completely, and forces workloads running on-fabric to participate in routing on at least some level, because Leafs aren't aware of each other directly, but has certain reliability gains:

Imagine if you could do ASIC-level troubleshooting internal to a switch
Now imagine if you, as a network engineer, could do this without having to learn how to do ASIC-level troubleshooting. Instead, routing protocols that are familiar to you are your interface into the fabric
Now imagine that all failure domains are constrained to the individual ASIC you're working on and won't have higher repercussions to the switch

Pretty big upsides, right?

So here's where we need to diverge a bit, due to what I mentioned here. There are quite a few ways to deploy Spine-and-Leaf networks, and nearly all are highly reliable. Some are even used as production networks!

Humor set aside - the usability problem is a big one. My recommendation and the order of this series of blog posts would be to choose whatever platform, protocols, and administration methods best suit your organizational needs, 'cause they all work. Even RIP.

Before we move on, I'd like to cover some fairly serious problems I've seen when discussing the use of L3LS in the datacenter. I apologize for the length but there is a lot to cover here. The statements listed below are misconceptions that I've seen kill adoption of this technological principal.

L3LS is expensive: This is just flat out wrong. Generation 1 Catalyst 3560s with routing licensed can run it. All you need is layer 3 switching. While this is expensive in some cases, product selection can help a bit. Even older Layer 2-only datacenter switches cost quite a bit when compared to newer 10/25g switch options. If your department can afford new, unused 10 gigabit switches, L3LS probably won't cost more, if at all.
L3LS is a product: While some products like Big Switch, Cisco ACI, or Juniper's QFabric provide a pre-made, self-provisioning network solution that loosely conforms to these design principles, it's not particularly difficult to build your own if a canned solution meets your needs.
L3LS is difficult: We'll cover that in later posts, but it's mildly difficult to design, but easy to maintain and grow.
L3LS has to use {{ insert protocol here }}. Pretty much anything goes.

With that out of the way, let's have a bit of fun on the next one - running L3LS with RIPv2/3 as the designated routing protocol. In all cases the goal will be to provide a dual-stack network - IPv6 went final over 7 years ago. I'll be using CSR1000v and virtual NX-OS images for these examples, but your routing platform flavor of choice will work just fine. My point is that you know your platform, and should be able to map it out. This isn't stack overflow :)

Traditional Datacenter Network, a Preamble to Spine-and-Leaf

Sun, 21 Apr 2019 10:09:00 -0800

Datacenter Network Engineers have two problems:

CHANGE

LOOPS (let's cover this one later!)

Change is everywhere. Emerging trends such as DevOps and/or CI/CD have created the need for dynamic, ephemeral allocation of data center resources; not only in large-scale deployments, but medium-sized companies are starting down this direction as well.
...but we still have to schedule change windows to add/remove networks from our datacenters due to the risks involved with network changes.

Current State

Today, most datacenter network deployments consist of 2 or 3 layers, with a huge variety of opinions on the "core" or topside layer. I'll start from the bottom up, as this will generally cover the areas of primary focus first.

Please keep in mind that I'm not throwing shade on this type of design. It's *highly* reliable and is the backbone of many companies. If it's working well for you, you don't have to throw it away. In many cases, the possibilities I will discuss may not even feasible for you. Eventually, I'll have enough time to cover all the various aspects that result in successful data center networks - but for now, I am going to cover a topic that tends to have a great deal of misinformation and markitechture that confuses many network engineers.

Core-Aggregation-Access Topologies

This reference design consists of three tiers, Core, Aggregation, and Access:

Datacenter Access

This particular layer is where the rubber hits the road. Servers directly connect to the Access layer, and the design of the service is geared primarily toward facilitating the needs of the subtending servers. All kinds of atypical services are typically deployed at this point, such as MC-LAG (vPC, MEC, etc). Generally speaking, this should be where the most change frequency will occur, as it is the least.
Most deployments at this point are Layer 2, trunking server/workload VLANs so that workloads do not have to change addressing as they traverse different switches. This is a workload problem that for the majority of deployments have not been solved - and must be mitigated by this design.
There are a few downsides, however:

New network turn-ups involve all access-tier switches, and at a minimum, the aggregation layer. You're not mitigating risk with change if you have to modify pretty much every single device in your network!
Loop prevention methods must all be Layer 2, e.g. spanning tree, FabricPath, TRILL, MC-LAG. These loop prevention methods are not very resilient, and any failures will cascade through the entire data center in most cases.

I do have some recommendations when facing this problem:

When creating new networks, always explore the possibility of routed access. Adding SVIs to your access layer mitigates a great deal of this, but you lose workload portability. Perhaps not all workloads need portability, ex. storage over IP, host management.
Preconfigure all ports with a default configuration that will support new server turn-ups. Server administrators love being able to just plug their equipment in and have it work. Spend a lot of time planning this default port configuration with your systems team - it'll pay off.

Datacenter Aggregation

This is where most of the meat and potatoes are as far as data center networking, and for most deployments, this is as far as most designs go. This section of a data center network will be running tons of services, into one place they can be aggregated (thus the name). You'll typically see the following connected to / running on the access layer:

Firewalls
Load Balancers
Datacenter Interconnects, if there's no Core
Loop prevention methods such as MC-LAG
Layer 3 gateways for the majority of VLANs
All your VLANs are belong to the aggregation layer

The Aggregation Layer is probably the riskiest device in a data center network to modify. I recommend doing a few things to mitigate these risks:

Waste TONS of address space. Create lots of new networks, and keep them relatively small if you can (sub-/24). Deliver them to all of the access layers in a scalable manner, and preconfigure it all at the outset. Remember, no matter what capacity you allocate to, customers will overrun it.
Don't pile too much on the aggregation devices. You can connect a separate firewall, LB, etc to the aggregation layer, keeping these devices as simple as possible will ensure that administration work is as simple as possible.
Ensure you have an adequate port count. The move to adopting a data center core is an expensive one and is necessitated by the 3rd set of aggregation layer devices, typically.

Datacenter Core

This is the one where your VAR starts seeing dollar signs. Most deployments will not need this layer, even up when thousands of workloads (VMs, containers, I don't discriminate), as your port-count with an average Aggregation-Access network (we'll call them pods from now on) will be:

32-48 Aggregation ports
32-48 Access ports

You can dual-home 1,024-2,304 servers, or quad-home 512-1,152 servers on paper with one pod. Of course, most of these ports are wasted because you can't always fit 24-48 servers into a cabinet. Real-world maximum server count per pod would be in the hundreds.

The primary point where a data center network would expand to a network core would be when interconnecting 3 or more pods or physical locations. I do have recommendations on design here as well:

Don't budge on IP portability here. Keep it Layer 3
When I say Layer 3, I mean it. No VLANs at all - use stuff like `no switchport` and .1q tags if necessary. Eliminate spanning-tree completely
Carefully choose your routing protocols here. BGP is fault-tolerant and complex, OSPF rapidly recovers from failure due to link state and often overreacts to changes. I won't talk about EIGRP because I don't like proprietary routing protocols. Deal with it.

So this is where most people are at with their data centers - and when properly designed, the biggest danger to reliability is the network engineer. Once you finish building this design, network additions, configuration changes, software upgrades will be the leading cause of network outages. Most network gear available for purchase today is highly reliable, and since everything is Layer 2, this network design will not fail unless an anomaly is introduced or a change is made.

The next section will be for those of us that suffer undue stress and pressure due to a high frequency of change - it's possible to have the level of comfort that most systems engineers have when performing their work.

Forward Error Correction, a story about Generation 14 PowerEdge and 25 Gigabit connectivity

Fri, 15 Mar 2019 17:36:00 -0800

25 Gigabit implementations

First of all - anyone who assumes they think Layer 1 is simple is wrong.

That being said, 25G/50/100G/QSFP28 services are different beyond simply being 2.5x faster than 10G. 802.3by (or 25 Gig for those who use it):

Full-Duplex is mandatory.
Energy Efficient Operation
Stackable Lanes supporting speeds of up to 28Gbits/s
For those who love it, Twinax maxes out at 5 meters for now (http://www.ieee802.org/3/by/P802_3by_Objectives.pdf)
Currently, cost for 25G/100G silicon appears to be less than for 10G/100G at the switch level.

What has NOT changed

BER minimums are still 10^12
All existing 802.1* protocols remain supported

That being said, I've been working to implement 25G to the server for quite some time now, and we waited with bated breath as the new servers (sporting bcnxnet 2x25G NICs) booted up...

and proceeded not to establish any link-level connectivity.

Well, we followed the usual suspects, attempting to statically negotiate speed-duplex (which is probably a platform oddity) to no avail.

As it turns out - Forward Error Correction is the culprit. Upon reviewing the IEEE's docs on 802.3by, we found this gem, indicating the difficulties with negotiating different FEC modes:

http://www.ieee802.org/3/by/public/Jan16/hidaka_3by_01_0116.pdf

Clause 73 outlines a set of bits for FEC auto-negotiation that would allow (over 5 bits) signaling to establish a same-same connection for agreement on which mode to use - keep in mind that any active connection (all optics, twinax over 5 meters) will require some form of FEC to detect whether errors will probably occur on a link:

F0: 10G FEC Offered
F1: 10G FEC Requested
F2: 25G RS-FEC Offered (ideal)
F3: 25G RS-FEC Requested
F4: 25G Base-R (Fire Code) FEC requested

This is important for preventing downstream failures - now that we're transmitting data at considerably higher speeds, but since 802.3by has been released as recently as 2016 (where RS-FEC came out in 2017) support for various modes can be a bit lopsided. Here's the order of preference with a reliability bias - invert the list if latency is the primary goal / you use really good cables:

RS-FEC
FC-FEC
FEC Disabled

Currently, Generation 14 Dell Poweredge appears to support all modes, but defaults to "disabled" and completely fails to auto-negotiate. No matter what, using the Broadcom NICs onboard, you will need to consciously select an option here, and then apply it to your switch.

In addition, early-generation 802.3by switches like Cisco's Nexus EX will not support RS-FEC on single-lane modes, but will support in multi-lane transceivers:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-736651.html

This can also be resolved by buying newer generation switches (FX+), but all generations appear to auto-negotiate with no issues within the switch-to-switch realm.

What is FEC?

Well, the wikipedia article is a pretty good start ( https://en.wikipedia.org/wiki/Forward_error_correction) but is awfully vague. Long story short, you have the option of adding about 80-250 billionths of a second in latency to essentially achieve a "what-if" analysis on a links apparent reliability. This is great, especially with twinax, where bit errors are a bit more common than with fiber optics. FEC can also provide feedback on bit errors between destinations, allowing it to "train" or "self-heal" links - allowing for much higher link reliability.

What this means to me

In this case, the following design impacts should be made:

If it's important, use multi-lane slots for it:
If you're egressing a fabric, you should use QSFP28 transceivers if cost allows. This will provide RS-FEC where it counts
If you have spine switches, use QSFP28 transceivers.
If you're buying now, read the product sheets for both your servers and your switches to ensure that RS-FEC is supported, and use optical cabling.

Gotchas with NSX-T 2.4

Sun, 03 Mar 2019 13:09:00 -0900

NSX-T 2.4 is a major software upgrade with a multitude of new features, listed here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4.0/rn/VMware-NSX-T-Data-Center-240-Release-Notes.html

The documentation for this release is not very mature, so I've compiled some gotchas I found while installing NSX-T 2.4 below:

Ensure that when you configure Host Transport nodes (such as ESXi) that all transport zones you need are provisioned on the host! The node summary should have a minimum of two Transport zones, one for underlay and one for overlay:
Don't forget your uplink VLAN! The uplink VLAN must be configured under "Advanced Networking & Security" -> Networking -> Switching, and should participate in your underlay transport zone:
NSX Controllers are no more. This functionality is merged into the NSX manager - which now has clustering support. You'll need to configure a vIP for the manager as well, for these reasons.
NSX Managers need more RAM. VMWare Recommends 24 GBytes of Memory per manager if you have less than 64 hosts - I'm running stable with 16 and 1 host, so consider 24GB the minimum.

Running a serial console server over ESXi

Sun, 03 Mar 2019 12:24:00 -0900

Since I'm building a hybrid systems/networking lab, one of the key features I'll need is a serial console server to administer the lab switches. There are a few options here:

Find an old Cisco Router and some async octal cables (Rare, takes up rack space)
Purchase a serial console server like MRV, Perle, Internetwatchdogs, etc ($$$)
Build a RPi as the console server (current solution, consumes 1 outlet)
Build a VM, and connect the USB-to-Serial Adapter

The last one is interesting, here's why. I have an ansible server that I intend to use for most patching/administration tasks, and to trial out certain aspects of network automation, and ansible lists a very interesting feature, proxies:

https://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html

I could plausibly list the ansible VM's loopback address as a proxy, allowing me to use it to automate early-stage network provisioning without network connectivity. I know it's a petty thing to want to automate, but that particular aspect of network devices provisioning is pretty tedious, you have to:

Upgrade to your baselined code revision
Configure basic networking
Download baseline config, and then customize it
Restart to new config

Step 1 is a pretty slow task, and I'd like to automate it - it'd be great to let ansible babysit switches while they provision instead of having to be right there building on it the entire time. These are pretty simple tasks for most route-switch platforms - typically only requiring a binary copy and a reboot or two.

Anyhow, let's get down to configuring the basics. I'm performing this from the vCenter 6.7 GUI, so YMMV on user interfaces. All you have to do is plug in your USB-to-Serial adapter, and then add it to the VM as a "Host USB Device." I'd recommend FTDI-type adapters, they don't typically require any driver install to work on either ESXi or Linux.

Now, let's see if they show up:

1ansible:~ # ls /dev/ttyU*  
2/dev/ttyUSB0  /dev/ttyUSB1  /dev/ttyUSB2  /dev/ttyUSB3

We're all set! I typically use screen as a direct console emulator, but they all more or less do the same thing. At this point we're really just trying to test the console ports to see if they work:

 1ansible:~ # screen /dev/ttyUSB0  
 2         --- System Configuration Dialog ---  
 3Would you like to enter the initial configuration dialog? [yes/no]:  
 4ansible:~ # screen /dev/ttyUSB1  
 5User Access Verification  
 6Username:  
 7ansible:~ # screen /dev/ttyUSB2  
 8Would you like to terminate autoinstall? [yes]: yes  
 9ansible:~ # screen /dev/ttyUSB3  
10Switch>  
11ansible:~ # killall screen

Looks like we're fully functional on all serial ports - I have 3 unprovisioned WS-C3560-24-TS-E for future lab use. The last commmand was to ensure that the proxy software wouldn't have to compete with screen for ownership of a serial device.

We'll be installing ser2net next - it only supports telnet, but you can tunnel SSH in a prod environment. Honestly, if you want this in your work environment it'd be much better to use a dedicated console server - 48 ports will net you less than a Dell R430, and can connect to phone lines. They're worth it.

 1ansible:~ # zypper in ser2net  
 2Loading repository data... 
 3Reading installed packages... 
 4Resolving package dependencies... 
 5  
 6The following NEW package is going to be installed:  
 7  ser2net  
 8  
 91 new package to install. 
10Overall download size: 92.3 KiB. Already cached: 0 B. After the operation, additional 200.1 KiB will be used. 
11Continue? [y/n/...? shows all options] (y): y  
12Retrieving package ser2net-3.5-2.2.x86_64                                          (1/1),  92.3 KiB (200.1 KiB unpacked)  
13Retrieving: ser2net-3.5-2.2.x86_64.rpm ...........................................................................[done]  
14Checking for file conflicts: ----------------------------------------------------------------------------------------[done]  
15(1/1) Installing: ser2net-3.5-2.2.x86_64 ----------------------------------------------------------------------------[done]

Then we create a config file:

 1#  ::::  
 210000:telnet:3600:/dev/ttyUSB0:9600  
 310001:telnet:3600:/dev/ttyUSB1:9600  
 410002:telnet:3600:/dev/ttyUSB2:9600  
 510003:telnet:3600:/dev/ttyUSB3:9600  
 6  
 7BANNER:banner:SERIAL EMULATED PORT \p\r\n  
 8BANNER:banner1:TCP port \p device \d\r\n  
 9BANNER:banner2:TCP port \p device \d\r\n  
10BANNER:banner3:TCP port \p device \d  serial parms \s\r\n  
11TRACEFILE:tw1:/tmp/tw-\p-\Y-\M-\D-\H:\i:\s.\U  
12TRACEFILE:tr1:/tmp/tr-\p-\Y-\M-\D-\H:\i:\s.\U  
13OPENSTR:open1:Open str\r\n  
14CLOSEON:closehtml:  
15# Default value settings. The given values are the defaults. For non  
16# boolean values the possible values are given above. 
17#  
18#** serial device and SOL **  
19# speed: standard speeds shown above  
20#DEFAULT:speed:9600  
21# databits: 5,6,7,8  
22#DEFAULT:nobreak:false  
23#** serial device only **  
24#DEFAULT:databits:8  
25# stopbits: 1,2  
26#DEFAULT:stopbits:1  
27# parity: none, even, odd  
28#DEFAULT:parity:none  
29#DEFAULT:xonxoff:false  
30#DEFAULT:local:false  
31#DEFAULT:hangup_when_done:false  
32#DEFAULT:kickolduser:false

And we're set! Systemd will automatically start ser2net with the VM.

Minemeld installation, continued

Sun, 03 Mar 2019 08:04:00 -0900

I cheated/pivoted a little bit - decided to simulate a bit more closely what I'd be using at work. I bootstrapped a CentOS VM and followed the instructions in: https://github.com/PaloAltoNetworks/minemeld-ansible

1sudo yum install -y wget git gcc python-devel libffi-devel openssl-devel zlib-dev sqlite-devel bzip2-devel  
2wget https://bootstrap.pypa.io/get-pip.py  
3sudo -H python get-pip.py  
4sudo -H pip install ansible  
5git clone https://github.com/PaloAltoNetworks/minemeld-ansible.git  
6cd minemeld-ansible  
7ansible-playbook -K -i 127.0.0.1, local.yml  
8usermod -a -G minemeld  # add your user to minemeld group, useful for development

Everything worked fine - I had to retry the playbook once to get it to run, but the install playbook even enabled/started the requisite services. I'd highly recommend this approach over the OVA - it took me ~ 30 minutes in total to get Minemeld up and running in my lab, including the CentOS ISO download.

Anyone else who is doing this may find it useful to know that the usermod above doesn't grant you login access to minemeld - it has its own credential set. Default credentials are admin|minemeld.

My next objective will be to integrate with my lab firewall using EDLs. Here's a preview of it running without any custom miners - eventually I'd like to mine NSX-T's manager to share object groups between systems.

Minemeld installation, Part 1

Sun, 24 Feb 2019 12:10:00 -0900

Palo Alto Networks has provided a tool for public use - Minemeld - that will collate threat intelligence feeds and other indicators for a more dynamic security policy enforcement strategy with their firewalls:
https://www.paloaltonetworks.com/products/secure-the-network/subscriptions/minemeld

I have a slightly different use case - I want my lab firewall to be aware of each virtual machine in my lab, and to be able to use it intelligently. Some of this is available via the "VM Information Sources" (more information here) feature, but it doesn't appear to be aware of details like NSX-T security groups, etc. My goal will be to implement these features using Minemeld, with some future uses on the horizon as well.

Getting Started

First I browse to https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld to download any requisite packages.

Just a note - the provided OVA is based off Ubuntu 14.04 - a pretty old release. Performing a deeper search, I discovered that an ansible playbook is provided for install on recommended systems!
https://github.com/PaloAltoNetworks/minemeld-ansible

I'm going to install this on my ansible host - running openSUSE Tumbleweed:

  1admin@ansible:~> sudo zypper in wget git gcc python-devel libffi-devel openssl-devel  
  2Loading repository data...  
  3Reading installed packages...  
  4'openssl-devel' not found in package names. Trying capabilities.  
  5'wget' is already installed.  
  6No update candidate for 'wget-1.20.1-2.1.x86_64'. The highest available version is already installed.  
  7Resolving package dependencies...  
  83 Problems:  
  9Problem: python-devel-2.7.15-4.3.x86_64 requires glibc-devel, but this requirement cannot be provided  
 10Problem: gcc-8-2.4.x86_64 requires gcc8, but this requirement cannot be provided  
 11Problem: ruby2.5-rubygem-cfa-0.7.0-1.1.x86_64 requires ruby(abi) = 2.5.0, but this requirement cannot be provided  
 12  
 13Problem: python-devel-2.7.15-4.3.x86_64 requires glibc-devel, but this requirement cannot be provided  
 14  not installable providers: glibc-devel-2.29-1.3.i586[download.opensuse.org-oss]  
 15                   glibc-devel-2.29-1.3.i686[download.opensuse.org-oss]  
 16                   glibc-devel-2.29-1.3.x86_64[download.opensuse.org-oss]  
 17                   glibc-devel-2.29-1.3.i586[openSUSE-20190126-0]  
 18                   glibc-devel-2.29-1.3.i686[openSUSE-20190126-0]  
 19                   glibc-devel-2.29-1.3.x86_64[openSUSE-20190126-0]  
 20 Solution 1: Following actions will be done:  
 21  deinstallation of yast2-ruby-bindings-4.1.2-1.1.x86_64  
 22  deinstallation of yast2-samba-client-4.0.4-1.1.noarch  
 23  deinstallation of yast2-ntp-client-4.1.7-1.1.noarch  
 24  deinstallation of yast2-packager-4.1.24-1.1.x86_64  
 25  deinstallation of yast2-tftp-server-4.1.6-1.1.noarch  
 26  deinstallation of yast2-snapper-4.1.0-1.1.x86_64  
 27  deinstallation of yast2-vpn-4.0.1-1.2.noarch  
 28  deinstallation of yast2-users-4.1.7-1.1.x86_64  
 29  deinstallation of yast2-update-4.1.8-1.1.x86_64  
 30  deinstallation of yast2-tune-4.0.2-1.2.x86_64  
 31  deinstallation of yast2-transfer-4.0.0-1.3.x86_64  
 32  deinstallation of yast2-sysconfig-4.1.2-1.2.noarch  
 33  deinstallation of yast2-support-4.1.0-1.1.noarch  
 34  deinstallation of yast2-sudo-4.0.1-1.2.noarch  
 35  deinstallation of yast2-slp-4.0.0-1.3.x86_64  
 36  deinstallation of yast2-services-manager-4.1.14-1.1.noarch  
 37  deinstallation of yast2-security-4.1.2-1.2.noarch  
 38  deinstallation of yast2-samba-server-4.1.3-1.2.noarch  
 39  deinstallation of yast2-storage-ng-4.1.48-1.1.x86_64  
 40  deinstallation of yast2-proxy-4.1.0-1.1.noarch  
 41  deinstallation of yast2-printer-4.0.3-1.2.x86_64  
 42  deinstallation of yast2-pam-4.0.0-1.2.noarch  
 43  deinstallation of yast2-online-update-4.0.2-1.2.noarch  
 44  deinstallation of yast2-nis-client-4.1.0-1.1.x86_64  
 45  deinstallation of yast2-nfs-client-4.1.4-1.1.noarch  
 46  deinstallation of yast2-metapackage-handler-4.0.0-1.2.noarch  
 47  deinstallation of yast2-mail-4.1.0-1.2.noarch  
 48  deinstallation of yast2-journal-4.1.5-1.1.noarch  
 49  deinstallation of yast2-iscsi-client-4.1.4-1.1.noarch  
 50  deinstallation of yast2-hardware-detection-4.0.0-1.6.x86_64  
 51  deinstallation of yast2-firewall-4.1.10-1.1.noarch  
 52  deinstallation of yast2-country-data-4.1.7-1.2.x86_64  
 53  deinstallation of yast2-auth-server-4.1.0-1.2.noarch  
 54  deinstallation of yast2-auth-client-4.1.0-1.2.noarch  
 55  deinstallation of yast2-apparmor-4.1.7-1.1.noarch  
 56  deinstallation of yast2-add-on-4.1.10-1.1.noarch  
 57  deinstallation of autoyast2-installation-4.1.1-1.1.noarch  
 58  deinstallation of yast2-installation-4.1.34-1.1.noarch  
 59  deinstallation of yast2-online-update-frontend-4.0.2-1.2.noarch  
 60 Solution 2: Following actions will be done:  
 61  deinstallation of ruby2.5-2.5.3-2.1.x86_64  
 62  deinstallation of ruby2.5-rubygem-cfa_grub2-1.0.1-1.1.x86_64  
 63  deinstallation of ruby2.5-rubygem-cheetah-0.5.0-1.10.x86_64  
 64  deinstallation of ruby2.5-rubygem-fast_gettext-2.0.0-1.1.x86_64  
 65  deinstallation of ruby2.5-rubygem-gem2rpm-0.10.1-13.6.x86_64  
 66  deinstallation of ruby2.5-rubygem-ruby-augeas-0.5.0-3.9.x86_64  
 67  deinstallation of ruby2.5-rubygem-ruby-dbus-0.15.0-1.1.x86_64  
 68  deinstallation of ruby2.5-rubygem-simpleidn-0.1.1-1.1.x86_64  
 69  deinstallation of ruby2.5-rubygem-unf-0.1.4-1.9.x86_64  
 70  deinstallation of ruby2.5-rubygem-unf_ext-0.0.7.5-1.2.x86_64  
 71  deinstallation of ruby2.5-stdlib-2.5.3-2.1.x86_64  
 72 Solution 3: do not install python-devel-2.7.15-4.3.x86_64  
 73 Solution 4: break python-devel-2.7.15-4.3.x86_64 by ignoring some of its dependencies  
 74  
 75Choose from above solutions by number or skip, retry or cancel [1/2/3/4/s/r/c] (c): 2  
 76  
 77Problem: gcc-8-2.4.x86_64 requires gcc8, but this requirement cannot be provided  
 78  not installable providers: gcc8-8.2.1+r268506-1.1.i586[download.opensuse.org-oss]  
 79                   gcc8-8.2.1+r268506-1.1.x86_64[download.opensuse.org-oss]  
 80                   gcc8-8.2.1+r268506-1.1.i586[openSUSE-20190126-0]  
 81                   gcc8-8.2.1+r268506-1.1.x86_64[openSUSE-20190126-0]  
 82 Solution 1: Following actions will be done:  
 83  deinstallation of yast2-4.1.53-1.1.x86_64  
 84  deinstallation of yast2-ntp-client-4.1.7-1.1.noarch  
 85  deinstallation of yast2-packager-4.1.24-1.1.x86_64  
 86  deinstallation of yast2-tftp-server-4.1.6-1.1.noarch  
 87  deinstallation of yast2-snapper-4.1.0-1.1.x86_64  
 88  deinstallation of yast2-vpn-4.0.1-1.2.noarch  
 89  deinstallation of yast2-users-4.1.7-1.1.x86_64  
 90  deinstallation of yast2-update-4.1.8-1.1.x86_64  
 91  deinstallation of yast2-tune-4.0.2-1.2.x86_64  
 92  deinstallation of yast2-transfer-4.0.0-1.3.x86_64  
 93  deinstallation of yast2-sysconfig-4.1.2-1.2.noarch  
 94  deinstallation of yast2-support-4.1.0-1.1.noarch  
 95  deinstallation of yast2-sudo-4.0.1-1.2.noarch  
 96  deinstallation of yast2-slp-4.0.0-1.3.x86_64  
 97  deinstallation of yast2-services-manager-4.1.14-1.1.noarch  
 98  deinstallation of yast2-security-4.1.2-1.2.noarch  
 99  deinstallation of yast2-samba-server-4.1.3-1.2.noarch  
100  deinstallation of yast2-storage-ng-4.1.48-1.1.x86_64  
101  deinstallation of yast2-proxy-4.1.0-1.1.noarch  
102  deinstallation of yast2-printer-4.0.3-1.2.x86_64  
103  deinstallation of yast2-pam-4.0.0-1.2.noarch  
104  deinstallation of yast2-online-update-4.0.2-1.2.noarch  
105  deinstallation of yast2-nis-client-4.1.0-1.1.x86_64  
106  deinstallation of yast2-nfs-client-4.1.4-1.1.noarch  
107  deinstallation of yast2-metapackage-handler-4.0.0-1.2.noarch  
108  deinstallation of yast2-mail-4.1.0-1.2.noarch  
109  deinstallation of yast2-journal-4.1.5-1.1.noarch  
110  deinstallation of yast2-iscsi-client-4.1.4-1.1.noarch  
111  deinstallation of yast2-hardware-detection-4.0.0-1.6.x86_64  
112  deinstallation of yast2-firewall-4.1.10-1.1.noarch  
113  deinstallation of yast2-country-data-4.1.7-1.2.x86_64  
114  deinstallation of yast2-auth-server-4.1.0-1.2.noarch  
115  deinstallation of yast2-auth-client-4.1.0-1.2.noarch  
116  deinstallation of yast2-apparmor-4.1.7-1.1.noarch  
117  deinstallation of yast2-add-on-4.1.10-1.1.noarch  
118  deinstallation of autoyast2-installation-4.1.1-1.1.noarch  
119  deinstallation of yast2-installation-4.1.34-1.1.noarch  
120  deinstallation of yast2-ldap-4.0.0-1.5.x86_64  
121  deinstallation of patterns-yast-yast2_basis-20181130-1.1.x86_64  
122  deinstallation of yast2-online-update-frontend-4.0.2-1.2.noarch  
123 Solution 2: Following actions will be done:  
124  deinstallation of ruby2.5-rubygem-abstract_method-1.2.1-2.10.x86_64  
125  deinstallation of ruby2.5-rubygem-ruby-augeas-0.5.0-3.9.x86_64  
126  deinstallation of ruby2.5-rubygem-ruby-dbus-0.15.0-1.1.x86_64  
127  deinstallation of ruby2.5-rubygem-simpleidn-0.1.1-1.1.x86_64  
128  deinstallation of ruby2.5-rubygem-unf-0.1.4-1.9.x86_64  
129  deinstallation of ruby2.5-rubygem-unf_ext-0.0.7.5-1.2.x86_64  
130  deinstallation of ruby2.5-stdlib-2.5.3-2.1.x86_64  
131 Solution 3: do not install gcc-8-2.4.x86_64  
132 Solution 4: break gcc-8-2.4.x86_64 by ignoring some of its dependencies  
133  
134Choose from above solutions by number or skip, retry or cancel [1/2/3/4/s/r/c] (c): 2  
135  
136Problem: ruby2.5-rubygem-cfa-0.7.0-1.1.x86_64 requires ruby(abi) = 2.5.0, but this requirement cannot be provided  
137  deleted providers: ruby2.5-2.5.3-2.1.x86_64  
138 Solution 1: Following actions will be done:  
139  deinstallation of yast2-country-4.1.7-1.1.x86_64  
140  deinstallation of yast2-packager-4.1.24-1.1.x86_64  
141  deinstallation of yast2-ntp-client-4.1.7-1.1.noarch  
142  deinstallation of yast2-network-4.1.34-1.1.noarch  
143  deinstallation of yast2-snapper-4.1.0-1.1.x86_64  
144  deinstallation of yast2-installation-4.1.34-1.1.noarch  
145  deinstallation of autoyast2-installation-4.1.1-1.1.noarch  
146  deinstallation of yast2-storage-ng-4.1.48-1.1.x86_64  
147  deinstallation of yast2-add-on-4.1.10-1.1.noarch  
148  deinstallation of yast2-apparmor-4.1.7-1.1.noarch  
149  deinstallation of yast2-auth-client-4.1.0-1.2.noarch  
150  deinstallation of yast2-auth-server-4.1.0-1.2.noarch  
151  deinstallation of yast2-country-data-4.1.7-1.2.x86_64  
152  deinstallation of yast2-firewall-4.1.10-1.1.noarch  
153  deinstallation of yast2-hardware-detection-4.0.0-1.6.x86_64  
154  deinstallation of yast2-iscsi-client-4.1.4-1.1.noarch  
155  deinstallation of yast2-journal-4.1.5-1.1.noarch  
156  deinstallation of yast2-mail-4.1.0-1.2.noarch  
157  deinstallation of yast2-metapackage-handler-4.0.0-1.2.noarch  
158  deinstallation of yast2-nfs-client-4.1.4-1.1.noarch  
159  deinstallation of yast2-nis-client-4.1.0-1.1.x86_64  
160  deinstallation of yast2-online-update-4.0.2-1.2.noarch  
161  deinstallation of yast2-pam-4.0.0-1.2.noarch  
162  deinstallation of yast2-printer-4.0.3-1.2.x86_64  
163  deinstallation of yast2-proxy-4.1.0-1.1.noarch  
164  deinstallation of yast2-samba-server-4.1.3-1.2.noarch  
165  deinstallation of yast2-security-4.1.2-1.2.noarch  
166  deinstallation of yast2-services-manager-4.1.14-1.1.noarch  
167  deinstallation of yast2-slp-4.0.0-1.3.x86_64  
168  deinstallation of yast2-sudo-4.0.1-1.2.noarch  
169  deinstallation of yast2-support-4.1.0-1.1.noarch  
170  deinstallation of yast2-sysconfig-4.1.2-1.2.noarch  
171  deinstallation of yast2-transfer-4.0.0-1.3.x86_64  
172  deinstallation of yast2-tune-4.0.2-1.2.x86_64  
173  deinstallation of yast2-update-4.1.8-1.1.x86_64  
174  deinstallation of yast2-users-4.1.7-1.1.x86_64  
175  deinstallation of yast2-vpn-4.0.1-1.2.noarch  
176  deinstallation of patterns-yast-yast2_basis-20181130-1.1.x86_64  
177  deinstallation of yast2-online-update-frontend-4.0.2-1.2.noarch  
178 Solution 2: Following actions will be done:  
179  deinstallation of ruby2.5-rubygem-cfa-0.7.0-1.1.x86_64  
180  deinstallation of ruby2.5-rubygem-cheetah-0.5.0-1.10.x86_64  
181  deinstallation of ruby2.5-rubygem-fast_gettext-2.0.0-1.1.x86_64  
182  deinstallation of ruby2.5-rubygem-gem2rpm-0.10.1-13.6.x86_64  
183  deinstallation of ruby2.5-rubygem-ruby-augeas-0.5.0-3.9.x86_64  
184  deinstallation of ruby2.5-rubygem-ruby-dbus-0.15.0-1.1.x86_64  
185  deinstallation of ruby2.5-rubygem-simpleidn-0.1.1-1.1.x86_64  
186  deinstallation of ruby2.5-rubygem-unf-0.1.4-1.9.x86_64  
187  deinstallation of ruby2.5-rubygem-unf_ext-0.0.7.5-1.2.x86_64  
188  deinstallation of ruby2.5-stdlib-2.5.3-2.1.x86_64  
189 Solution 3: do not ask to install a solvable providing openssl-devel  
190 Solution 4: break ruby2.5-rubygem-cfa-0.7.0-1.1.x86_64 by ignoring some of its dependencies  
191  
192Choose from above solutions by number or skip, retry or cancel [1/2/3/4/s/r/c] (c): 2  
193Resolving dependencies...  
194Resolving package dependencies...  
195  
196The following 68 NEW packages are going to be installed:  
197  cpp cpp8 cvs cvsps gcc gcc8 git git-core git-cvs git-email git-gui gitk git-svn glibc-devel glibc-locale-base  
198  libapr1 libapr-util1 libasan5 libatomic1 libcrypt1 libffi-devel libgomp1 libisl19 libitm1 liblsan0 libmpc3 libmpfr6  
199  libmpx2 libmpxwrappers2 libopenssl-1_1-devel libopenssl-devel libruby2_6-2_6 libserf-1-1 libsha1detectcoll1  
200  libtsan0 libubsan1 libutf8proc2 libxcrypt-devel libXss1 linux-glibc-devel perl-Authen-SASL perl-DBD-SQLite perl-DBI  
201  perl-Digest-HMAC perl-Error perl-MailTools perl-Net-SMTP-SSL python python-devel ruby2.6  
202  ruby2.6-rubygem-abstract_method ruby2.6-rubygem-cfa ruby2.6-rubygem-cfa_grub2 ruby2.6-rubygem-cheetah  
203  ruby2.6-rubygem-fast_gettext ruby2.6-rubygem-gem2rpm ruby2.6-rubygem-ruby-augeas ruby2.6-rubygem-ruby-dbus  
204  ruby2.6-rubygem-simpleidn ruby2.6-rubygem-unf ruby2.6-rubygem-unf_ext subversion subversion-bash-completion  
205  subversion-perl tcl tk xhost zlib-devel  
206  
207The following 13 packages are going to be REMOVED:  
208  ruby2.5 ruby2.5-rubygem-abstract_method ruby2.5-rubygem-cfa ruby2.5-rubygem-cfa_grub2 ruby2.5-rubygem-cheetah  
209  ruby2.5-rubygem-fast_gettext ruby2.5-rubygem-gem2rpm ruby2.5-rubygem-ruby-augeas ruby2.5-rubygem-ruby-dbus  
210  ruby2.5-rubygem-simpleidn ruby2.5-rubygem-unf ruby2.5-rubygem-unf_ext ruby2.5-stdlib  
211  
212The following 15 packages are going to be upgraded:  
213  glibc glibc-extra glibc-locale nscd ruby yast2 yast2-bootloader yast2-core yast2-country yast2-network  
214  yast2-ntp-client yast2-packager yast2-ruby-bindings yast2-snapper yast2-tftp-server  
215  
216The following 6 recommended packages were automatically selected:  
217  git-cvs git-email git-gui gitk git-svn subversion-bash-completion  
218  
219The following 2 packages are suggested, but will not be installed:  
220  git-daemon git-web  
221  
22215 packages to upgrade, 68 new, 13 to remove.  
223Overall download size: 81.4 MiB. Already cached: 0 B. After the operation, additional 319.4 MiB will be used.  
224Continue? [y/n/...? shows all options] (y): y

Looks like this conflicts with Ruby somewhat - a non-issue for me. Time to run pip and install ansible:

 1admin@ansible:~> sudo -H python get-pip.py  
 2[sudo] password for root:  
 3Traceback (most recent call last):  
 4  File "get-pip.py", line 21361, in <module>  
 5    main()  
 6  File "get-pip.py", line 197, in main  
 7    bootstrap(tmpdir=tmpdir)  
 8  File "get-pip.py", line 82, in bootstrap  
 9    import pip._internal  
10  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/__init__.py", line 40, in <module>  
11  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/cli/autocompletion.py", line 8, in <module>  
12  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/cli/main_parser.py", line 12, in <module>  
13  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/commands/__init__.py", line 6, in <module>  
14  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/commands/completion.py", line 6, in <module>  
15  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/cli/base_command.py", line 25, in <module>  
16  File "/tmp/tmpqrZ_FD/pip.zip/pip/_internal/index.py", line 14, in <module>  
17  File "/tmp/tmpqrZ_FD/pip.zip/pip/_vendor/html5lib/__init__.py", line 25, in <module>  
18  File "/tmp/tmpqrZ_FD/pip.zip/pip/_vendor/html5lib/html5parser.py", line 7, in <module>  
19  File "/tmp/tmpqrZ_FD/pip.zip/pip/_vendor/html5lib/_inputstream.py", line 13, in <module>  
20  File "/tmp/tmpqrZ_FD/pip.zip/pip/_vendor/html5lib/_utils.py", line 10, in <module>  
21ImportError: No module named xml.etree.ElementTree

And it seems the pip install step is not necessary, as openSUSE handles this through the package manager. We're going to need to go a bit off-script here:

 1  
 2admin@ansible:~> zypper se pip  
 3Loading repository data...  
 4Reading installed packages...  
 5  
 6S | Name                                     | Summary                                                                  | Type  
 7--+------------------------------------------+--------------------------------------------------------------------------+--------  
 8i | python3-pip                              | Pip installs packages. Python packages. An easy_install replacement      | package  
 9ansible:/home/admin # pip install ansible  
10Collecting ansible  
11  Downloading https://files.pythonhosted.org/packages/e4/22/4325212e609071cd93b8142722d770f5defab34a95511f183e262f8de983/ansible-2.7.8.tar.gz (11.8MB)  
12    100% |████████████████████████████████| 11.8MB 3.4MB/s  
13Collecting jinja2 (from ansible)  
14  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)  
15    100% |████████████████████████████████| 133kB 20.5MB/s  
16Collecting PyYAML (from ansible)  
17  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)  
18    100% |████████████████████████████████| 276kB 2.3MB/s  
19Collecting paramiko (from ansible)  
20  Downloading https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3-none-any.whl (193kB)  
21    100% |████████████████████████████████| 194kB 19.3MB/s  
22Collecting cryptography (from ansible)  
23  Downloading https://files.pythonhosted.org/packages/98/71/e632e222f34632e0527dd41799f7847305e701f38f512d81bdf96009bca4/cryptography-2.5-cp34-abi3-manylinux1_x86_64.whl (2.4MB)  
24    100% |████████████████████████████████| 2.4MB 6.4MB/s  
25Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from ansible) (40.6.3)  
26Collecting MarkupSafe>=0.23 (from jinja2->ansible)  
27  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl  
28Collecting bcrypt>=3.1.3 (from paramiko->ansible)  
29  Downloading https://files.pythonhosted.org/packages/d0/79/79a4d167a31cc206117d9b396926615fa9c1fdbd52017bcced80937ac501/bcrypt-3.1.6-cp34-abi3-manylinux1_x86_64.whl (55kB)  
30    100% |████████████████████████████████| 61kB 17.2MB/s  
31Collecting pyasn1>=0.1.7 (from paramiko->ansible)  
32  Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)  
33    100% |████████████████████████████████| 81kB 20.0MB/s  
34Collecting pynacl>=1.0.1 (from paramiko->ansible)  
35  Downloading https://files.pythonhosted.org/packages/27/15/2cd0a203f318c2240b42cd9dd13c931ddd61067809fee3479f44f086103e/PyNaCl-1.3.0-cp34-abi3-manylinux1_x86_64.whl (759kB)  
36    100% |████████████████████████████████| 768kB 20.3MB/s  
37Collecting cffi!=1.11.3,>=1.8 (from cryptography->ansible)  
38  Downloading https://files.pythonhosted.org/packages/be/99/3a088b41d93aa46f07cf7fd4da1b3287e6899ad7b2b75f1a177edf025e1a/cffi-1.12.1-cp36-cp36m-manylinux1_x86_64.whl (428kB)  
39    100% |████████████████████████████████| 430kB 20.8MB/s  
40Requirement already satisfied: six>=1.4.1 in /usr/lib/python3.6/site-packages (from cryptography->ansible) (1.12.0)  
41Collecting asn1crypto>=0.21.0 (from cryptography->ansible)  
42  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)  
43    100% |████████████████████████████████| 102kB 14.4MB/s  
44Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->ansible)  
45  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)  
46    100% |████████████████████████████████| 163kB 18.0MB/s  
47Installing collected packages: MarkupSafe, jinja2, PyYAML, pycparser, cffi, bcrypt, asn1crypto, cryptography, pyasn1, pynacl, paramiko, ansible  
48  Running setup.py install for PyYAML ... done  
49  Running setup.py install for pycparser ... done  
50  Running setup.py install for ansible ... done  
51Successfully installed MarkupSafe-1.1.1 PyYAML-3.13 ansible-2.7.8 asn1crypto-0.24.0 bcrypt-3.1.6 cffi-1.12.1 cryptography-2.5 jinja2-2.10 paramiko-2.4.2 pyasn1-0.4.5 pycparser-2.19 pynacl-1.3.0  
52  
53ansible:/home/admin # git clone https://github.com/PaloAltoNetworks/minemeld-ansible.git  
54Cloning into 'minemeld-ansible'...  
55remote: Enumerating objects: 170, done.  
56remote: Counting objects: 100% (170/170), done.  
57remote: Compressing objects: 100% (121/121), done.  
58remote: Total 1042 (delta 89), reused 110 (delta 46), pack-reused 872  
59Receiving objects: 100% (1042/1042), 140.92 KiB | 1.35 MiB/s, done.  
60Resolving deltas: 100% (450/450), done.  
61ansible:/home/admin # cd minemeld-ansible/  
62admin@ansible:~/minemeld-ansible> ansible-playbook -K -i 127.0.0.1, local.yml  
63SUDO password:  
64  
65PLAY [minemeld playbook] *******************************************************************************************************************************************************************************************************************************************************  
66  
67TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************************************************************  
68ok: [127.0.0.1]  
69  
70TASK [infrastructure : debug] **************************************************************************************************************************************************************************************************************************************************  
71ok: [127.0.0.1] => {  
72    "msg": "Loading vars for openSUSE Tumbleweed 20190219"  
73}  
74  
75TASK [infrastructure : include_vars] *******************************************************************************************************************************************************************************************************************************************  
76fatal: [127.0.0.1]: FAILED! => {"msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}  
77        to retry, use: --limit @/home/admin/minemeld-ansible/local.retry  
78  
79PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************  
80127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=1

Looks like we need to find out where in the playbook with_first_found is defined.

1admin@ansible:~/minemeld-ansible> grep first_found */*/*/*  
2roles/infrastructure/tasks/main.yml:  with_first_found:  
3roles/minemeld/tasks/main.yml:  with_first_found:

With either location, the following YAML reference is made. I'll do some more research on what that does in another blog entry:

1# from http://serverfault.com/questions/587727/how-to-unify-package-installation-tasks-in-ansible  
2- include_vars: "{{ item }}"  
3  with_first_found:  
4    - "{{ ansible_distribution }}-{{ ansible_distribution_version }}.yml"  
5    - "{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml"  
6    - "{{ ansible_distribution }}.yml"

My Home Lab!

Sun, 24 Feb 2019 10:30:00 -0900

I've removed NSX-V 6.4.4, and am about to start some more datacenter/route-switch projects - this is the lab network diagram for reference:

And, the meatspace version - the cheetos are there in case of an emergency:

RFC 7710

Sun, 24 Feb 2019 10:21:00 -0900

Captive portals suck.

We need them for any public access network, but with the widespread adoption of HTTPS, it is really difficult for users to find a hint as to where to go to get their access.

How captive portals work

In most real-world applications, the client station is placed in a network where all of the station's generated traffic is not allowed until authentication is performed.

The problem

Any scalable, well-engineered, easy-to-use guest platform needs some way of notifying the user as to why they're being blocked, and how to properly connect.
The most common approach taken by Aruba, Palo Alto, most firewall platforms will enforce a destination NAT policy on unauthenticated guest traffic, translating every possible destination address -> your client access gateway. This method still requires DNS be allowed through unless you implement a separate rule for DNS (and a corresponding no-NAT policy).

As a side-effect, this implementation will make it appear as if your client access gateway is impersonating every site a user visits - pretty much all client web browsers will abort any connections that appear insecure in this manner. The only workaround is for the client station to browse to an HTTP web site - like http://neverssl.com/. As a consolation, some network devices attempt to access a captive portal detection URL automatically, like Apple's http://captive.apple.com/hotspot-detect.html, but these implementations vary greatly based on the end device and are not always reliable.

Enter RFC 7710

This is not an uncommon issue, as most modern enterprises, conventions, events will all require heavy use of this particular network type. Due to the usability issues presented above, most conventions and other platforms will typically just provide you the WPA2-PSK and leave it as-is.

The IETF did not ignore the difficulties that most have had with guest usability at-scale, and have proposed an internet standard to notify client stations that a captive portal exists:

https://tools.ietf.org/html/rfc7710

In a nutshell, the IETF proposed that the network discovery protocols used by a client station when connecting to a new network would present the most efficient path for notification for this case, specifying options for DHCPv4, DHCPv6, and IPv6 Router Advertisements:

DHCPv4 Option 160
DHCPv6 Option 103
RA Type 37

This completely eliminates the need for a MiTM if the client supports this RFC. It also lets you authenticate your client access gateway, which is vulnerable to spoofing otherwise:

In all of these cases, the implementation should be pretty simple - you just punch in the URI of your captive portal server, and you're done!

WPA and Open System Authentication

Thu, 14 Feb 2019 19:21:00 -0900

Did you know that before you authenticate to your wireless network, it's using the same security mechanisms as open Wi-Fi?

With TLS, it's fairly well known how (most) cipher suites implement the Diffie-Hellman exchange to provide reasonably effective forward secrecy. ECC Diffie-Hellman has largely superseded RSA, but the underlying reason for implementation remains the same - until you establish an encrypted session, confidentiality does not exist. The ultimate solution would be out-of-band exchanged pads, but that is technically infeasible. There will always be a compromise with sacrificial cipher exchanges to achieve forward secrecy.

This is a really helpful video that visually describes the Diffie-Hellman Exchange:
https://www.youtube.com/watch?v=YEBfamv-_do

Aruba's early implementation of WPA3, mixed mode

Thu, 14 Feb 2019 19:01:00 -0900

Aruba has released for general availability ArubaOS 8.4, which includes WPA3-PSK: https://www.arubanetworks.com/techdocs/ArubaOS/8.4.x.x/Default.htm

Understandably, I was pretty excited to try it out and promptly upgraded my instant cluster, contained within the "safe zone" of my home lab. It was running 8.3.0.3 before, and the upgrade required me to stand up an HTTP server to distribute binaries. The one-click upgrade worked with no issues and took ~10 minutes for both APs with no client-side downtime.

I'll try not to gush too much here, but this is a pretty wicked software release. The virtual controller UI is vastly improved and had a few new options:

Configuring WPA3-SAE was also pretty easy:

Once configured, I was able to connect to the WPA3 SSID I had created.

Wait, WHAT? Windows 10 doesn't have any WPA3 support yet! Digging a little deeper, I found that I was connected to an SSID that supported WPA2-Personal.

It'd appear that we have the capability to run both WPA2 and 3 at the same time. Of course, we can trust but verify with a packet capture. This is not normally feasible without a software-defined radio, but Aruba provides a tool (PEEKREMOTE) that will let you remotely pull a packet capture. If you're interested in doing this yourself, the guide on how is at the end of this article. There are some important steps to follow when decoding the PCAP.

Here's what I found on the RSN IE portion of the 802.11 beacon frame:

From the looks of it, the RSN IEs allow for multiple cipher suites and AKMs. This isn't surprising, as this was how WPA1/2 works in mixed mode. From the PCAP, I would surmise that 00:0f:ac:08 is the 802.11i designation for SAE.

WPA2 & 3 Differences, courtesy of Ruckus Networks:
https://theruckusroom.ruckuswireless.com/wired-wireless/technologytrends/wpa2-wpa3-new-changed-future/
802.11i Robust Security Network Information Elements:
https://mrncciew.com/2014/08/21/cwsp-rsn-information-elements/
Airheads announcement:

https://www.youtube.com/watch?v=RWGPCdP47E8
Details on WPA3 and why it matters (not a light read):
https://www.mdpi.com/2079-9292/7/11/284/htm
InstantOS Packet Captures:
https://wifiromigh.wordpress.com/2018/03/22/capturing-client-traffic-on-aruba-campus-and-instant-access-points-using-wireshark/

engyak.co Blog Archive

Mon, 01 Jan 0001 00:00:00 +0000

Search

Mon, 01 Jan 0001 00:00:00 +0000

engyak.co

Idempotently Manage Ubiquiti Unifi resources with Ansible

Building an Action

Action

requirements.txt

Building a Playbook

Starting an IaC Repository with GitHub and Terraform

Repositories (Structure and Naming)

Actions (Release Management)

Terraform Starter Kit

Visualize and Report Ansible with OpenTelemetry and Syslog

Types of Callback Plugins

aggregate

stdout

Using stdout callbacks

notification

Directing results to OpenTelemetry

Directing Results to Syslog

Modernizing the Monitoring Stack

Automate DNS Zone Generation and Deployment with Ansible and Netbox

The Pattern

The Code

Retrospective

The Gist

VM Deployment Pipelines with Proxmox

Proxmox Setup

Debian Linux Setup

Deployment Pipeline

Conclusion

Starting from scratch with Netbox IPAM

Different IP design strategies

IPv4

Bogons, and the basics

IPv6

Constructing an IP hierarchy

Guidance on prefix allocation

Automating it

Manage Linux patching with Ansible and Netbox!

Patching all of my random experiments took too much of my free time, so I automated it

Requirements

Iteration 1: Ansible with Jenkins

Retrospective

Iteration 2: Ansible, Netbox, GitHub Actions

Retrospective

Abstracting DNS Record Management with Ansible and Jinja 2

Build and Consume Alpine Linux vSphere Images

Deploying Linux for the impatient

Customizing and building the ISO

Modifying the deployment pipeline

Apollo 13 "Failure is not an option", and how non-engineers misinterpret it

Industry-wide changes take time

Don't fear failure, understand it

Applying failure analysis to IT

Example - a maintenance window backfired

Why?

Internet Load Balancing with pfSense

With full-time remote work, internet outages transform from a nuisance to a real problem

Internet Load Balancing

The Internet and its scaling issues

The Future (IPv6) Solution

Retrospectives

Handoff to Day-N Automation with vSphere Content Libraries and Netbox

The challenge with build automation is too much convenience

System Integration

Circumventing Coders block and starting a new project

It's difficult to start a new software project

Documentation

CI

...Then start writing code

Hindsight

Why wait? Eventual Consistency and Reliability

Patience is tough when deploying automated code; Here's why it is important

Infrastructure-as-Code

The Downsides

Infrastructure Engineering

Network Assurance

Python Virtual Environment Setup in Jenkins

Prerequisites

Set Jenkins Global Parameters

Install Packages (example from Debian 12)

`requirements.txt`

`aggregate`

`stdout`

Using `stdout` callbacks

`notification`

Peer Review / `pull request`