Track Certificate Expiration with Jenkins and Python 3!

CI/CD tools aren't just for automatically deploying apps! Jenkins excels at enabling an engineer to automatically execute and test code - but, it has a hidden super-power: Automating boring and intensive IT tasks(removing toil).

Let's take a common and relatable IT problem - it doesn't matter if you're a DevOps engineer, a Agilista, or even a "normal" systems engineer. Tracking certificate expiration is not an enjoyable task, and can often involve either manual checking or (usually) outages to discover that a certificate has expired.

This solution will have several major elements:

• An inventory of TLS-issued hosts
• A Python 3 script (leveraging OpenSSL) to open up TLS connections and fetch certificates
• A Jenkins pipeline to execute that script against that inventory daily, emailing the results

Inventory

Full transparency, this example is executed in a home lab. It's naive to assume that this task is trivial for any enterprise, but here are some potential approaches to building an inventory at scale:

• Write a Python script to ingest DNS zone files, and loop curl to see if any listen on port 443
• Fetch a report for a vulnerability scanner (Retina, Qualys, Nexpose)
• Searching PKI issuance reports (if available)

We also want to write our inventory file in a way that's friendly to our execution approach. Python is dynamically typed, and most IT automation is fine with that - we're not doing any hardcore programming for most of it. The vast majority of IT automation involves sending and processing files and I/O.

Python will change a variable to any data type when you tell it to, so it's useful to map out what we want. Here are the relevant data types. I will also include the symbols JSON uses to signify them (if applicable):

• String (""): This is a type that encapsulates a series of text characters
• Integer (No wrapping): Whole number, and Signed. It can be positive or negative, but there's no decimal point (decimal points are their own unique flavor of complexity in computer programming)
• List ([]): To a dyed in the wool software developer, this will be similar to an array. Lists are indexed by an integer, and can contain any data type below it
• Python has a neat trick where a for loop can return a list item instead of the index, which saves a great deal of code
• Python can sort a list by executing the function .sort() on that object
• Dictionary ({}): This is an advanced construct, and provides an engineer with a great deal of capability (at the expense of performance, and code simplicity in some cases)
• dicts store entries as key-value pairs, and the index is usually a string
• Python can add to a dictionary by adding a new key, e.g. dictionary["newkey"] = "b33f"

When planning software functionality, we want to always use the right tool for the job. Downstream APIs (e.g. OpenSSL) want to see a particular format for a parameter(e.g. TCP port should be an integer), so documentation research is a must at this phase. I'll explain my logic for this file:

• I want to easily iterate through the list, without addressing indexes, and I want it to be fast. I should use a list for the top-level data in the inventory ([])
• I want to ensure that I don't accidentally address the keys wrong, so each individual entry should be a dictionary ({}) with the following keys: fqdn, port
• fqdn should store a string
• port should store an integer

Example:

[ { "fqdn": "vcenter.engyak.co", "port": 443 }, { "fqdn": "nsx.engyak.co", "port": 443 } ]

Python Code

Here's a copy of my code. To execute it, the following pip packages need to be installed:

• fqdn
• OpenSSL
• ruamel.yaml

datetime in particular does quite a bit of heavy lifting here. The package providers in the Python community have managed to solve most of the truly difficult work, so interpreting expiration dates is a simple comparison operation.

I heavily rely on functions for this code to work in a maintainable format - this code is only 167 lines long, but most of the usage is for readability.

Another point of note - when writing Python to execute in a pipeline, it helps to be Perl levels of dramatic when crashing code. Jenkins doesn't evaluate output by default, and the easiest way to notify of a problem is by using sys.exit(""). This is why I placed a crash if errors exist at the end of the list.

Jenkins

This configuration example should provide some basic level of functionality. Jenkins has a lot of capability, so this tooling can be endlessly tweaked to your needs.

First, let's set up a SMTP server. With a default installation, the settings are under Dashboard -> Manage Jenkins -> Configure System:

Advanced Settings will allow you to configure SMTP auth, port, if applicable. If you use Gmail, you can still leverage MFA and app passwords, preserving MFA and avoiding password proliferation.

Now, let's set up a freestyle pipeline:

With Jenkins, all things should be executed from source code. This is the way.

We want to run this daily, irrespective of source code changes. This requires a slight deviation from the usual Poll SCM approach:

As always, amnesic workspaces are best:

Python and the inventory file simplify the Jenkins configuration as well. Just execute the script as-is:

The final step is to add a Post-Build Action to email if there is a failure:

It really is this simple. Jenkins will now execute daily and email you a list of expired, and soon to expire certificates!

Lessons Learned

I'm going to improve this code. Here are some of my ideas:

• Provide a set of rules for each warning level, e.g. 14 days for Critical
• JSON Schema validation will help make the ingest more intuitive
• Write/Add a Jenkins parser to chart different severity issues, not unlike Jenkins' flake8 plugin

I'm continually amazed at what the open source community can achieve with this level of simplicity. Would you consider this approach out of reach or too challenging?