Saturday, December 3, 2022

Why Automate? Programmability is about solving new problems without fear of failure.

Have you ever heard someone say "I'm not a coder" at work?

The IT industry is changing again. Our humble origins began as polymaths moved from adjacent industries and created a new world from scratch. The pioneering phase led to unique opportunities, creating our transport protocols, programming languages, and ways of building.

The appetite for trying new things is fundamentally different now. We don't worry as much about functional quality with our IT products in this day and age. Even Windows, the butt of jokes in the 2000s provides a consistent and reliable user experience.

Is this the denouement for IT innovation? Neal Stephenson predicted this issue in 2011, examining the creativity and breakneck pace the aerospace industry developed in the 1960s/1970s.

More importantly, he brings to light a painful pattern that IT engineers often go through when trying to create new things for their company's goals - "done before" means something isn't worth doing. Don't we all buy products from (more or less) the same companies to achieve similar outcomes? Why should we care if an idea was executed before?

Liability for failure and high expectations both in quality and reduced risk are prolific in today's market. I'd argue that we have a new problem - after a decade or so of easy-to-implement, highly reliable products, we've forgotten what it feels like to try something new. We're told it costs too much, or might hurt the business when infrastructure engineers want to attempt something novel, and removing something costly is too much of a problem.

The software development market has this figured out. The shift to artistic creativity has provided some growing pains, but we see a potential bridge to the future here. Infrastructure engineers may not "be coders" but uncertain outcomes are what engineers (pragmatic creatives, not artistic creatives) excel at. Our analog in the industry, actual engineers incrementally improve a physical resource, creating safer cars or city designs that promote creative growth. They don't need to worry about the low-level components functioning.

IT is maturing, and our goals are changing, but we can't forget where we came from. Software Development is bifurcating from IT infrastructure. The internal focus for infrastructure is shifting to  providing tools and resources to developers as typical customers. We need to find strength in our pragmatic creativity.

Through rose-tinted glasses, "melting pot" innovation imparts a culture of "can-do" wherever success lives - but the transition to disposable electronics/mechanical products is removing opportunities for the development of the required skills. Deep down we know this is bad, and key themes are being set by those who can, "maker spaces" and "hackerspaces" are good examples of this key trend. We need to teach new engineers not to fear failure or the practice of trying.

This doesn't mean that we can throw caution to the wind. While I admire a farmer's ability to innovate at work, creating some trailblazing (albeit somewhat unsafe) fixes in the field is not what we need in IT. (Check out FarmCraft101 for some of the stuff they do)

We need to change how the IT infrastructure industry operates. Educating new engineers will always be at least a little bit of trial and error, and the most important thing we can do is create an environment where we balance the values of trailblazing and reliable delivery. Programmability does this for us, but we don't know how to use it fully yet - but we can look at the other engineering industries to see what might work for us.

What Works

IT Engineers all seem to agree that setting up "labs" facilitates healthy innovation. Home labs offer an environment where an individual can break and rebuild, with an onus to fix it. Resources like VMware's Hands-on-Labs provide a zero-cost method to learn without consequence, albeit with product marketing.

Dyed-in-the-wool engineers love testing. New engineers learn by examining failure without negative context; engineers may work for years before actually building anything themselves. The search term "failure analysis" provides a wealth of information on the processes used by pragmatically creative individuals to steadily improve modular designs to the point where they achieve an artistically creative outcome.

Continuous Delivery practices (DevOps) supercharge these practices; We don't have to deal with physicality. If we manage to automate testing, it costs us virtually zero time, and we can pick up the failure modes as educational resources for new engineers.

How We Can Change

I'd like to see a practice that is two-thirds engineering and one-third "redneck farm repairs". With the FarmCraft101 example, we see an admirable attitude instead of apprehension towards trying new things, and we need to combine it with mature, reliable practices.

The combination of removing or reducing costs for failure and a drive to try new things is about to reach a critical point in the IT industry. We're seeing waves of new engineers enter the industry born in the 2000s, and they don't remember having to try and set IRQs, flip DIP switches with the right type of ballpoint pen, and 32-bit memory ceilings. Personal Computers have become throwaway devices that we don't have to understand well to use, and we need ways to preserve the resilience that comes with "I can solve any problem that comes my way". Raspberry Pi, Arduino, and their lookalikes revitalize this mindset and provide a quality of education that we wish we had when we were young - let's make sure the younglings use it.

I'd also like to see some self-awareness. Most people who "can't code" are in the same boat as those who "can't write", they just don't feel artistically creative. Pragmatic creativity is the backbone of modern engineering - a concept artist doesn't design a car or make it real beyond visual aesthetic and non-functional requirements. The inability to write creatively or "code" is fixed first by identifying a useful goal and achieving it. Infrastructure engineers already do this - just look at how network engineers make long-haul connectivity meet a business objective, or how HTTP forwarding rules make an application behave better.

Let's remove the belief that coding is impossible - most of the truly "propeller-hat" stuff has been done by vendors and community members already - so this leaves most actual software development as "exchanging text files over a network" or "making deterministic paths for behavior". The reason why Object-Oriented Programming and other best practices are emphasized is to ensure you know why it's there. Understanding how an automatic transmission works and should be appropriately used helps a driver improve their skills, but the point to which we learn varies from person to person, and a deep understanding isn't always necessary.

Know your strengths, and know that you can meaningfully contribute by using them. It might take a lifetime to figure out how, but it's worth it.

Sunday, November 6, 2022

Tweak Application behaviors with NSX ALB

Not every application is designed to leverage a load balancer.

Early load balancing solutions capitalized on this fact - which may seem odd. The solution to the supportability issue creates new opportunity; we'll look at some examples here. Adjusting application behaviors quickly becomes a powerful tool when enhancing customer experience.

This was the story before digital transformation patterns began to emerge, at least. Business now prefer to use Commercial Off-The-Shelf (COTS) software and Content Management System (CMS) systems instead of writing raw code or web frameworks. This means that the software our SWEs are using is considerably less "tunable" than it used to be, and that the business line expects victory when adopting newer software.

They aren't wrong, the advantage at the other end of this transition is more maintainable software. It's important to be accommodating and compassionate when helping web developers improve their app - they didn't write it, and it's a new approach for them too.

Knowledge of how HTTP works is valuable here. For this exercise, I'm going to proxy Jenkins - it's not designed for a load balancer and is open source.

It's really important to standardize behavior with load balancers. Any profile or rule should be captured either in source control or a manager of managers. I prefer both, because tools like Git Pull Request provide a valuable place for peer review and seeking input on standards. It's good to have standards, it's better all qualified engineers agree on the standard (and there's a way to version/release new ones!)

We'll be using the following profiles for this application as a starting point:

  • http-profile-Jenkins-v1.0.0
    • This profile looks comprehensive - Avi Vantage stores all variables explicitly when anything is customized. Here are the highlights:
      • Enable HSTS: (Not required, I just like it. NSX ALB/Avi doesn't configure a separate service for port 80, so this ensures we don't run cleartext traffic.
      • Reset HTTP on HTTPS connections: This is an Avi oddity - by default, HTTPS services will return an HTTP 400 series code instead of resetting the connection. I don't like giving out thumbprint data on a given port when it asks for illegal output.
      • X-Forwarded-For: Avi will include an HTTP Header with this label and the client's IP address. This allows a web engineer to configure logging with the real address in 1-arm deployment scenarios
  • clienttls-v1.0.0
    • This client TLS profile is short because we can be strict on security.
      • Include a short acceptable cipher suite/cipher list, and enforce forward secrecy
      • Only support TLS 1.2 and 1.3
      • Prefer server cipher ordering (enforce good cryptography)

We've already covered significant ground in terms of standardization and user experience here. Many web apps don't plan to implement user-friendly TLS, for example, and benefit from NSX ALB/Avi in front of their web tier, acting as a security guard until the client successfully negotiates TLS and navigates to the appropriate port. Many offerings also layer WAF, DDoS protection into their products for this very reason.

Let's configure a basic virtual service:

This basic wizard will establish some acceptable defaults. Let's modify the virtual service and add our settings:

  • Application Profile
  • TLS Profile
  • Certificate

Let's also configure this service to listen on port 80.

Note: Avi will forward unencrypted traffic to the pool, so some loss of privacy may occur if you don't configure the redirect rule under Profiles -> HTTP Request prior to saving!

The new service should be healthy under Application -> Dashboard:

Let's try and use the app!

We encounter an error when clicking "Manage Jenkins":


This is the configured name of my Jenkins server - and it's no longer using the virtual server's name.

We encounter this issue commonly with CMS, and "hard linked" applications can prove difficult to resolve. There are two approaches to resolving this issue:

  • Leverage redirects, either:
    • Rewriting any redirects that flow through the load balancer
    • Redirecting from port 8080 to the correct service
  • Rewrite HTML as it streams through the load balancer
    • Avoid this if possible, it's computationally very costly and may not support production traffic levels. It's still useful while developing a new website, as it might be a quicker hypothesis test than a coded solution or a vendor request for change (RFC)

Before diving into a specific fix, it's quite easy to categorize the problem with Firefox. Press F12 on your keyboard to bring up the developer view, and select the Network tab:

Note: This is a big part of why web developers always think it's the network that broke their application.

Click the link again, and you will see a play-by-play of the HTTP transactions executed complete the transaction:

That's a bingo. Jenkins is forwarding a non-relative HTTP 302 (also known as: The link is in another castle HTTP code). Let's rewrite it with NSX ALB by creating an HTTP Response Policy

Here's a quick cheat sheet:

  • HTTP Response policies let an engineer edit HTTP transactions (but not the body) from the server to the client
  • HTTP Request policies let an engineer edit HTTP transactions (but not the body) from the client to the server

The `Location` header in an HTTP transaction applies specifically to redirects (301,302) - and as such is easy for a load balancer to rewrite. In this case, it wasn't the HTML pages that had a bad link - the redirect chain `/manage->/manage/` was the culprit.

VMware NSX ALB (and any other self-respecting ADC) presents an infrastructure engineer with near unlimited power to ensure that an application is delivered correctly (even if there's a flaw in the product). Check out the Developer Screen's network tab (F12) in Firefox/Chrome on some common websites - you will find that many of them employ similar "fixer" tactics.

Popular Posts