Tweak Application behaviors with NSX ALB
Not every application is designed to leverage a load balancer.
Early load balancing solutions capitalized on this fact - which may seem odd. The solution to the supportability issue creates new opportunity; we'll look at some examples here. Adjusting application behaviors quickly becomes a powerful tool when enhancing customer experience.
This was the story before digital transformation patterns began to emerge, at least. Business now prefer to use Commercial Off-The-Shelf (COTS) software and Content Management System (CMS) systems instead of writing raw code or web frameworks. This means that the software our SWEs are using is considerably less "tunable" than it used to be, and that the business line expects victory when adopting newer software.
They aren't wrong, the advantage at the other end of this transition is more maintainable software. It's important to be accommodating and compassionate when helping web developers improve their app - they didn't write it, and it's a new approach for them too.
Knowledge of how HTTP works is valuable here. For this exercise, I'm going to proxy Jenkins - it's not designed for a load balancer and is open source.
It's really important to standardize behavior with load balancers. Any profile or rule should be captured either in source control or a manager of managers. I prefer both, because tools like Git Pull Request provide a valuable place for peer review and seeking input on standards. It's good to have standards, it's better all qualified engineers agree on the standard (and there's a way to version/release new ones!)
We'll be using the following profiles for this application as a starting point:
-
This profile looks comprehensive - Avi Vantage stores all variables explicitly when anything is customized. Here are the highlights:
-
Enable HSTS: (Not required, I just like it. NSX ALB/Avi doesn't configure a separate service for port 80, so this ensures we don't run cleartext traffic.
-
Reset HTTP on HTTPS connections: This is an Avi oddity - by default, HTTPS services will return an HTTP 400 series code instead of resetting the connection. I don't like giving out thumbprint data on a given port when it asks for illegal output.
-
X-Forwarded-For: Avi will include an HTTP Header with this label and the client's IP address. This allows a web engineer to configure logging with the real address in 1-arm deployment scenarios
-
This client TLS profile is short because we can be strict on security.
-
Include a short acceptable cipher suite/cipher list, and enforce forward secrecy
-
Only support TLS 1.2 and 1.3
-
Prefer server cipher ordering (enforce good cryptography)
We've already covered significant ground in terms of standardization and user experience here. Many web apps don't plan to implement user-friendly TLS, for example, and benefit from NSX ALB/Avi in front of their web tier, acting as a security guard until the client successfully negotiates TLS and navigates to the appropriate port. Many offerings also layer WAF, DDoS protection into their products for this very reason.
Let's configure a basic virtual service:
This basic wizard will establish some acceptable defaults. Let's modify the virtual service and add our settings:
- Application Profile
- TLS Profile
- Certificate
Let's also configure this service to listen on port 80.
Note: Avi will forward unencrypted traffic to the pool, so some loss of privacy may occur if you don't configure the redirect rule under Profiles -> HTTP Request prior to saving!
The new service should be healthy under Application -> Dashboard:
Let's try and use the app!
We encounter an error when clicking "Manage Jenkins":
This is the configured name of my Jenkins server - and it's no longer using the virtual server's name.
We encounter this issue commonly with CMS, and "hard linked" applications can prove difficult to resolve. There are two approaches to resolving this issue:
-
Leverage redirects, either:
-
Rewriting any redirects that flow through the load balancer
-
Redirecting from port 8080 to the correct service
-
Rewrite HTML as it streams through the load balancer
-
Avoid this if possible, it's computationally very costly and may not support production traffic levels. It's still useful while developing a new website, as it might be a quicker hypothesis test than a coded solution or a vendor request for change (RFC)
Before diving into a specific fix, it's quite easy to categorize the problem with Firefox. Press F12 on your keyboard to bring up the developer view, and select the Network tab:
Note: This is a big part of why web developers always think it's the network that broke their application.
Click the link again, and you will see a play-by-play of the HTTP transactions executed complete the transaction:
That's a bingo. Jenkins is forwarding a non-relative HTTP 302 (also known as: The link is in another castle HTTP code). Let's rewrite it with NSX ALB by creating an HTTP Response Policy.
Here's a quick cheat sheet:
- HTTP Response policies let an engineer edit HTTP transactions (but not the body) from the server to the client
- HTTP Request policies let an engineer edit HTTP transactions (but not the body) from the client to the server
The Location header in an HTTP transaction applies specifically to redirects (301,302) - and as such is easy for a load balancer to rewrite. In this case, it wasn't the HTML pages that had a bad link - the redirect chain /manage->/manage/ was the culprit.
VMware NSX ALB (and any other self-respecting ADC) presents an infrastructure engineer with near unlimited power to ensure that an application is delivered correctly (even if there's a flaw in the product). Check out the Developer Screen's network tab (F12) in Firefox/Chrome on some common websites - you will find that many of them employ similar "fixer" tactics.