ENGYAK.co Logo
ENGYAK.co

NSX-T Transitive Networking

One major advantage to NSX-T is that Edge Transport Nodes (ETNs) are transitive

Transitivity (Wikipedia) (Consortium GARR) is an extremely important concept in network science, and in computer networking.

In simple terms, a network node (any speaker capable of transmitting or receiving on a network) can have the following transitivity patterns:

vSphere Switch Transitive Networking Design

To fully understand VMware’s approach, it is important to first understand earlier approaches to network virtualization. vSphere switches are a bit of a misnomer, as you don’t actually switch at any given point. Instead, vSphere switches leverage a “Layer 2 Proxy” of sorts, where NIC-accelerated software replaces ASIC flow-based transitive switching.

This approach offers incredible flexibility, but is theoretically slower than software switching; to preserve this capability VMware noticed early on that loop prevention would become an issue. Pre-empting this problem, making the platform completely non-transitive to ensure that this flexibility will be more readily adopted.

Note: VMware’s design choices here contained the direct intent to simplify the execution and management of virtualized networking. This choice made computer networking simple enough for most typical VI administrators to perform, but more of the advanced features (QoS, teaming configurations) require more direct involvement from network engineers to execute well. Generally speaking, the lack of need for direct networking intervention for a VSS/vDS to work has led to a negative trend with the VI administrator community. Co-operation between VI administration and networking teams often suffer due to this lack of synchronization, and with it systems performance as well.

vSphere Distributed Switch Transitivity

NSX-T Transitive Networking Design

NSX-T is highly prescriptive in terms of topology. VMware has known for years that a highly controlled design for transitive networking will provide stability to the networks it may participate in - just look at the maturity/popularity of vDS vs Nexus 1000v.

NSX-T does depend on VDS for Layer 2 forwarding (as we’ve established, not really switching), but does follow the same general principles for design.

NSX Transitivity

To be stable, you have to sacrifice flexibility. This is for your own protection. These choices are artificial design limitations, intentionally placed for easy network virtualization deployment.

VMware NSX-T Tier-0 logical routers have to be transitive to perform their main goal, transporting overlay traffic to underlay network nodes. Every time a network node becomes transitive in this way, specific design decisions must be made to ensure that anti-transitive measures are appropriately used to achieve network stability.

NSX-T Tier-1 Distributed routers are completely nontransitive, and NSX-T Tier-1 Service Routers have severely limited transitive capabilities. I have diagrammed this interaction as non-transitive because the Tier-1 services provided are technically owned by that logical router.

Applications for Transitive Tier-0 Routers

Given how tightly controlled transit is with NSX-T, the only place we can perform these tasks is via the Tier-0 Logical Router. Let’s see if it’ll let us transit networks originated from a foreign device, shall we?

Hypothesis

NSX-T Tier-0 Logical Routers are capable as transit providers, and the only constructs preventing transit are open standards (BGP import/export filters)

Unit Test

Peer with vCLOS network via (transiting) NSX-T Tier-0 Logical Router:

Transitivity Testing

Let’s build it, starting with the vn-segments:

Segment Configuration

Then, configuring Tier-0 External Interfaces:

External Interfaces

Ensure that we’re re-distributing External Interface Subnets:

Route Re-Distribution

Ensure that the additional prefixes are being advertised. Note: This is a pretty big gripe of mine with the NSX GUI - we really ought to be able to drill down further here…

Route Advertisement

Configure BGP Peering to the VyOS vCLOS Network:

VyOS BGP Peering

We’re good to go on the NSX Side. In theory, this should provide a transitive peering, as BGP learned routes are not Re-Distributed but learned.

(The other side is VyOS, configured in the pipeline method outlined in a previous post. This pipeline delivery method is really growing on me)

We can verify that prefixes are propagating transitively via the NSX-T Tier-0 in both protocol stacks by checking in on the spines that previously had no default route:

vyos@vyos-s1.engyak.net:~$ show ip route  
Codes: K - kernel route, C - connected, S - static, R - RIP,  
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,  
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,  
       F - PBR, f - OpenFabric,  
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup  
  
B>* 0.0.0.0/0 [20/0] via 10.6.194.1, eth1, weight 1, 00:15:20  
B>* 10.0.0.0/8 [20/0] via 10.6.194.1, eth1, weight 1, 00:15:20  
vyos@vyos-s1.engyak.net:~$ show ipv6 route  
Codes: K - kernel route, C - connected, S - static, R - RIpng,  
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,  
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,  
       f - OpenFabric,  
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup  
  
B>* ::/0 [20/0] via fe80::250:56ff:febc:b05, eth1, weight 1, 00:15:25

Now, to test whether or not packets actually forward:

vyos@vyos-s0.engyak.net:~$ ping 1.1.1.1  
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.  
64 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=49.7 ms  
64 bytes from 1.1.1.1: icmp_seq=2 ttl=53 time=48.10 ms  
64 bytes from 1.1.1.1: icmp_seq=3 ttl=53 time=45.9 ms  
64 bytes from 1.1.1.1: icmp_seq=4 ttl=53 time=45.0 ms

Looks like Tier-0 Logical Routers are transitive! This can have a lot of future implications - because NSX-T can become a launchpad for all sorts of virtualized networking. Some easy examples: