Vendor interoperability with multiple STP instances

Spanning Tree is the all-important loop prevention method for Layer 2 topologies and source of ire to network engineers worldwide.

Usually IT engineers list the Dunning-Kruger Effect in a negative context, depicting an oblivious junior or an unaware manager, but I like to focus on the opposite end of the curve with meta-cognition. Striving to developing meta-cognition and developing self-awareness is difficult and competes with the ego, but is an incredibly powerful tool for learning. I cannot stress enough how important getting comfortable with one's limitations is to a career.

Spanning Tree is a key topic that should be revisited frequently, building upon knowledge growth.

Let's examine some methods for plural instantiation with Spanning Tree.

The OSI Model and sublayers

Ethernet by itself is surprisingly limited in scope for the utility we glean from it. It provides:

• Source/Destination forwarding
• Variable length payloads
• Segmentation with 802.1q (VLAN tagging) or 802.3ad (Q-in-Q tagging)

It also doesn't provide a Time-to-Live field at all, which is why switching loops are so critically dangerous in production networks.

Ethernet needs a supporting upper layer (the Logical Link Control sublayer) to relay instructions on how to handle packets. Internal to a switch ASIC, the hardware itself needs some form of indicator to select pipelines for forwarding or processing.  

802.2 LLC and Subnetwork Access Protocol (SNAP) are typically used in concert to bridge this gap and allow a forwarding plane to classify frames and send to the appropriate pipeline. Examples are provided with this article, where LLC and SNAP are used to say "this is a control plane packet, don't copy it, process it" or "this packet is for hosts, copy and forward it".

Multiple Ethernet Versions

Over the years, network vendors implemented multiple versions of the Lower Ethernet sublayer, and in many cases did not update the Control Plane code on network equipment.  manage to proliferate throughout computer networks for , which appear to resemble simply stitching together ancient PDU types for compatibility . It's not entirely surprising that multiple ethernet editions exist given the fragmentation throughout the industry.

I'd strongly recommend reading this study by Muhammad Farooq-i-Azam from the COMSATS Institute of Information Technology. The author outlines methods of testing common forms of Ethernet in production formats, and provides a detailed overview of our progress to standardization. Spanning Tree is a major cause for the remaining consolidation work, as it turns out.

Generally, Ethernet II is what you want to see in a production network, and most host frames will follow this standard. Variable-length fields over 1536 bytes are supported by this protocol, which is a big advantage in data centers.

The original Ethernet standard, 802.3 did not support ethertypes or frames larger than 1536 bytes, and is typically used by legacy code in a switching control plane. Two major variants are used within this protocol, extended by LLC and (optionally) with SNAP as an extension.

So, why does this matter when talking about spanning tree? Network Equipment Providers (NEP) haven't all updated their Layer 2 control plane protocols in a long time. Bridge Protocol Data Units (BPDUs) are inconsistently (consistently in their hardware, but inconsistent with others) transmitted, causing a wide variety of interoperability issues.

Per-VLAN Spanning Tree

From a protocol standpoint, Per-VLAN STP and RSTP are probably the simplest method with the fewest design implications and most intuitive protocol layout - but some dangers are inherent when running multi-vendor networks.

Let's examine a few captured packets containing the STP control plane

Cisco PVRST+ Dissection

Cisco structures the per-VLAN control plane by wrapping instantiated BPDUs:

• With an Ethernet II Layer 2 Lower
• With an 802.1q tag encapsulating the VLAN ID
• With a SNAP Layer 2 Upper, PID of PVSTP+ (0x010b)
• Destination of 0100.0ccc.cccd

Arista and Cisco Vendor Interoperability

Interestingly enough, I discovered that Arista appears to implement compatible PVRST to Cisco (with some adjustments covered in Arista's whitepaper). To validate this, I executed another packet dissection with Arista's vEOS, which is available to the public. I have provided the results here, but the PDUs are nearly identical to the Cisco implementation.

MST Dissection

For the majority of vendor interoperable spanning-tree implementations, this will be a network engineer's best option. MST allows an engineer to specify up to 16 separate instances of Spanning Tree, either 802.1d or 802.1w. The primary hazards with leveraging MST have a great deal to do with trunking edge ports, as each topology must be accounted for and carefully planned. BPDU Guard, Loop Guard, and VLAN pruning are absolutely necessary when planning MST, in addition to diagramming each topology that will be instantiated.

IEEE's MST standard is implemented per-instance, and relayed with one common BPDU. It's pleasingly efficient, but...

• With an 802.3 Ethernet Layer 2 Lower
• With no 802.1q tag
• With an LLC header, ID of BPDU (0x42)
• With a destination MAC 0180.c200.0000
• After the BPDU typical attributes, MST attaches an extension indicating priorities, but no unique bridge IDs. If a topology differs, it would be separated onto its own control plane frame.

CST Dissection

For comparison, I also compared against a Mikrotik Routerboard, which should follow the implements RSTP as a single instance (Common Spanning Tree) and optionally supports Multiple Spanning-Tree (MST). I found the following attributes with default settings:

• Destination of 0180.c200.0000 (STP All-Bridges Destination Address)
• 802.3 Ethernet Frame
• Spanning-tree BPDU Type

Conclusion

Spanning Tree is a foundation to all enterprise networks, but there really seems to be some legacy code and general weirdness in place here. The industry, starting with the data center, is moving to a more deterministic control plane to replace it, whether that be EVPN or a controller-based model like NSX.

Campus enterprise deployments are beginning to do the same as businesses realize that a shared, multi-tenant campus network can increase the value against the cost of the same equipment. With the downsizing of corporate offices, the only thing stopping office providers from also providing a consolidated campus network is a general lack of expertise and an industry full of under-developed solutions. As platforms converge, the same pattern will emerge in campus deployments soon.

In terms of design implications, supplanting Layer 2 legacy control planes is no small feat. Even EVPN requires STP at the edge, but containment and exceptions management are both clear design decisions to make when building enterprise networks.