ENGYAK.co Logo
ENGYAK.co

Forward Error Correction, a story about Generation 14 PowerEdge and 25 Gigabit connectivity

25 Gigabit implementations

First of all - anyone who assumes they think Layer 1 is simple is wrong.

That being said, 25G/50/100G/QSFP28 services are different beyond simply being 2.5x faster than 10G. 802.3by (or 25 Gig for those who use it):

What has NOT changed

That being said, I’ve been working to implement 25G to the server for quite some time now, and we waited with bated breath as the new servers (sporting bcnxnet 2x25G NICs) booted up…

and proceeded not to establish any link-level connectivity.

Well, we followed the usual suspects, attempting to statically negotiate speed-duplex (which is probably a platform oddity) to no avail.

As it turns out - Forward Error Correction is the culprit. Upon reviewing the IEEE’s docs on 802.3by, we found this gem, indicating the difficulties with negotiating different FEC modes:

http://www.ieee802.org/3/by/public/Jan16/hidaka_3by_01_0116.pdf

Clause 73 outlines a set of bits for FEC auto-negotiation that would allow (over 5 bits) signaling to establish a same-same connection for agreement on which mode to use - keep in mind that any active connection (all optics, twinax over 5 meters) will require some form of FEC to detect whether errors will probably occur on a link:

This is important for preventing downstream failures - now that we’re transmitting data at considerably higher speeds, but since 802.3by has been released as recently as 2016 (where RS-FEC came out in 2017) support for various modes can be a bit lopsided. Here’s the order of preference with a reliability bias - invert the list if latency is the primary goal / you use really good cables:

  1. RS-FEC
  2. FC-FEC
  3. FEC Disabled

Currently, Generation 14 Dell Poweredge appears to support all modes, but defaults to “disabled” and completely fails to auto-negotiate. No matter what, using the Broadcom NICs onboard, you will need to consciously select an option here, and then apply it to your switch.

In addition, early-generation 802.3by switches like Cisco’s Nexus EX will not support RS-FEC on single-lane modes, but will support in multi-lane transceivers:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-736651.html

This can also be resolved by buying newer generation switches (FX+), but all generations appear to auto-negotiate with no issues within the switch-to-switch realm.

What is FEC?

Well, the wikipedia article is a pretty good start ( https://en.wikipedia.org/wiki/Forward_error_correction) but is awfully vague. Long story short, you have the option of adding about 80-250 billionths of a second in latency to essentially achieve a “what-if” analysis on a links apparent reliability. This is great, especially with twinax, where bit errors are a bit more common than with fiber optics. FEC can also provide feedback on bit errors between destinations, allowing it to “train” or “self-heal” links - allowing for much higher link reliability.

What this means to me

In this case, the following design impacts should be made: