Spine and Leaf Practical Applications, EGP and IGP combined!

So far, all examples to date have been extremely simple, and stand rather well on their own (OSPF Clos would work just FINE in a campus network if you have cheap L3) but may not effectively address some more advanced use cases.

In short, we’re about to enter niche territory.

Generally speaking, BGP is regarded as the most advanced networking topic out there, until it isn’t. Most of the complexity lies in iBGP (same Autonomous System everywhere), because BGP’s primary loop prevention mechanism is AS-Path (count on the number of AS this route transits).

To successfully implement iBGP (a staple of every carrier, so totally do-able), a network engineer must choose one of the following paths (non-inclusive list):

Fully mesh all BGP speakers with each other (factorial scaling)
Implement a Route Reflector: https://en.wikipedia.org/wiki/Route_reflector

In this case, the more scalable option is to opt for a route reflector, but it isn’t that easy.

iBGP, when compared to eBGP:

Doesn’t care about hop count to a peer speaker. As long as the route reflector is less than 255 hops away, there is no issue
Usually doesn’t care about resolving paths to peer speakers - that’s a problem for something else.

To provide a good example of an expandable, scalable fabric that can offer eBGP as a service to subtending network devices, we will implement IS-IS as the intra-fabric routing protocol, and then leverage iBGP with the spine switches as route reflectors.

First things first, diagram is here: (YAML)

First, we’d configure the spines. Note that future releases (my home lab is rockin’ IOS 12.2.55, too old for this) the BGP Dynamic Neighbor that comes with more modern network operating systems is really useful.

Note: route reflector client status is configured on the SERVER side:

router bgp 65000  
 bgp log-neighbor-changes  
 neighbor 10.6.0.0 remote-as 65000  
 neighbor 10.6.0.0 update-source L0  
 neighbor 10.6.0.0 route-reflector-client  
 neighbor 10.6.0.1 remote-as 65000  
 neighbor 10.6.0.1 update-source L0  
 neighbor 10.6.0.1 route-reflector-client  
 neighbor FD00:6::0 remote-as 65000  
 neighbor FD00:6::0 update-source L0  
 neighbor FD00:6::0 route-reflector-client  
 neighbor FD00:6::1 remote-as 65000  
 neighbor FD00:6::1 update-source L0  
 neighbor FD00:6::1 route-reflector-client  
 maximum-paths 2  
 !  
 address-family ipv4  
  neighbor 10.6.0.0 activate  
  neighbor 10.6.0.1 activate  
  maximum-paths 2  
  no auto-summary  
  network 10.6.0.240 mask 255.255.255.254  
  network 10.6.240.0 mask 255.255.255.254  
  network 10.6.240.2 mask 255.255.255.254  
 exit-address-family  
 !  
 address-family ipv6  
  neighbor FD00:6::0 activate  
  neighbor FD00:6::1 activate  
  network FD00:6::240/127  
  network FD00:6:240::/126  
  network FD00:6:240::4/126  
  no synchronization  
  maximum-paths 2  
 exit-address-family  
!

And then the leaf configuration:

  
router bgp 65000  
 bgp log-neighbor-changes  
 neighbor 10.6.0.240 remote-as 65000  
 neighbor 10.6.0.240 update-source L0  
 neighbor 10.6.0.241 remote-as 65000  
 neighbor 10.6.0.241 update-source L0  
 neighbor FD00:6::240 remote-as 65000  
 neighbor FD00:6::240 update-source L0  
 neighbor FD00:6::241 remote-as 65000  
 neighbor FD00:6::241 update-source L0  
 maximum-paths 2  
 !  
 address-family ipv4  
  neighbor 10.6.0.240 activate  
  neighbor 10.6.0.241 activate  
  maximum-paths 2  
  no auto-summary  
  network 10.6.0.240 mask 255.255.255.254  
  network 10.6.240.0 mask 255.255.255.254  
  network 10.6.240.2 mask 255.255.255.254  
 exit-address-family  
 !  
 address-family ipv6  
  neighbor FD00:6::240 activate  
  neighbor FD00:6::241 activate  
  network FD00:6::240/127  
  network FD00:6:240::/126  
  network FD00:6:240::4/126  
  no synchronization  
  maximum-paths 2  
 exit-address-family  
!

Note that no BGP peers are up yet - and BGP knows what the problem is, too!

bgp-rr0-s0#show ip bgp  
06:12:15: %SYS-5-CONFIG_I: Configured from console by consoleneigh  
BGP neighbor is 10.6.0.0,  remote AS 65000, internal link  
  BGP version 4, remote router ID 0.0.0.0  
  BGP state = Active  
  Last read 00:03:02, last write 00:03:02, hold time is 180, keepalive interval is 60 seconds  
  Message statistics:  
    InQ depth is 0  
    OutQ depth is 0  
                         Sent       Rcvd  
    Opens:                  0          0  
    Notifications:          0          0  
    Updates:                0          0  
    Keepalives:             0          0  
    Route Refresh:          0          0  
    Total:                  0          0  
  Default minimum time between advertisement runs is 0 seconds  
  
 For address family: IPv4 Unicast  
  BGP table version 1, neighbor version 0/0  
  Output queue size : 0  
  Index 1, Offset 0, Mask 0x2  
  Route-Reflector Client  
  1 update-group member  
                                 Sent       Rcvd  
  Prefix activity:               ----       ----  
    Prefixes Current:               0          0  
    Prefixes Total:                 0          0  
    Implicit Withdraw:              0          0  
    Explicit Withdraw:              0          0  
    Used as bestpath:             n/a          0  
    Used as multipath:            n/a          0  
  
                                   Outbound    Inbound  
  Local Policy Denied Prefixes:    --------    -------  
    Total:                                0          0  
  Number of NLRIs in the update sent: max 0, min 0  
  
  Address tracking is enabled, the RIB does not have a route to 10.6.0.0  
  Address tracking requires at least a /0 route to the peer  
  Connections established 0; dropped 0  
  Last reset never  
  Transport(tcp) path-mtu-discovery is enabled  
  No active TCP connection

Note how it says the RIB does not have a route to 10.6.0.0 - that’s because iBGP doesn’t resolve next-hops for us. Let’s fix it by rolling out an Interior Gateway Protocol (IGP) to support iBGP here. I’m using IS-IS for a few reasons - namely:

Like BGP, one routing protocol for both IPv4 and IPv6
Selective flooding with ISPF
I’m too hip for OSPF now

router isis CLOS-1  
 net 42.0000.0000.0000.0240.00  
 is-type level-2-only  
 ispf level-2  
 log-adjacency-changes  
!  
interface   
ip router isis CLOS-1  
interface Loopback0  
ip router isis CLOS-1

This is applied to every router, while changing the net-ID for each device.It’s fun watching the adjacencies pop up, so I’ll add that here too.

  
*Mar  1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0240 (FastEthernet0/22) Up, new adjacency  
*Mar  1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0241 (FastEthernet0/23) Up, new adjacency   
*Mar  1 06:31:41.565: %BGP-5-ADJCHANGE: neighbor 10.6.0.240 Up  
*Mar  1 06:31:47.169: %BGP-5-ADJCHANGE: neighbor 10.6.0.241 Up

Note how BGP pops up immediately after IS-IS resolves the next-hop for the loopback in this case. Sadly, it doesn’t look like my ancient lab switches support IS-IS for IPv6 - so I’ll add OSPFv3 for the next topic- actually using Clos in a datacenter network

bgp-rr0-s0#show ip bgp sum  
BGP router identifier 10.6.0.240, local AS number 65000  
BGP table version is 3, main routing table version 3  
2 network entries using 234 bytes of memory  
4 path entries using 208 bytes of memory  
3/1 BGP path/bestpath attribute entries using 420 bytes of memory  
0 BGP route-map cache entries using 0 bytes of memory  
0 BGP filter-list cache entries using 0 bytes of memory  
BGP using 862 total bytes of memory  
BGP activity 4/0 prefixes, 6/0 paths, scan interval 60 secs  
  
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
10.6.0.0        4 65000       8       7        3    0    0 00:03:03        1  
10.6.0.1        4 65000       6       5        3    0    0 00:01:51        1  
FD00:6::        4 65000       0       0        0    0    0 never    Active  
FD00:6::1       4 65000       0       0        0    0    0 never    Active

Configurations generated by this lab, if you want to replicate are here.