Spine and Leaf Practical Applications, EGP and IGP combined!
So far, all examples to date have been extremely simple, and stand rather well on their own (OSPF Clos would work just FINE in a campus network if you have cheap L3) but may not effectively address some more advanced use cases.
In short, we're about to enter niche territory.
Generally speaking, BGP is regarded as the most advanced networking topic out there, until it isn't. Most of the complexity lies in iBGP (same Autonomous System everywhere), because BGP's primary loop prevention mechanism is AS-Path (count on the number of AS this route transits).
To successfully implement iBGP (a staple of every carrier, so totally do-able), a network engineer must choose one of the following paths (non-inclusive list):
- Fully mesh all BGP speakers with each other (factorial scaling)
- Implement a Route Reflector: https://en.wikipedia.org/wiki/Route_reflector
In this case, the more scalable option is to opt for a route reflector, but it isn't that easy.
iBGP, when compared to eBGP:
- Doesn't care about hop count to a peer speaker. As long as the route reflector is less than 255 hops away, there is no issue
- Usually doesn't care about resolving paths to peer speakers - that's a problem for something else.
To provide a good example of an expandable, scalable fabric that can offer eBGP as a service to subtending network devices, we will implement IS-IS as the intra-fabric routing protocol, and then leverage iBGP with the spine switches as route reflectors.
First things first, diagram is here: (YAML)
First, we'd configure the spines. Note that future releases (my home lab is rockin' IOS 12.2.55, too old for this) the BGP Dynamic Neighbor that comes with more modern network operating systems is really useful.
Note: route reflector client status is configured on the SERVER side:
1router bgp 65000
2 bgp log-neighbor-changes
3 neighbor 10.6.0.0 remote-as 65000
4 neighbor 10.6.0.0 update-source L0
5 neighbor 10.6.0.0 route-reflector-client
6 neighbor 10.6.0.1 remote-as 65000
7 neighbor 10.6.0.1 update-source L0
8 neighbor 10.6.0.1 route-reflector-client
9 neighbor FD00:6::0 remote-as 65000
10 neighbor FD00:6::0 update-source L0
11 neighbor FD00:6::0 route-reflector-client
12 neighbor FD00:6::1 remote-as 65000
13 neighbor FD00:6::1 update-source L0
14 neighbor FD00:6::1 route-reflector-client
15 maximum-paths 2
16 !
17 address-family ipv4
18 neighbor 10.6.0.0 activate
19 neighbor 10.6.0.1 activate
20 maximum-paths 2
21 no auto-summary
22 network 10.6.0.240 mask 255.255.255.254
23 network 10.6.240.0 mask 255.255.255.254
24 network 10.6.240.2 mask 255.255.255.254
25 exit-address-family
26 !
27 address-family ipv6
28 neighbor FD00:6::0 activate
29 neighbor FD00:6::1 activate
30 network FD00:6::240/127
31 network FD00:6:240::/126
32 network FD00:6:240::4/126
33 no synchronization
34 maximum-paths 2
35 exit-address-family
36!
And then the leaf configuration:
1
2router bgp 65000
3 bgp log-neighbor-changes
4 neighbor 10.6.0.240 remote-as 65000
5 neighbor 10.6.0.240 update-source L0
6 neighbor 10.6.0.241 remote-as 65000
7 neighbor 10.6.0.241 update-source L0
8 neighbor FD00:6::240 remote-as 65000
9 neighbor FD00:6::240 update-source L0
10 neighbor FD00:6::241 remote-as 65000
11 neighbor FD00:6::241 update-source L0
12 maximum-paths 2
13 !
14 address-family ipv4
15 neighbor 10.6.0.240 activate
16 neighbor 10.6.0.241 activate
17 maximum-paths 2
18 no auto-summary
19 network 10.6.0.240 mask 255.255.255.254
20 network 10.6.240.0 mask 255.255.255.254
21 network 10.6.240.2 mask 255.255.255.254
22 exit-address-family
23 !
24 address-family ipv6
25 neighbor FD00:6::240 activate
26 neighbor FD00:6::241 activate
27 network FD00:6::240/127
28 network FD00:6:240::/126
29 network FD00:6:240::4/126
30 no synchronization
31 maximum-paths 2
32 exit-address-family
33!
Note that no BGP peers are up yet - and BGP knows what the problem is, too!
1bgp-rr0-s0#show ip bgp
206:12:15: %SYS-5-CONFIG_I: Configured from console by consoleneigh
3BGP neighbor is 10.6.0.0, remote AS 65000, internal link
4 BGP version 4, remote router ID 0.0.0.0
5 BGP state = Active
6 Last read 00:03:02, last write 00:03:02, hold time is 180, keepalive interval is 60 seconds
7 Message statistics:
8 InQ depth is 0
9 OutQ depth is 0
10 Sent Rcvd
11 Opens: 0 0
12 Notifications: 0 0
13 Updates: 0 0
14 Keepalives: 0 0
15 Route Refresh: 0 0
16 Total: 0 0
17 Default minimum time between advertisement runs is 0 seconds
18
19 For address family: IPv4 Unicast
20 BGP table version 1, neighbor version 0/0
21 Output queue size : 0
22 Index 1, Offset 0, Mask 0x2
23 Route-Reflector Client
24 1 update-group member
25 Sent Rcvd
26 Prefix activity: ---- ----
27 Prefixes Current: 0 0
28 Prefixes Total: 0 0
29 Implicit Withdraw: 0 0
30 Explicit Withdraw: 0 0
31 Used as bestpath: n/a 0
32 Used as multipath: n/a 0
33
34 Outbound Inbound
35 Local Policy Denied Prefixes: -------- -------
36 Total: 0 0
37 Number of NLRIs in the update sent: max 0, min 0
38
39 Address tracking is enabled, the RIB does not have a route to 10.6.0.0
40 Address tracking requires at least a /0 route to the peer
41 Connections established 0; dropped 0
42 Last reset never
43 Transport(tcp) path-mtu-discovery is enabled
44 No active TCP connection
Note how it says the RIB does not have a route to 10.6.0.0 - that's because iBGP doesn't resolve next-hops for us. Let's fix it by rolling out an Interior Gateway Protocol (IGP) to support iBGP here. I'm using IS-IS for a few reasons - namely:
- Like BGP, one routing protocol for both IPv4 and IPv6
- Selective flooding with ISPF
- I'm too hip for OSPF now
1router isis CLOS-1
2 net 42.0000.0000.0000.0240.00
3 is-type level-2-only
4 ispf level-2
5 log-adjacency-changes
6!
7interface
8ip router isis CLOS-1
9interface Loopback0
10ip router isis CLOS-1
This is applied to every router, while changing the net-ID for each device.It's fun watching the adjacencies pop up, so I'll add that here too.
1
2*Mar 1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0240 (FastEthernet0/22) Up, new adjacency
3*Mar 1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0241 (FastEthernet0/23) Up, new adjacency
4*Mar 1 06:31:41.565: %BGP-5-ADJCHANGE: neighbor 10.6.0.240 Up
5*Mar 1 06:31:47.169: %BGP-5-ADJCHANGE: neighbor 10.6.0.241 Up
Note how BGP pops up immediately after IS-IS resolves the next-hop for the loopback in this case. Sadly, it doesn't look like my ancient lab switches support IS-IS for IPv6 - so I'll add OSPFv3 for the next topic- actually using Clos in a datacenter network
1bgp-rr0-s0#show ip bgp sum
2BGP router identifier 10.6.0.240, local AS number 65000
3BGP table version is 3, main routing table version 3
42 network entries using 234 bytes of memory
54 path entries using 208 bytes of memory
63/1 BGP path/bestpath attribute entries using 420 bytes of memory
70 BGP route-map cache entries using 0 bytes of memory
80 BGP filter-list cache entries using 0 bytes of memory
9BGP using 862 total bytes of memory
10BGP activity 4/0 prefixes, 6/0 paths, scan interval 60 secs
11
12Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1310.6.0.0 4 65000 8 7 3 0 0 00:03:03 1
1410.6.0.1 4 65000 6 5 3 0 0 00:01:51 1
15FD00:6:: 4 65000 0 0 0 0 0 never Active
16FD00:6::1 4 65000 0 0 0 0 0 never Active
Configurations generated by this lab, if you want to replicate are here.
Posts in this Series
- Spine and Leaf Networks, an Outline
- Spine and Leaf Practical Applications, The IP Portability Problem
- Spine and Leaf Practical Applications, EGP and IGP combined!
- Spine and Leaf Practical Applications, eBGP
- Spine and Leaf Practical Applications, OSPF
- Spine and Leaf Practical Applications, RIPv2
- Spine and Leaf Networks, a Topology
- Spine and Leaf Networks, an Introduction