Spine and Leaf Practical Applications, eBGP
Overview
First off, here's the reference diagram (YAML):
Assumptions about difficulty
Most people I've met outside of the carrier space are pretty intimidated by BGP, as it is truly impressive in scope. Here, we're going to break-out BGP usage into two categories:
- iBGP: This is where all nodes have the same Autonomous system number. A great deal of complexity exists with this deployment model, because BGP's primary loop prevention mechanism is a string with all of the autonomous system numbers to that route, counting each entry as a "hop" as it were.
- eBGP: Every single device has its own ASN. Loops are easy to prevent by simply reading the AS-Path.
eBGP is not very difficult to learn.
This is worthwhile, because BGP has a pretty substantial strength within data center networks, and that is an emphasis on reliability.
I'm not going to be doing a deep-dive on BGP here - but can recommend some truly excellent resources on the subject:
- BGP: Building Reliable Networks with the Border Gateway Protocol, by Iljitsch van Beijnum
- Pretty much anything written by Narbik Kocharians
How is BGP different from IGPs like OSPF, EIGRP?
First, we must examine some key differences between BGP and IGPs:
- IGPs are multicast-based, and dynamically generate peers. BGP is TCP-based and needs statically defined peers (note: you can define a dynamic range, which in a future example will be truly valuable)
- EIGRP has one area, OSPF generally supports up to 16 without getting specific hardware. BGP supports 65,536 with 2-byte ASNs, or 4,294,967,295
- IGPs are designed to trust their routing protocol peers to prevent loops, while BGP is designed to control route advertisement
- IGPs (other than IS-IS, of course) only support IP-based address families, while MP-BGP can support any number of units defined as "Network Layer Reachability Information," making it extensible in numerous ways like EVPN or Segment Routing, or even MPLS. The key thematic point here is that BGP behaves more like a distributed database than a routing protocol would normally.
- IGPs value fast reconvergence, while BGP values reliable reconvergence. It's slow moving, but is extremely change-friendly.
Applying Concepts
In a controlled environment, like a Clos fabric, eBGP is pretty easy to setup, troubleshoot, and maintain. So let's get started!
First, we configure the spines with the appropriate AS and neighbors. It looks like there's a lot going on here, but that's simply because we're running two address-families: IPv4 and IPv6:
1bgp-as65000-s0#conf t
2Enter configuration commands, one per line. End with CNTL/Z.
3bgp-as65000-s0(config)#
4router bgp 65000
5 bgp log-neighbor-changes
6 neighbor 10.6.240.1 remote-as 64900
7 neighbor 10.6.240.1 update-source FastEthernet0/24
8 neighbor 10.6.240.3 remote-as 64901
9 neighbor 10.6.240.3 update-source FastEthernet0/22
10 neighbor FD00:6:240::2 remote-as 64900
11 neighbor FD00:6:240::2 update-source FastEthernet0/24
12 neighbor FD00:6:240::6 remote-as 64901
13 neighbor FD00:6:240::6 update-source FastEthernet0/22
14 maximum-paths 2
15 !
16 address-family ipv4
17 neighbor 10.6.240.1 activate
18 neighbor 10.6.240.3 activate
19 no neighbor FD00:6:240::2 activate
20 no neighbor FD00:6:240::6 activate
21 maximum-paths 2
22 no auto-summary
23 no synchronization
24 exit-address-family
25 !
26 address-family ipv6
27 neighbor FD00:6:240::2 activate
28 neighbor FD00:6:240::6 activate
29 exit-address-family
30
31bgp-as65001-s1#conf t
32Enter configuration commands, one per line. End with CNTL/Z.
33bgp-as65001-s1(config)#
34router bgp 65001
35 bgp log-neighbor-changes
36 neighbor 10.6.241.1 remote-as 64900
37 neighbor 10.6.241.1 update-source FastEthernet0/21
38 neighbor 10.6.241.3 remote-as 64901
39 neighbor 10.6.241.3 update-source FastEthernet0/23
40 neighbor FD00:6:241::2 remote-as 64900
41 neighbor FD00:6:241::2 update-source FastEthernet0/21
42 neighbor FD00:6:241::6 remote-as 64901
43 neighbor FD00:6:241::6 update-source FastEthernet0/23
44 maximum-paths 2
45 !
46 address-family ipv4
47 neighbor 10.6.241.1 activate
48 neighbor 10.6.241.3 activate
49 no neighbor FD00:6:241::2 activate
50 no neighbor FD00:6:241::6 activate
51 maximum-paths 2
52 no auto-summary
53 no synchronization
54 exit-address-family
55 !
56 address-family ipv6
57 neighbor FD00:6:241::2 activate
58 neighbor FD00:6:241::6 activate
59 exit-address-family
And then the leafs:
1
2bgp-as64900-l0#conf t
3Enter configuration commands, one per line. End with CNTL/Z.
4bgp-as64900-l0(config)#
5router bgp 64900
6 bgp log-neighbor-changes
7 neighbor 10.6.240.0 remote-as 65000
8 neighbor 10.6.240.0 update-source FastEthernet1/0/24
9 neighbor 10.6.241.0 remote-as 65001
10 neighbor 10.6.241.0 update-source FastEthernet1/0/21
11 neighbor FD00:6:240::1 remote-as 65000
12 neighbor FD00:6:240::1 update-source FastEthernet1/0/24
13 neighbor FD00:6:241::1 remote-as 65001
14 neighbor FD00:6:241::1 update-source FastEthernet1/0/21
15 maximum-paths 2
16 !
17 address-family ipv4
18 neighbor 10.6.240.0 activate
19 neighbor 10.6.241.0 activate
20 no neighbor FD00:6:240::1 activate
21 no neighbor FD00:6:241::1 activate
22 maximum-paths 2
23 no auto-summary
24 no synchronization
25 exit-address-family
26 !
27 address-family ipv6
28 neighbor FD00:6:240::1 activate
29 neighbor FD00:6:241::1 activate
30 exit-address-family
31
32bgp-as64901-l1#conf t
33Enter configuration commands, one per line. End with CNTL/Z.
34bgp-as64901-l1(config)#
35router bgp 64901
36 bgp log-neighbor-changes
37 neighbor 10.6.240.2 remote-as 65000
38 neighbor 10.6.240.2 update-source FastEthernet0/22
39 neighbor 10.6.241.2 remote-as 65001
40 neighbor 10.6.241.2 update-source FastEthernet0/23
41 neighbor FD00:6:240::5 remote-as 65000
42 neighbor FD00:6:240::5 update-source FastEthernet0/22
43 neighbor FD00:6:241::5 remote-as 65001
44 neighbor FD00:6:241::5 update-source FastEthernet0/23
45 maximum-paths 2
46 !
47 address-family ipv4
48 neighbor 10.6.240.2 activate
49 neighbor 10.6.241.2 activate
50 no neighbor FD00:6:240::5 activate
51 no neighbor FD00:6:241::5 activate
52 maximum-paths 2
53 no auto-summary
54 no synchronization
55 exit-address-family
56 !
57 address-family ipv6
58 neighbor FD00:6:240::5 activate
59 neighbor FD00:6:241::5 activate
60 exit-address-family
We can now verify that all peers are up with both stacks:
1bgp-as65000-s0#show ip bgp sum
2BGP router identifier 10.6.0.240, local AS number 65000
3BGP table version is 1, main routing table version 1
4
5Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
610.6.240.1 4 64900 23 23 1 0 0 00:20:39 0
710.6.240.3 4 64901 19 18 1 0 0 00:17:04 0
8bgp-as65000-s0#show bgp ipv6 unicast summary
9BGP router identifier 10.6.0.240, local AS number 65000
10BGP table version is 1, main routing table version 1
11
12Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
13FD00:6:240::2 4 64900 13 12 1 0 0 00:10:17 0
14FD00:6:240::6 4 64901 9 9 1 0 0 00:06:30 0
We do still have a problem - there are no prefixes received! Let's fix that by adding network statements to all relevant devices. In the demo equipment, the network statement must be an exact match to advertise.
Network statements are not required for interfaces, as in this case, multicast is not used for peer discovery:
1bgp-as64900-l0(config)#router bgp 64900
2bgp-as64900-l0(config-router)#address-family ipv4
3bgp-as64900-l0(config-router-af)#network 10.6.0.0 mask 255.255.255.255
After this is completed, we'll see more routes - note that the above step must be repeated on the spines for all applicable networks, to ensure end to end reachability. This hardware does not appear to support ECMP for IPv6.
1bgp-as64900-l0#show ip bgp sum
2BGP router identifier 10.6.0.0, local AS number 64900
3BGP table version is 13, main routing table version 13
48 network entries using 936 bytes of memory
59 path entries using 468 bytes of memory
68/4 BGP path/bestpath attribute entries using 1120 bytes of memory
76 BGP AS-PATH entries using 144 bytes of memory
80 BGP route-map cache entries using 0 bytes of memory
90 BGP filter-list cache entries using 0 bytes of memory
10BGP using 2668 total bytes of memory
11BGP activity 16/0 prefixes, 22/1 paths, scan interval 60 secs
12
13Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1410.6.240.0 4 65000 47 47 13 0 0 00:40:51 4
1510.6.241.0 4 65001 47 45 13 0 0 00:40:03 4
16
17bgp-as64900-l0#show bgp ipv6 unicast summary
18BGP router identifier 10.6.0.0, local AS number 64900
19BGP table version is 10, main routing table version 10
208 network entries using 1128 bytes of memory
2112 path entries using 912 bytes of memory
228/4 BGP path/bestpath attribute entries using 1120 bytes of memory
236 BGP AS-PATH entries using 144 bytes of memory
240 BGP route-map cache entries using 0 bytes of memory
250 BGP filter-list cache entries using 0 bytes of memory
26BGP using 3304 total bytes of memory
27BGP activity 16/0 prefixes, 22/1 paths, scan interval 60 secs
28
29Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
30FD00:6:240::1 4 65000 44 43 10 0 0 00:37:56 5
31FD00:6:241::1 4 65001 43 43 10 0 0 00:37:18 6
32
33
34bgp-as64900-l0#show ipv6 ro
35IPv6 Routing Table - Default - 11 entries
36Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
37 B - BGP, R - RIP, D - EIGRP, EX - EIGRP external
38 ND - Neighbor Discovery
39 O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
40 ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
41LC FD00:6::/128 [0/0]
42 via Loopback0, receive
43B FD00:6::1/128 [20/0]
44 via FE80::216:C8FF:FE04:4742, FastEthernet1/0/24
45B FD00:6::240/128 [20/0]
46 via FE80::216:C8FF:FE04:4742, FastEthernet1/0/24
47B FD00:6::241/128 [20/0]
48 via FE80::223:4FF:FE42:F3C1, FastEthernet1/0/21
49C FD00:6:240::/126 [0/0]
50 via FastEthernet1/0/24, directly connected
51L FD00:6:240::2/128 [0/0]
52 via FastEthernet1/0/24, receive
53B FD00:6:240::4/126 [20/0]
54 via FE80::216:C8FF:FE04:4742, FastEthernet1/0/24
55C FD00:6:241::/126 [0/0]
56 via FastEthernet1/0/21, directly connected
57L FD00:6:241::2/128 [0/0]
58 via FastEthernet1/0/21, receive
59B FD00:6:241::4/126 [20/0]
60 via FE80::223:4FF:FE42:F3C1, FastEthernet1/0/21
61L FF00::/8 [0/0]
62 via Null0, receive
63bgp-as64900-l0#show ip ro
64Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
65 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
66 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
67 E1 - OSPF external type 1, E2 - OSPF external type 2
68 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
69 ia - IS-IS inter area, * - candidate default, U - per-user static route
70 o - ODR, P - periodic downloaded static route
71
72Gateway of last resort is not set
73
74 10.0.0.0/8 is variably subnetted, 8 subnets, 2 masks
75C 10.6.0.0/32 is directly connected, Loopback0
76B 10.6.0.1/32 [20/0] via 10.6.240.0, 00:12:49
77C 10.6.240.0/31 is directly connected, FastEthernet1/0/24
78B 10.6.0.240/32 [20/0] via 10.6.240.0, 00:10:03
79C 10.6.241.0/31 is directly connected, FastEthernet1/0/21
80B 10.6.0.241/32 [20/0] via 10.6.241.0, 00:07:40
81B 10.6.240.2/31 [20/0] via 10.6.240.0, 00:08:47
82B 10.6.241.2/31 [20/0] via 10.6.241.0, 00:07:40
I have posted the base configs here.
Posts in this Series
- Spine and Leaf Networks, an Outline
- Spine and Leaf Practical Applications, The IP Portability Problem
- Spine and Leaf Practical Applications, EGP and IGP combined!
- Spine and Leaf Practical Applications, eBGP
- Spine and Leaf Practical Applications, OSPF
- Spine and Leaf Practical Applications, RIPv2
- Spine and Leaf Networks, a Topology
- Spine and Leaf Networks, an Introduction