Configure BGP peering
We have configured Calico to distribute routing information over the Border Gateway Protocol (BGP). This scalable protocol powers routing on the global public Internet.
In many on-premise data centers, each server connects to a top-of-rack (ToR) router operating at the IP layer (layer 3). In that situation, we would need to peer each node with its corresponding ToR router, so that the ToR learns routes to the containers. That configuration is beyond the scope of this guide.
Since we are running in an AWS VPC within a single subnet, the hosts have ethernet (layer 2) connectivity with one another, meaning there are no routers between them. Thus, they can peer directly with each other.
On one of the nodes in your cluster where you have
calicoctl installed, check the status.
sudo calicoctl node status
Notice there are four BGP sessions, one to each other node in the cluster. In a small cluster, this works well and is highly resilient. However, the total number of BGP sessions scales as the square of the number of nodes, and in a large cluster this creates a lot of overhead.
In this lab we will configure a fixed number of route reflectors. Route reflectors annouce their own routes and the routes they receive from other peers. This means nodes only need to peer with the route reflectors to get all the routes in the cluster. This peering arrangement means that the number of BGP sessions scales linearly with the number of nodes.
Choose and label nodes
We will establish three route reflectors, which means we avoid a single point of failure even if we take down a route reflector node for maintenance. In a five node cluster that means that only one BGP session is not needed, since the two non-reflector nodes don’t need to peer with one another, but it will save lots of overhead in a large cluster.
Choose three nodes and perform the following for each of them.
Save the node YAML.
calicoctl get node <node name> -o yaml --export > node.yaml
Edit the YAML to add
metadata: labels: calico-route-reflector: "" spec: bgp: routeReflectorClusterID: 18.104.22.168
Reapply the YAML
calicoctl apply -f node.yaml
Configure all non-reflector nodes to peer with all route reflectors
calicoctl apply -f - <<EOF kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: peer-to-rrs spec: nodeSelector: "!has(calico-route-reflector)" peerSelector: has(calico-route-reflector) EOF
Configure all route reflectors to peer with each other
calicoctl apply -f - <<EOF kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: rrs-to-rrs spec: nodeSelector: has(calico-route-reflector) peerSelector: has(calico-route-reflector) EOF
Disable the node-to-node mesh
calicoctl create -f - << EOF apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: nodeToNodeMeshEnabled: false asNumber: 64512 EOF
On a non-reflector node, you should now see only three peerings.
sudo calicoctl node status