Calico node status

6 MINUTE READ

A Calico node status resource (CalicoNodeStatus) represents a collection of status information for a node that Calico reports back to the user for use during troubleshooting.

As of today, status of BGP agents, BGP sessions and routes exposed to BGP agents are collected from Linux nodes only. Windows nodes are not supported at this time. Calico node status resource is only valid when Calico BGP networking is in use.

Notes

The updating of CalicoNodeStatus will have a small performance impact on CPU/Memory usage of the node as well as adding load to kubernetes apiserver.

In our testing on a ten node, full mesh cluster, a CalicoNodeStatus resource was created for each node where the update interval was set to ten seconds. On each node, this resulted in an increase in CPU use of 5% of a vCPU and an increase of 4MB of memory. The control plane node recorded an increase in CPU usage of 5% of a vCPU for these 10 nodes.

Warning: The implementation of CalicoNodeStatus is designed to handle a small number of nodes (less than 10 is recommended) reporting back status in the same time. If CalicoNodeStatus are created for a large number of nodes, and with short update interval, the kubernetes apiserver may become slower and less responsive.

You should create CalicoNodeStatus for the node you are interested in and for debugging purpose only. CalicoNodeStatus resource should be deleted upon the completion of the debugging process.

Sample YAML

To use this function, the user creates a CalicoNodeStatus object for the node, specifying the information to collect and the interval it should be collected at. This example collects information for node “my-kadm-node-0” with an update interval of 10 seconds.

kubectl apply -f -<<EOF
apiVersion: projectcalico.org/v3
kind: CalicoNodeStatus
metadata:
  name: my-caliconodestatus-1
spec:
  classes:
    - Agent
    - BGP
    - Routes
  node: my-kadm-node-0
  updatePeriodSeconds: 10
EOF

The user then reads back the same resource using the command kubectl get caliconodestatus my-caliconodestatus-1 -o yaml, and the information collected can be found in the status field. For example:

apiVersion: projectcalico.org/v3
kind: CalicoNodeStatus
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"crd.projectcalico.org/v1","kind":"CalicoNodeStatus","metadata":{"annotations":{},"name":"my-caliconodestatus-1"},"spec":{"classes":["Agent","BGP","Routes"],"node":"my-kadm-node-0","updatePeriodSeconds":10}}
    projectcalico.org/metadata: '{"uid":"6e3b676b-546a-41db-aeda-ad44f25ff73f","generation":2,"creationTimestamp":"2021-10-18T10:30:43Z","managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"projectcalico.org/v3","time":"2021-10-18T10:30:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:node":{},"f:updatePeriodSeconds":{}}}},{"manager":"kubectl-edit","operation":"Update","apiVersion":"crd.projectcalico.org/v1","time":"2021-10-18T12:59:25Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:classes":{}}}}]}'
  creationTimestamp: "2021-10-18T10:30:43Z"
  generation: 139
  name: my-caliconodestatus-1
  resourceVersion: "203255"
  uid: 6e3b676b-546a-41db-aeda-ad44f25ff73f
spec:
  classes:
  - Agent
  - BGP
  - Routes
  node: my-kadm-node-0
  updatePeriodSeconds: 10
status:
  agent:
    birdV4:
      lastBootTime: "2021-10-18 09:23:19"
      lastReconfigurationTime: "2021-10-18 09:23:19"
      routerID: 10.128.0.240
      state: Ready
      version: v0.3.3+birdv1.6.8
    birdV6:
      lastBootTime: "2021-10-18 09:23:19"
      lastReconfigurationTime: "2021-10-18 09:23:19"
      routerID: 10.128.0.240
      state: Ready
      version: v0.3.3+birdv1.6.8
  bgp:
    numberEstablishedV4: 2
    numberEstablishedV6: 0
    numberNotEstablishedV4: 1
    numberNotEstablishedV6: 0
    peersV4:
    - peerIP: 10.128.0.242
      since: "09:23:22"
      state: Established
      type: NodeMesh
    - peerIP: 10.128.0.200
      since: "09:23:21"
      state: Established
      type: NodeMesh
    - peerIP: 10.128.0.241
      since: "09:23:22"
      state: OpenSent
      type: NodeMesh
  lastUpdated: "2021-10-18T13:22:03Z"
  routes:
    routesV4:
    - destination: 0.0.0.0/0
      gateway: 172.18.0.1
      interface: eth0
      learnedFrom:
        sourceType: Kernel
      type: FIB
    - destination: 192.168.82.0/26
      gateway: 10.128.0.200
      interface: eth0
      learnedFrom:
        peerIP: 10.128.0.200
        sourceType: NodeMesh
      type: FIB
    - destination: 192.168.36.0/26
      gateway: 10.128.0.242
      interface: eth0
      learnedFrom:
        peerIP: 10.128.0.242
        sourceType: NodeMesh
      type: FIB
    - destination: 10.128.0.5/16
      gateway: N/A
      interface: eth0
      learnedFrom:
        sourceType: Direct
      type: FIB

Calico node status definition

Metadata

Field Description Accepted Values Schema
name The name of this CalicoNodeStatus resource. Required. Alphanumeric string with optional ., _, or -. string

Spec

Field Description Accepted Values Schema
classes List of classes that declare the types of information to monitor for this node and allows for selective status reporting about certain subsets of information. Required. List of Classes list of string
node The node name identifies the Calico node instance to collect node status for. Required. valid Kubernetes node name string
updatePeriodSeconds Period in seconds at which CalicoNodeStatus should be updated. Set to 0 to disable CalicoNodeStatus refresh. Maximum update period is one day. Required. 0 - 86400 int

Status

Field Description Values
lastUpdated Timestamp representing the server time when the CalicoNodeStatus object last updated. It is represented in RFC3339 form and is in UTC.  
agent List of BGP daemon status on the node. ‘birdV4’ holds status of IPv4 bird daemon, ‘birdV6’ holds status of IPv6 bird daemon
bgp List of BGP status on the node. ‘peersV4’ holds status of IPv4 BGP peers, ‘peersV6’ holds status of IPv6 BGP peers
routes List of Route status learned by BGP daemon on the node. ‘routesV4’ holds status of IPv4 routes , ‘routesV6’ holds status of IPv6 routes

Classes

Accepted Values Description
Agent Status of BGP daemon agent
BGP Status of BGP sessions
Routes Status of routes exposed to BGP daemon

BGPDaemonStatus

Field Description Schema Possible Values
state State of BGP daemon. string Ready, NotReady
version Version of BGP daemon string  
routerID Router ID used by BGP daemon string  
lastBootTime Last boot time of BGP daemon string  
lastReconfigurationTime Last reconfiguration time of BGP daemon string  

BGPStatus

Field Description Schema
numberEstablishedV4 The total number of IPv4 established bgp sessions. int
numberNotEstablishedV4 The total number of IPv4 non-established bgp sessions. int
numberEstablishedV6 The total number of IPv6 established bgp sessions. int
numberNotEstablishedV6 The total number of IPv6 non-established bgp sessions. int
peersV4 IPv4 BGP peers status on the node. List of BGP peer
peersV6 IPv6 BGP peers status on the node. List of BGP peer

RouteStatus

Field Description Schema
routesV4 IPv4 routes learned by BGP daemon on the node. List of route
routesV6 IPv6 routes learned by BGP daemon on the node. List of route

BGPPeer

Field Description Schema Possible Values
peerIP IP address of the peer whose condition we are reporting. ip  
type Type indicates whether this peer is configured via the node-to-node mesh, or via en explicit global or per-node BGPPeer object. string NodeMesh, NodePeer, GlobalPeer
state BGP session state. Value of an empty string means BGP session state is unknown. string Idle, Connect, Active, OpenSent, OpenConfirm, Established, Close
since Timestamp of last BGP session state change. time  

Route

Field Description Schema Possible Values
type Type indicates if the route is being used for forwarding or not. string FIB, RIB
destination Destination IP of the route. ip  
gateway Gateway IP of the route. If the route does not involve a gateway, for example a direct route, the value is N/A. ip or N/A  
interface Network interface for the destination. For a blackhole route, the value is blackhole. For a unreachable route, the value is unreachable. string  
learnedFrom Information regarding where this route originated. learned from  

LearnedFrom

Field Description Schema Possible Values
sourceType Type of the source where a route is learned from. string Kernel, Static, Direct, NodeMesh, BGPPeer
peerIP If sourceType is NodeMesh or BGPPeer, IP address of the router that sent us this route. ip