Try out the eBPF tech preview

7 MINUTE READ

Big picture

Install Calico and enable the tech preview of the new eBPF dataplane.

Warning! eBPF mode is a tech preview and should not be used in production clusters. It has had very limited testing and it will contain bugs such as dropping packets or slowing down responses (please report these on the Calico Users slack or Github). In addition, it does not support all the features of Calico and it is missing some security features (for example anti-spoofing protection).

Value

The new eBPF dataplane mode has several advantages over standard linux networking pipeline mode:

  • It scales to higher throughput.
  • It uses less CPU per GBit.
  • It has native support for Kubernetes services (without needing kube-proxy) that:

    • Reduces first packet latency for packets to services.
    • Preserves external client source IP addresses all the way to the pod.
    • Supports DSR (Direct Server Return) for more efficient service routing.
    • Uses less CPU than kube-proxy to keep the dataplane in sync.

Trying out the tech preview will give you a taste of these benefits and an opportunity to give feedback to the Calico team.

To learn more and see performance metrics from our test environment, see the blog, Introducing the Calico eBPF dataplane.

Features

This how-to guide uses the following Calico features:

  • calico/node
  • eBPF dataplane

Concepts

eBPF

eBPF (or “extended Berkeley Packet Filter”), is a technology that allows you to write mini programs that can be attached to various low-level hooks in the Linux kernel, for a wide variety of uses including networking, security, and tracing. You’ll see a lot of non-networking projects leveraging eBPF, but for Calico our focus is on networking, and in particular, pushing the networking capabilities of the latest Linux kernels to the limit.

eBPF mode manifest

To make it easier to try out the tech preview, we have created a custom Kubernetes manifest (.yaml file) with eBPF mode pre-enabled.

Before you begin…

In the tech preview release, eBPF mode has the following pre-requisites:

  • A v5.3+ Linux kernel, for the tech preview release, we have limited our testing to Ubuntu 19.10 so we strongly recommend that you start with that too.
  • An underlying network that doesn’t require Calico to use an overlay. The instructions below guide you through setting up a cluster in a single AWS subnet; an alternative would be to set up your cluster on-prem with a routed network topology.
  • The network must be configured to allow VXLAN packets between Calico hosts.
  • Single-homed hosts; eBPF mode currently assumes a single “main” host IP and interface.
  • IPv4 only. The tech preview release does not support IPv6.
  • Kubernetes API Datastore only.
  • Typha is not supported in the tech preview.
  • The base requirements also apply.

How to

Set up a suitable cluster

We recommend using kubeadm to bootstrap a suitable cluster on AWS.

  1. In AWS, create a controller node and at least 2 worker nodes in the same VPC subnet. Use Ubuntu 19.10 as the image.

  2. Important disable the source/destination check on each node’s interface. This is required in order to use non-overlay networking.

    1. Open the Amazon EC2 console and navigate to “Instances”.

    2. Select the instance and choose Actions > Networking > Change Source/Dest. Check.

    3. Verify that source/destination check is disabled. Otherwise, choose Yes, Disable.

  3. On each node, install kubeadm; we recommend using docker as the container runtime in order to match our test environment as closely as possible.

  4. Create the controller node of a new cluster. On the controller VM, execute:

    sudo kubeadm init --pod-network-cidr 192.168.0.0/16
    
  5. To set up kubectl for the ubuntu user, run:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    

    The final line of the kubeadm init output contains the command for joining your workers to the controller. Run this on each worker, prepending sudo to run it as root. It will look something like this:

    sudo kubeadm join 10.240.0.11:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
    
  6. On the controller, verify that all nodes have joined

    kubectl get nodes
    

    which should output something similar to:

    NAME                                           STATUS      ROLES    AGE   VERSION
    ip-172-16-101-119.us-west-2.compute.internal   NotReady    <none>   86m   v1.17.3
    ip-172-16-101-27.us-west-2.compute.internal    NotReady    <none>   86m   v1.17.3
    ip-172-16-101-49.us-west-2.compute.internal    NotReady    master   88m   v1.17.3
    ip-172-16-102-174.us-west-2.compute.internal   NotReady    <none>   86m   v1.17.3
    

    The nodes will report NotReady until we complete the installation of Calico below.

Install Calico on nodes

For the tech preview, only the Kubernetes API Datastore is supported. Since Calico replaces kube-proxy in eBPF mode, it requires the IP and port of your API server to be set in its config map.

  1. Download the following Calico install manifest.

    curl https://docs.projectcalico.org/manifests/calico-bpf.yaml -O
    

    The manifest is configured to:

    • use the Kubernetes API Datastore
    • turn on BPF mode
    • use no encapsulation (which requires using a single subnet in AWS).
  2. Find the real IP and port of your API server. One way to do this is to run the following command:
    kubectl get endpoints kubernetes
    

    The output should look like this:

    kubectl get endpoints kubernetes
    NAME         ENDPOINTS           AGE
    kubernetes   <IP>:<port>         43h
    

    Record the <IP> and <port>.

  3. Modify the kubernetes_service_host and kubernetes_service_port variables in the config map at the top of the manifest. Set kubernetes_service_host to the IP you recorded above. Set kubernetes_service_port to the port.
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: calico-config
      namespace: kube-system
    data:
      ...
      kubernetes_service_host: "<IP>"
      kubernetes_service_port: "<port>"
      ...
    
  4. If the underlying network MTU of your cluster is less then 1460, set the veth_mtu configuration parameter to your MTU - 50 (to allow for the VXLAN header).

    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: calico-config
      namespace: kube-system
    data:
      ...
      veth_mtu: "<MTU>"
      ...
    
  5. Disable kube-proxy on your cluster. The following command adds a node selector to the kube-proxy daemon set that won’t match anything:

    kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}'
    

    Since the selector won’t match anything, kube-proxy won’t run anywhere but it can be easily re-enabled by removing the selector.

  6. Apply the Calico manifest using the following command.

    kubectl apply -f calico-bpf.yaml
    
  7. Verify that the calico-node pods start and become ready and that the kube-proxy pods shut down:
    watch kubectl get po -n kube-system
    

    Should give output like this:

    NAME                                            READY   STATUS    RESTARTS   AGE
    calico-kube-controllers-78d64db6fd-dlqwm        1/1     Running   0          28m
    calico-node-6ffzz                               1/1     Running   0          13m
    calico-node-9ccpx                               1/1     Running   0          13m
    calico-node-mx2v4                               1/1     Running   0          13m
    calico-node-n9pc9                               1/1     Running   0          13m
    coredns-5644d7b6d9-bpddg                        1/1     Running   0          46m
    coredns-5644d7b6d9-glfr7                        1/1     Running   0          46m
    etcd-host-name                                  1/1     Running   0          46m
    kube-apiserver-host-name                        1/1     Running   0          46m
    kube-controller-manager-host-name               1/1     Running   0          46m
    kube-scheduler-host-name                        1/1     Running   0          46m
    

Try out DSR mode

Direct return mode skips a hop through the network for traffic to services (such as node ports) from outside the cluster. This reduces latency and CPU overhead but it requires the underlying network to allow nodes to send traffic with each other’s IPs. In AWS, this requires all your nodes to be in the same subnet and for the source/dest check to be disabled.

DSR mode is disabled by default; to enable it, set the BPFExternalServiceMode felix configuration parameter to "DSR". One way to do that is to set the corresponding environment variable on the calico-node DaemonSet:

kubectl set env -n kube-system ds/calico-node FELIX_BPFExternalServiceMode="DSR"

Toggle between eBPF and the standard linux networking pipeline

After following the above instructions you’ll have a cluster running in eBPF mode. If you’d like to switch to the standard linux networking pipeline mode for comparison purposes:

  1. In the manifest above, eBPF mode is configured with an environment variable. To disable it, you can patch the calico-node daemon set:
    kubectl set env -n kube-system ds/calico-node FELIX_BPFENABLED="false"
    
  2. To re-enable the Kubernetes kube-proxy you can use the following command to reverse the node selector change we made above:
    kubectl patch ds -n kube-system kube-proxy --type merge -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": null}}}}}'
    

To re-enable BPF mode:

  1. Disable kube-proxy on your cluster. The following command adds a node selector to the kube-proxy daemon set that won’t match anything:

    kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}'
    
  2. Re-enable eBPF mode:

    kubectl set env -n kube-system ds/calico-node FELIX_BPFENABLED="true"
    

Warning! Switching between eBPF and standard linux networking can cause long-lived flows to be silently dropped since the two dataplane modes do not share connection-tracking state.

Send us feedback

We want to hear about your experience, so please don’t hesitate to connect with us via the Calico Users Slack group.