Determine best networking option
Calico supports several container networking options for scalability, network performance, and interoperability with existing infrastructure.
Different network implementations are better suited to different environments. Calico provides several networking implementations based on IP routing without the need for encapsulation. If your deployment requires encapsulation, Calico provides overlay networking (IP in IP or VXLAN). Calico also supports enforcing policy with a number of other Kubernetes networking options. This document helps you choose the best networking option for your cluster.
About Calico networking
Calico provides a few ways to allow pods to connect to other pods, to the host, and to outside networks (for example, the internet).
- Assigns IP addresses to pods using Calico’s IP address management (IPAM)
- Programs the local node’s routing table
- Distributes routes to other nodes and network devices
For details of Calico network design and architecture, see a lightboard presentation on the Calico dataplane, components, and how traffic is routed, Calico network architecture and fundamentals.
Calico optionally supports using the Border Gateway Protocol (BGP) for sharing routing information into the network. Calico supports cloud deployments with full node-to-node mesh (with and without route reflectors), and on-premises deployments with BGP peering directly to Top of Rack (ToR) routers; allowing traffic to be routed directly to your workloads without needing NAT or encapsulation.
Calico can also use selective VXLAN encapsulation for workload traffic in cloud deployments without the need for BGP.
Other Kubernetes networking options
Calico can perform network policy enforcement with a number of other Kubernetes networking options.
Amazon AWS VPC CNI
The Amazon VPC CNI plugin uses AWS elastic network interfaces to provide pod networking. It is the default networking used in Amazon EKS, with Calico for network policy enforcement.
The Azure CNI plugin configures the Azure virtual network to provide pod networking. It is the default networking used in Microsoft AKS, with Calico for network policy enforcement.
Flannel routes pod traffic using static per-node CIDRs. It provides a number of networking backends. Calico can be used for network policy enforcement.
Google cloud networking
The table below shows common networking options when using Calico.
|Networking Option||Suitable Environments||Dataplane Performance and Visibility||Setup Complexity||Notes|
|Calico, Unencapsulated, peered with physical infrastructure||On-prem||Best||Moderate||Allows pods to be directly accessed from outside the cluster|
|Calico, Unencapsulated, not peered with physical infrastructure||On-prem L2 networks, AWS, Azure||Best||Low||IP in IP or VXLAN can be added for cross-subnet traffic|
|Calico, Encapsulated, IPIP||On-prem, most public clouds other than Azure||Good to excellent depending on NIC hardware capabilities||Low|
|Calico, Encapsulated, VXLAN||On-prem, any public cloud||Good to excellent depending on NIC hardware capabilities||Low|
|AWS VPC CNI||Amazon EKS||Excellent||Low||Does not support full Calico IPAM feature set, limited to AWS.|
|Azure CNI||Microsoft AKS||Excellent||Low||Does not support full Calico IPAM feature set, limited to Azure.|
|Google cloud||Google GKE||Excellent||Low||Does not support full Calico IPAM feature set, limited to GCP.|
|Flannel||Any public cloud||Poor to excellent, depending on chosen backend||Medium||Does not support full Calico IPAM feature set.|
This section provides more details on Calico’s built-in networking options:
- Unencapsulated, peered with physical infrastructure
- Unencapsulated, not peered with physical infrastructure
- IP in IP or VXLAN encapsulation
Unencapsulated, peered with physical infrastructure
Calico can peer with your routers using BGP. This provides great performance and easy debugging of unencapsulated traffic, and a wide range of options for network topology and connectivity. On top of the advantages of non-peered unencapsulated traffic:
- Your cluster can span multiple L2 subnets without needing encapsulation
- Resources outside your cluster can talk directly to your pods without NAT
- You can even expose pods directly to the internet if you want!
To configure BGP peering and determine the right topology, see Configure BGP peering. This option requires the ability to configure BGP peers on your routers. If this is not an option, see the next section.
Unencapsulated, not peered with physical infrastructure
This option also provides near host-to-host levels of performance and allows the network direct visibility of traffic.
Calico can route pod traffic between nodes without encapsulation when all nodes are on a single L2 subnet, and if the underlying network doesn’t enforce IP address checks. If your network consists of multiple L2 subnets then you can either peer over BGP with your routers, or use cross-subnet encapsulation to encapsulate only traffic that crosses subnet boundaries.
Traffic cannot be routed between pods and destinations that aren’t also in the Calico cluster without allowing workload access outside cluster, or peering with infrastructure.
Tip: On AWS, you can disable source/destination checking to use this option within a VPC subnet. Can I run Calico in a public cloud environment?.
IP in IP or VXLAN encapsulation
If possible, we recommend running Calico without network overlay/encapsulation. This provides the highest performance and simplest network; the packet that leaves your workload is the packet that goes on the wire.
However, selectively using overlays (IP in IP or VXLAN) can be useful when running on top of an underlying network that cannot easily be made aware of workload IPs. Calico can perform encapsulation on: all traffic, no traffic, or only on traffic that crosses a subnet boundary.
IP in IP or VXLAN encapsulation can also be used selectively between subnets – this provides the performance benefits of unencapsulated traffic within subnets, for environments where the fabric contains multiple L2 networks and peering isn’t available. For example, if you are using Calico networking in AWS across multiple VPCs/subnets, Calico can selectively encapsulate only the traffic that is routed between the VPCs/subnets, and run without encapsulation within each VPC/subnet. For help, see Overlay networking.
Above and beyond
- Big picture
- Above and beyond