Apply policy to services exposed externally as cluster IPs

Big picture

Control access to services exposed through clusterIPs that are advertised outside the cluster using BGP.

Value

Calico network policy uses standard Kubernetes Services that allow you to expose services within clusters to external clients in the following ways:

Features

This how-to guide uses the following Calico features:

  • Calico cluster IP advertisement
  • HostEndpoints
  • GlobalNetworkPolicy
    • applyOnForward
    • preDNAT
  • NetworkPolicy

Concepts

A cluster IP is a virtual IP address that represents a Kubernetes Service. Kube Proxy on each host translates the clusterIP into a pod IP for one of the pods backing the service, acting as a reverse proxy and load balancer.

Cluster IPs were originally designed for use within the Kubernetes cluster. Calico allows you to advertise Cluster IPs externally – so external clients can use them to access services hosted inside the cluster. This means that Calico ingress policy can be applied at one or both of the following locations:

  • Host interface, when the traffic destined for the clusterIP first ingresses the cluster
  • Pod interface of the backend pod

Traffic routing: local versus cluster modes

Calico implements Kubernetes service external traffic policy, which controls whether external traffic is routed to node-local or cluster-wide endpoints. The following table summarizes key differences between these settings. The default is cluster mode.

Service setting Traffic is load balanced… Pros and cons Required service type
externalTrafficPolicy: Cluster(default) Across all nodes in the cluster Equal distribution of traffic among all pods running a service.

Possible unnecessary network hops between nodes for ingress external traffic.When packets are rerouted to pods on another node, traffic is SNAT’d (source network address translation).

Destination pod can see the proxying node’s IP address rather than the actual client IP.
ClusterIP
externalTrafficPolicy: Local Across the nodes with the endpoints for the service Avoids extra hops so better for apps that ingress a lot external traffic.

Traffic is not SNAT’d so actual client IPs are preserved.

Traffic distributed among pods running a service may be imbalanced.
LoadBalancer (for cloud providers), or NodePort (for node’s static port)

Before you begin…

Configure Calico to advertise cluster IPs over BGP.

How to

Selecting which mode to use depends on your goals and resources. At an operational level, local mode simplifies policy, but load balancing may be uneven in certain scenarios. Cluster mode requires more work to manage clusterIPs, SNAT, and create policies that reference specific IP addresses, but you always get even load balancing.

Secure externally exposed cluster IPs, local mode

Using local mode, the original source address of external traffic is preserved, and you can define policy directly using standard Calico network policy.

  1. Create Calico NetworkPolicies or GlobalNetworkPolicies that select the same set of pods as your Kubernetes Service.
  2. Add rules to allow the external traffic.
  3. If desired, add rules to allow in-cluster traffic.

Secure externally exposed cluster IPs, cluster mode

In the following steps, we define GlobalNetworkPolicy and HostEndpoints.

Step 1: Verify Kubernetes Service manifest

Ensure that your Kubernetes Service manifest explicitly lists the clusterIP; do not allow Kubernetes to automatically assign the clusterIP because you need it for your policies in the following steps.

Step 2: Create global network policy at the host interface

In this step, you create a GlobalNetworkPolicy that selects all host endpoints. It controls access to the cluster IP, and prevents unauthorized clients from outside the cluster from accessing it. The hosts then forwards only authorized traffic.

Set policy to allow external traffic for cluster IPs

Add rules to allow the external traffic for each clusterIP. The following example allows connections to two cluster IPs. Make sure you add applyOnForward and preDNAT rules.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-cluster-ips
spec:
  selector: k8s-role == 'node'
  types:
  - Ingress
  applyOnForward: true
  preDNAT: true
  ingress:
  # Allow 50.60.0.0/16 to access Cluster IP A
  - action: Allow
    source:
      nets:
      - 50.60.0.0/16
    destination:
      nets:
      - 10.20.30.40/32 # Cluster IP A
  # Allow 70.80.90.0/24 to access Cluster IP B
  - action: Allow
    source:
      nets:
      - 70.80.90.0/24
    destination:
      nets:
      - 10.20.30.41/32 # Cluster IP B

Add a rule to allow traffic destined for the pod CIDR

Without this rule, normal pod-to-pod traffic is blocked because the policy applies to forwarded traffic.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-to-pods
spec:
  selector: k8s-role == 'node'
  types:
  - Ingress
  applyOnForward: true
  preDNAT: true
  ingress:
  # Allow traffic forwarded to pods
  - action: Allow
    destination:
      nets:
      - 192.168.0.0/16 # Pod CIDR

Add a rule to allow traffic destined for all host endpoints

Or, you can add rules that whitelist specific host traffic including Kubernetes and Calico. Without this rule, normal host traffic is blocked.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-traffic-hostendpoints
spec:
  selector: k8s-role == 'node'
  types:
  - Ingress
  # Allow traffic to the node (not nodePorts, TCP)
  - action: Allow
    protocol: TCP
    destination:
      selector: k8s-role == 'node'
      notPorts: ["30000:32767"] # nodePort range
  # Allow traffic to the node (not nodePorts, UDP)
  - action: Allow
    protocol: UDP
    destination:
      selector: k8s-role == 'node'
      notPorts: ["30000:32767"] # nodePort range
Step 3: Create a global network policy that selects pods

In this step, you create a GlobalNetworkPolicy that selects the same set of pods as your Kubernetes Service. Add rules that allow host endpoints to access the service ports.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-nodes-svc-a
spec:
  selector: k8s-svc == 'svc-a'
  types:
  - Ingress
  ingress:
  - action: Allow
    protocol: TCP
    source:
      selector: k8s-role == 'node'
    destination:
      ports: [80, 443]
  - action: Allow
    protocol: UDP
    source:
      selector: k8s-role == 'node'
    destination:
      ports: [80, 443]
Step 4: (Optional) Create network polices or global network policies that allow in-cluster traffic to access the service
Step 5: Create HostEndpoints

Create HostEndpoints for the interface of each host that will receive traffic for the clusterIPs. Be sure to label them so they are selected by the policy in Step 2 (Add a rule to allow traffic destined for the pod CIDR), and the rules in Step 3.

In the previous example policies, the label k8s-role: node is used to identify these HostEndpoints.

Above and beyond