Will you be at KubeCon this year? Join us for CalicoCon 2019!

Changing IP pools

About changing IP pools

When using Calico IPAM, each workload is assigned an address from the selection of configured IP pools. You may want to modify the CIDR of the IP pool of a running cluster for one of the following reasons:

  • To move to a larger CIDR that can accommodate more workloads.
  • To move off of a CIDR that was used accidentally.

Purpose of this page

Provide guidance on how to change from one IP pool CIDR to another on a running cluster.

Prerequisites

Calico IPAM

This guide only applies if you are using Calico IPAM.

Orchestrator support

While Calico supports changing IP pools, not all orchestrators do. Be sure to consult the documentation of the orchestrator you are using to ensure it supports changing the workload CIDR.

For example, in Kubernetes, all three of the following arguments must be equal to, or contain, the Calico IP pool CIDRs:

  • kube-apiserver: --pod-network-cidr
  • kube-proxy: --cluster-cidr
  • kube-controller-manager: --cluster-cidr

OpenShift does not support changing the pod network CIDR (as per their documentation on the osm_cluster_network_cidr configuration field.

Application availability impact

This process will require the recreation of all Calico-networked workloads, which will have some impact on the availability of your applications.

Consequences of deleting an IP pool without following this migration procedure

Removing an IP pool without following this migration procedure can cause network connectivity disruptions in any running workloads with addresses from that IP pool. Namely:

  • If IP-in-IP or VXLAN was enabled on the IP pool, those workloads will no longer have their traffic encapsulated.
  • If nat-outgoing was enabled on the IP pool, those workloads will no longer have their traffic NAT’d.
  • If using Calico BGP routing, routes to pods will no longer be aggregated.

Changing an IP pool

The basic process is as follows:

  1. Add a new IP pool.
  2. Disable the old IP pool. This prevents new IPAM allocations from the old IP pool without affecting the networking of existing workloads.
  3. Recreate all existing workloads that were assigned an address from the old IP pool.
  4. Remove the old IP pool.

Example: Kubernetes

In this example, we created a cluster with kubeadm. We wanted the pods to use IPs in the range 10.0.0.0/16 so we set --pod-network-cidr=10.0.0.0/16 when running kubeadm init. However, we installed Calico without setting the default IP pool to match. Running calicoctl get ippool -o wide shows Calico created its default IP pool of 192.168.0.0/16:

NAME                  CIDR             NAT    IPIPMODE   VXLANMODE   DISABLED
default-ipv4-ippool   192.168.0.0/16   true   Always     Never       false

Based on the output of calicoctl get wep --all-namespaces, we see kube-dns has already been allocated an address from the wrong range:

NAMESPACE     WORKLOAD                   NODE      NETWORKS            INTERFACE
kube-system   kube-dns-6f4fd4bdf-8q7zp   vagrant   192.168.52.130/32   cali800a63073ed

Let’s get started.

  1. Add a new IP pool:

    calicoctl create -f -<<EOF
    apiVersion: projectcalico.org/v3
    kind: IPPool
    metadata:
      name: new-pool
    spec:
      cidr: 10.0.0.0/16
      ipipMode: Always
      natOutgoing: true
    EOF
    

    We should now have two enabled IP pools, which we can see when running calicoctl get ippool -o wide:

    NAME                  CIDR             NAT    IPIPMODE   DISABLED
    default-ipv4-ippool   192.168.0.0/16   true   Always     false
    new-pool              10.0.0.0/16      true   Always     false
    
  2. Disable the old IP pool.

    First save the IP pool definition to disk:

    calicoctl get ippool -o yaml > pool.yaml
    

    pool.yaml should look like this:

    apiVersion: projectcalico.org/v3
    items:
    - apiVersion: projectcalico.org/v3
      kind: IPPool
      metadata:
        name: default-ipv4-ippool
      spec:
        cidr: 192.0.0.0/16
        ipipMode: Always
        natOutgoing: true
    - apiVersion: projectcalico.org/v3
      kind: IPPool
      metadata:
        name: new-pool
      spec:
        cidr: 10.0.0.0/16
        ipipMode: Always
        natOutgoing: true
    

    Note: Some extra cluster-specific information has been redacted to improve readibility.

    Edit the file, adding disabled: true to the default-ipv4-ippool IP pool:

    apiVersion: projectcalico.org/v3
    kind: IPPool
    metadata:
      name: default-ipv4-ippool
    spec:
      cidr: 192.0.0.0/16
      ipipMode: Always
      natOutgoing: true
      disabled: true
    

    Apply the changes:

    calicoctl apply -f pool.yaml
    

    We should see the change reflected in the output of calicoctl get ippool -o wide:

    NAME                  CIDR             NAT    IPIPMODE   DISABLED
    default-ipv4-ippool   192.168.0.0/16   true   Always     true
    new-pool              10.0.0.0/16      true   Always     false
    
  3. Recreate all existing workloads using IPs from the disabled pool. In this example, kube-dns is the only workload networked by Calico:

    kubectl delete pod -n kube-system kube-dns-6f4fd4bdf-8q7zp
    

    Check that the new workload now has an address in the new IP pool by running calicoctl get wep --all-namespaces:

    NAMESPACE     WORKLOAD                   NODE      NETWORKS            INTERFACE
    kube-system   kube-dns-6f4fd4bdf-8q7zp   vagrant   10.0.24.8/32   cali800a63073ed
    
  4. Delete the old IP pool:

    calicoctl delete pool default-ipv4-ippool
    

Next Steps

For more information on the structure of the IP pool resource, see the IP pools reference.