Customizing the manifests
About customizing manifests
We provide a number of manifests to make deployment of Calico easy. You can optionally modify the manifests before applying them. Or you can modify the manifest and reapply it to change settings as needed.
Refer to the section that corresponds to the manifest you wish to modify for more details.
Customizing Calico manifests
About customizing Calico manifests
Each manifest contains all the necessary resources for installing Calico on each node in your Kubernetes cluster.
It installs the following Kubernetes resources:
- Installs the
calico/nodecontainer on each host using a DaemonSet.
- Installs the Calico CNI binaries and network config on each host using a DaemonSet.
calico/kube-controllersas a deployment.
calico-etcd-secretssecret, which optionally allows for providing etcd TLS assets.
calico-configConfigMap, which contains parameters for configuring the install.
The sections that follow discuss the configurable parameters in greater depth.
Configuring the pod IP range
Calico IPAM assigns IP addresses from IP pools.
To change the default IP range used for pods, modify the
section of the
calico.yaml manifest. For more information, see
By default, the manifests enable IP-in-IP encapsulation across subnets. Many users may want to disable IP-in-IP encapsulation, such as under the following circumstances.
- Their cluster is running in a properly configured AWS VPC.
- All their Kubernetes nodes are connected to the same layer 2 network.
- They intend to use BGP peering to make their underlying infrastructure aware of pod IP addresses.
To disable IP-in-IP encapsulation, modify the
CALICO_IPV4POOL_IPIP section of the
manifest. For more information, see Configuring calico/node.
Switching from IP-in-IP to VXLAN
By default, the Calico manifests enable IP-in-IP encapsulation. If you are on a network that blocks IP-in-IP, such as Azure, you may wish to switch to Calico’s VXLAN encapsulation mode. To do this at install time (so that Calico creates the default IP pool with VXLAN and no IP-in-IP configuration has to be undone):
- Start with one of the Calico for policy and networking manifests.
- Replace environment variable name
CALICO_IPV4POOL_VXLAN. Leave the value of the new variable as “Always”.
- Optionally, (to save some resources if you’re running a VXLAN-only cluster) completely disable Calico’s BGP-based
calico_backend: "vxlan". This disables BIRD.
- Comment out the line
- -bird-readyfrom the calico/node readiness check (otherwise disabling BIRD will cause the readiness check to fail on every node):
readinessProbe: exec: command: - /bin/calico-node # - -bird-ready - -felix-ready
For more information on calico/node’s configuration variables, including additional VXLAN settings, see Configuring calico/node.
CALICO_IPV4POOL_VXLANenvironment variable only takes effect when the first calico/node to start creates the default IP pool. It has no effect after the pool has already been created. To switch to VXLAN mode after installation time, use calicoctl to modify the IPPool resource.
By default, these manifests do not configure secure access to etcd and assume an etcd proxy is running on each host. The following configuration options let you specify custom etcd cluster endpoints as well as TLS.
The following table outlines the supported
ConfigMap options for etcd:
|etcd_endpoints||Comma-delimited list of etcd endpoints to connect to.||http://127.0.0.1:2379|
|etcd_ca||The file containing the root certificate of the CA that issued the etcd server certificate. Configures
|etcd_key||The file containing the private key of the
|etcd_cert||The file containing the client certificate issued to
To use these manifests with a TLS-enabled etcd cluster you must do the following:
Download the v3.9 manifest that corresponds to your installation method.
Calico for policy and networking
curl https://docs.projectcalico.org/v3.9/manifests/calico-etcd.yaml -O
Calico for policy and flannel for networking
curl https://docs.projectcalico.org/v3.9/manifests/canal.yaml -O
ConfigMapsection, uncomment the
etcd_certlines so that they look as follows.
Ensure that you have three files, one containing the
etcd_cavalue, another containing the
etcd_keyvalue, and a third containing the
Using a command like the following to strip the newlines from the files and base64-encode their contents.
cat <file> | base64 -w 0
etcd_certand paste in the appropriate base64-encoded values.
Apply the manifest.
Calico for policy and networking
kubectl apply -f calico.yaml
Calico for policy and flannel for networking
kubectl apply -f canal.yaml
Calico’s manifests assign its components one of two service accounts. Depending on your cluster’s authorization mode, you’ll want to back these service accounts with the necessary permissions.
Configuring service advertisement
Calico supports advertising Kubernetes services over
so that service cluster IPs are routable from outside the cluster. To
enable this, add a
CALICO_ADVERTISE_CLUSTER_IPS variable setting to
the environment for calico/node in the
manifest, with value equal to the cluster IP range for your Kubernetes
cluster; for example:
env: [...] - name: CALICO_ADVERTISE_CLUSTER_IPS value: "10.96.0.0/12"
For more information, see Configuring calico/node.
Other configuration options
The following table outlines the remaining supported
|calico_backend||The backend to use.||
|cni_network_config||The CNI Network config to install on each node. Supports templating as described below.|
CNI network configuration template
cni_network_config configuration option supports the following template fields, which will
be filled in automatically by the
||The Kubernetes service Cluster IP, e.g
||The Kubernetes service port, e.g.,
||The service account token for the namespace, if one exists.|
||The etcd endpoints specified in
||The path to the automatically generated kubeconfig file in the same directory as the CNI network configuration file.|
||The path to the etcd key file installed to the host. Empty if no key is present.|
||The path to the etcd certificate file installed to the host, empty if no cert present.|
||The path to the etcd certificate authority file installed to the host. Empty if no certificate authority is present.|
Customizing application layer policy manifests
About customizing application layer policy manifests
Instead of installing from our pre-modified Istio manifests, you may wish to customize your Istio install or use a different Istio version. This section walks you through the necessary changes to a generic Istio install manifest to allow application layer policy to operate.
The standard Istio manifests for the sidecar injector include a ConfigMap that contains the template used when adding pods to the cluster. The template adds an init container and the Envoy sidecar. Application layer policy requires an additional lightweight sidecar called Dikastes which receives Calico policy from Felix and applies it to incoming connections and requests.
If you haven’t already done so, download an Istio release and untar it to a working directory.
install/kubernetes/istio-demo-auth.yaml file in an
editor, and locate the
istio-sidecar-injector ConfigMap. In the existing
istio-proxy container, add a new
- mountPath: /var/run/dikastes name: dikastes-sock
Add a new container to the template.
- name: dikastes image: calico/dikastes:v3.9.3 args: ["server", "-l", "/var/run/dikastes/dikastes.sock", "-d", "/var/run/felix/nodeagent/socket"] securityContext: runAsNonRoot: true allowPrivilegeEscalation: false livenessProbe: exec: command: - /healthz - liveness initialDelaySeconds: 3 periodSeconds: 3 readinessProbe: exec: command: - /healthz - readiness initialDelaySeconds: 3 periodSeconds: 3 volumeMounts: - mountPath: /var/run/dikastes name: dikastes-sock - mountPath: /var/run/felix name: felix-sync
Add two new volumes.
- name: dikastes-sock emptyDir: medium: Memory - name: felix-sync flexVolume: driver: nodeagent/uds
The volumes you added are used to create Unix domain sockets that allow communication between Envoy and Dikastes and between Dikastes and Felix. Once created, a Unix domain socket is an in-memory communications channel. The volumes are not used for any kind of stateful storage on disk.
Refer to the Calico ConfigMap manifest for an example with the above changes.