Networking Mesos Tasks with Calico-CNI

Prerequisites

This guide assumes you have a running Mesos Cluster whereby each Agent has CNI enabled, and Calico services and binaries are installed. See Adding Calico-CNI to an existing Mesos Cluster, or use the Docker-Compose Demo Cluster to meet these requirements.

Getting started

1. Configure a Calico CNI Network

Before we can start launching tasks, we must first define our CNI network definition in the configured --network_cni_config_dir:

cat <<EOF > $NETWORK_CNI_CONFIG_DIR/calico-net-1.conf
{
    "name": "calico-net-1",
    "type": "calico",
    "ipam": {
        "type": "calico-ipam"
    },
    "etcd_authority": "<etcd-ip:port>:2379"
}
EOF

Note: If you have used the Docker-Compose demo to launch a cluster, this network configuration has already been created for you.

Mesos-Agent loads these network configurations into memory at launch, so you will need to restart the Agent process for your new configuration to take effect.

2. Launch a Calico-Networked Task

With your CNI network configured, you are now ready to launch tasks on it. There are several common methods you can use to do so.

a.) Using Mesos-Execute

To quickly test that Calico’s network is functioning correctly, use mesos-execute and set --networks to launch a task on your Calico-CNI network:

mesos-execute --containerizer=mesos \
              --name=calico-cni-base-test \
              --master=172.17.0.4:5050 \
              --networks=calico-net-1 \
              --command="ifconfig"

The task will have been successfully networked by Calico if its return code is 0. Additionally, the task’s stdout log should show an IP address from the default Calico pool: 192.168.0.0/16

b.) Using Marathon

In Marathon v1.2.0+, set the task’s ipAddress.networkName to the name of the CNI network:

curl -X POST -H "Content-Type: application/json" http://localhost:8080/v2/apps -d '
{
    "id": "calico-cni-marathon-test",
    "cmd": "ifconfig && sleep 1000",
    "cpus": 0.1,
    "mem": 64.0,
    "ipAddress": {
        "networkName": "calico-net-1",
        "labels": {
          "app": "test",
          "group": "development"
        }
    }
}'

Again, this task should show a Calico IP in the Marathon UI.

c.) Using Marathon - Unified Containerizer

With the release of Mesos-1.0, the Unified Containerizer can now launch Docker Images using the Mesos Containerizer. Calico can perform networking for these tasks.

To make use of this feature, ensure you are using Mesos 1.0.0+ with agents configured to use the Unified Containerizer. Also ensure you are using Marathon v1.3.0+.

Our app.json looks similar to the one above, but with a few extra settings (and some more practical use):

curl -X POST -H "Content-Type: application/json" http://localhost:8080/v2/apps -d '
{
    "id": "frontend",
    "container": {
      "type": "MESOS",
      "docker": {
        "image": "nginx"
      }
    },
    "cpus": 0.1,
    "mem": 64.0,
    "ipAddress": {
        "networkName": "calico-net-1",
        "labels": {
          "app": "frontend",
          "group": "production"
        }
    }
}'

The application will show its Calico IP when viewed using the Marathon UI, however it will be unreachable until Calico policy is configured to allow traffic.

3. Configuring Policy

Calico-CNI v1.4.1+ supports selector-based policy for Mesos v1.0.0+.

The above Marathon Application Definition has assigned the labels app=frontend and group=production to the task’s NetworkInfo. Calico-CNI automatically reads these labels and assign them to the endpoint, allowing us to enforce policy based on them.

The following YAML policy spec describes rules based on these labels:

apiVersion: v1
kind: policy
metadata:
  name: frontend-policy
spec:
  order: 50
  selector: app == 'frontend'
  ingress:
  - action: allow
    protocol: tcp
    destination:
      ports:
      - 80
  egress:
  - action: allow
    protocol: tcp
    destination:
      selector: app == 'database'
      ports:
      - 6379

This alpha version of calicoctl can be used to register this policy spec:

calicoctl create -f frontend-policy.yaml

Note: The above link to the new resource-based calicoctl is neither the permanent nor future home for the binary. Expect the golang source for calicoctl to move. The linked release may also be deleted.