Warning: You're not viewing the latest Calico documentation.

Red Hat Enterprise Linux 7 Packaged Install Instructions

These instructions will take you through a first-time install of Calico. If you are upgrading an existing system, please see the Calico on OpenStack upgrade document instead for upgrade instructions.

There are three sections to the install: installing etcd, upgrading control nodes to use Calico, and upgrading compute nodes to use Calico. Follow the Common Steps on each node before moving on to the specific instructions in the control and compute sections. If you want to create a combined control and compute node, work through all three sections.

WARNING

Following the upgrade to use etcd as a data store, Calico currently only supports RHEL 7 and above. If support on RHEL 6.x or other versions of Linux is important to you, then please let us know.

Prerequisites

Before starting this you will need the following:

One or more machines running RHEL 7, with OpenStack Juno, Kilo, Liberty or Mitaka installed.
SSH access to these machines.
Working DNS between these machines (use /etc/hosts if you don’t have DNS on your network).

Common Steps

Some steps need to be taken on all machines being installed with Calico. These steps are detailed in this section.

Install OpenStack

If you haven’t already done so, install Openstack with Neutron and ML2 networking.

Configure YUM repositories

The latest version of Calico for OpenStack is 2.0, and we recommend using it with OpenStack Liberty or later. Other possible combinations are shown by the following table.

OpenStack release	Calico version	Repository
Mitaka	2.0	rpm/calico-2.0
Liberty	2.0	rpm/calico-2.0
Mitaka	1.4	rpm/calico-1.4
Liberty	1.4	rpm/calico-1.4
(deprecated) Kilo	1.3	rpm_kilo
(deprecated) Juno	1.3	rpm_juno

If you’re using CentOS with Juno or Kilo, check the yum priorities plugin is installed:

    yum install yum-plugin-priorities

Add the EPEL repository – see https://fedoraproject.org/wiki/EPEL. You may have already added this to install OpenStack.

Configure the Calico repository as indicated above for your chosen OpenStack release. For example, for Mitaka:

    cat > /etc/yum.repos.d/calico.repo <<EOF
    [calico]
    name=Calico Repository
    baseurl=http://binaries.projectcalico.org/rpm/calico-2.0/
    enabled=1
    skip_if_unavailable=0
    gpgcheck=1
    gpgkey=http://binaries.projectcalico.org/rpm/calico-2.0/key
    priority=97
    EOF

NOTE: With OpenStack Juno or Kilo, the priority setting in calico.repo is needed so that the Calico repository can install Calico-enhanced versions of some of the OpenStack Nova and Neutron packages.

Etcd Install

Calico requires an etcd database to operate - this may be installed on a single machine or as a cluster.

These instructions cover installing a single node etcd database. You may wish to co-locate this with your control node. If you want to install a cluster, please get in touch with us and we’ll be happy to help you through the process.

Install and configure etcd.

Download, unpack, and install the binary:

    curl -L  https://github.com/coreos/etcd/releases/download/v2.0.11/etcd-v2.0.11-linux-amd64.tar.gz -o etcd-v2.0.11-linux-amd64.tar.gz
    tar xvf etcd-v2.0.11-linux-amd64.tar.gz
    cd etcd-v2.0.11-linux-amd64
    mv etcd* /usr/local/bin/

WARNING

We’ve seen certificate errors downloading etcd - you may need to add --insecure to the curl command to ignore this.

Create an etcd user:

    adduser -s /sbin/nologin -d /var/lib/etcd/ etcd
    chmod 700 /var/lib/etcd/

Add the following line to the bottom of /etc/fstab. This will mount a ramdisk for etcd at startup:
```
    tmpfs /var/lib/etcd tmpfs nodev,nosuid,noexec,nodiratime,size=512M 0 0
```
Run mount -a to mount it now.

Get etcd running by providing an init file.

Place the following in /etc/sysconfig/etcd, replacing <hostname> and <public_ip> with their appropriate values for the machine.

    ETCD_DATA_DIR=/var/lib/etcd
    ETCD_NAME=<hostname>
    ETCD_ADVERTISE_CLIENT_URLS="http://<public_ip>:2379,http://<public_ip>:4001"
    ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001"
    ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
    ETCD_INITIAL_ADVERTISE_PEER_URLS="http://<public_ip>:2380"
    ETCD_INITIAL_CLUSTER="<hostname>=http://<public_ip>:2380"
    ETCD_INITIAL_CLUSTER_STATE=new

Check the uuidgen tool is installed (the output should change each time):

    # uuidgen
    11f92f19-cb5a-476f-879f-5efc34033b8b

If it is not installed, run yum install util-linux to install it.

Place the following in /usr/local/bin/start-etcd:

    #!/bin/sh
    export ETCD_INITIAL_CLUSTER_TOKEN=`uuidgen`
    exec /usr/local/bin/etcd

Then run chmod +x /usr/local/bin/start-etcd to make that file executable.

You then need to add the following file to /usr/lib/systemd/system/etcd.service:

    [Unit]
    Description=Etcd
    After=syslog.target network.target

    [Service]
    User=root
    ExecStart=/usr/local/bin/start-etcd
    EnvironmentFile=-/etc/sysconfig/etcd
    KillMode=process
    Restart=always

    [Install]
    WantedBy=multi-user.target

Launch etcd and set it to restart after a reboot:

    systemctl start etcd
    systemctl enable etcd

Etcd Proxy Install

Install an etcd proxy on every node running OpenStack services that isn’t running the etcd database itself (both control and compute nodes).

Install and configure etcd as an etcd proxy.

Download, unpack, and install the binary:

    curl -L  https://github.com/coreos/etcd/releases/download/v2.0.11/etcd-v2.0.11-linux-amd64.tar.gz -o etcd-v2.0.11-linux-amd64.tar.gz
    tar xvf etcd-v2.0.11-linux-amd64.tar.gz
    cd etcd-v2.0.11-linux-amd64
    mv etcd* /usr/local/bin/

WARNING

We’ve seen certificate errors downloading etcd - you may need to add --insecure to the curl command to ignore this.

Create an etcd user:

       adduser -s /sbin/nologin -d /var/lib/etcd/ etcd
       chmod 700 /var/lib/etcd/

Get etcd running by providing an init file.

Place the following in /etc/sysconfig/etcd, replacing <etcd_hostname> and <etcd_ip> with the values you used in the etcd install section.

        ETCD_PROXY=on
        ETCD_DATA_DIR=/var/lib/etcd
        ETCD_INITIAL_CLUSTER="<etcd_hostname>=http://<etcd_ip>:2380"

You then need to add the following file to `/usr/lib/systemd/system/etcd.service`

        [Unit]
        Description=Etcd
        After=syslog.target network.target

        [Service]
        User=root
        ExecStart=/usr/local/bin/etcd
        EnvironmentFile=-/etc/sysconfig/etcd
        KillMode=process
        Restart=always

        [Install]
        WantedBy=multi-user.target

Launch etcd and set it to restart after a reboot:

    systemctl start etcd
    systemctl enable etcd

Control Node Install

On each control node, perform the following steps:

Delete all configured OpenStack state, in particular any instances, routers, subnets and networks (in that order) created by the install process referenced above. You can do this using the web dashboard or at the command line.

HINT

The Admin and Project sections of the web dashboard both have subsections for networks and routers. Some networks may need to be deleted from the Admin section.

WARNING

The Calico install will fail if incompatible state is left around.
Run yum update. For OpenStack Kilo and earlier, this will bring in Calico-specific updates to the OpenStack packages. (OpenStack updates are not needed for Liberty and later.)
Edit the /etc/neutron/neutron.conf file. In the [DEFAULT] section:
- Find the line beginning with core_plugin, and change it to read core_plugin = calico.
With OpenStack releases earlier than Liberty, edit the /etc/neutron/neutron.conf file. In the [DEFAULT] section:
- Find the line for the dhcp_agents_per_network setting, uncomment it, and set its value to the number of compute nodes that you will have (or any number larger than that). This allows a DHCP agent to run on every compute node, which Calico requires because the networks on different compute nodes are not bridged together.
Install the calico-control package:
```
    yum install calico-control
```
Restart the neutron server process:
```
    service neutron-server restart
```

Compute Node Install

On each compute node, perform the following steps:

Make changes to SELinux and QEMU config to allow VM interfaces with type='ethernet' (this libvirt Wiki page explains why these changes are required):
```
    setenforce permissive
```
Edit /etc/selinux/config and change the SELINUX= line to the following:
```
    SELINUX=permissive
```
In /etc/libvirt/qemu.conf, add or edit the following four options:
```
    clear_emulator_capabilities = 0
    user = "root"
    group = "root"
    cgroup_device_acl = [
         "/dev/null", "/dev/full", "/dev/zero",
         "/dev/random", "/dev/urandom",
         "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
         "/dev/rtc", "/dev/hpet", "/dev/net/tun",
    ]
```
NOTE

The cgroup_device_acl entry is subtly different to the

default. It now contains /dev/net/tun.

Then restart libvirt to pick up the changes:
```
    service libvirtd restart
```
Open /etc/nova/nova.conf and remove the line from the [DEFAULT] section that reads:
```
    linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
```
Remove the lines from the [neutron] section setting service_neutron_metadata_proxy or service_metadata_proxy to True, if there are any. Additionally, if there is a line setting metadata_proxy_shared_secret, comment that line out as well.

Restart nova compute.
```
    service openstack-nova-compute restart
```
If this node is also a controller, additionally restart nova-api:
```
    service openstack-nova-api restart
```
If they’re running, stop the Open vSwitch services:
```
    service neutron-openvswitch-agent stop
    service openvswitch stop
```
Then, prevent the services running if you reboot:
```
    chkconfig openvswitch off
    chkconfig neutron-openvswitch-agent off
```
Then, on your control node, run the following command to find the agents that you just stopped:
```
    neutron agent-list
```
For each agent, delete them with the following command on your control node, replacing <agent-id> with the ID of the agent:
```
    neutron agent-delete <agent-id>
```
Run yum update. For OpenStack Kilo and earlier, this will bring in Calico-specific updates to the OpenStack packages. (OpenStack updates are not needed for Liberty and later.)
Install Neutron infrastructure code on the compute host:
```
    yum install openstack-neutron
```

For OpenStack Juno or Kilo, open /etc/neutron/dhcp_agent.ini, and in the [DEFAULT] section add the following line (removing any existing interface_driver = line):

    interface_driver = neutron.agent.linux.interface.RoutedInterfaceDriver

then restart and enable the Neutron DHCP agent:

    service neutron-dhcp-agent restart
    chkconfig neutron-dhcp-agent on

For OpenStack Liberty or later, modify /etc/neutron/neutron.conf. In the [oslo_concurrency] section, ensure that the lock_path variable is uncommented and set as follows:

    # Directory to use for lock files. For security, the specified directory should
    # only be writable by the user running the processes that need locking.
    # Defaults to environment variable OSLO_LOCK_PATH. If external locks are used,
    # a lock path must be set.
    lock_path = $state_path/lock

Then, stop and disable the Neutron DHCP agent, and install the Calico DHCP agent (which uses etcd, allowing it to scale to higher numbers of hosts):

    service neutron-dhcp-agent stop
    chkconfig neutron-dhcp-agent off
    yum install calico-dhcp-agent

Stop and disable any other routing/bridging agents such as the L3 routing agent or the Linux bridging agent. These conflict with Calico.
```
    service neutron-l3-agent stop
    chkconfig neutron-l3-agent off
    ... repeat for bridging agent and any others ...
```
If this node is not a controller, install and start the Nova Metadata API. This step is not required on combined compute and controller nodes.
```
    yum install openstack-nova-api
    service openstack-nova-metadata-api restart
    chkconfig openstack-nova-metadata-api on
```
Install the BIRD BGP client from EPEL:
```
    yum install -y bird bird6
```
Install the calico-compute package:
```
    yum install calico-compute
```
Configure BIRD. By default Calico assumes that you’ll be deploying a route reflector to avoid the need for a full BGP mesh. To this end, it includes useful configuration scripts that will prepare a BIRD config file with a single peering to the route reflector. If that’s correct for your network, you can run either or both of the following commands.

For IPv4 connectivity between compute hosts:
```
    calico-gen-bird-conf.sh <compute_node_ip> <route_reflector_ip> <bgp_as_number>
```
And/or for IPv6 connectivity between compute hosts:
```
    calico-gen-bird6-conf.sh <compute_node_ipv4> <compute_node_ipv6> <route_reflector_ipv6> <bgp_as_number>
```
Note that you’ll also need to configure your route reflector to allow connections from the compute node as a route reflector client. If you are using BIRD as a route reflector, follow the instructions in this document. If you are using another route reflector, refer to the appropriate instructions to configure a client connection.

If you are configuring a full BGP mesh you’ll need to handle the BGP configuration appropriately on each compute host. The scripts above can be used to generate a sample configuration for BIRD, by replacing the <route_reflector_ip> with the IP of one other compute host – this will generate the configuration for a single peer connection, which you can duplicate and update for each compute host in your mesh.

To maintain connectivity between VMs if BIRD crashes or is upgraded, configure BIRD graceful restart. Edit the systemd unit file /usr/lib/systemd/system/bird.service (and bird6.service for IPv6):
- Add -R to the end of the ExecStart line.
- Add KillSignal=SIGKILL as a new line in the [Service] section.
Ensure BIRD (and/or BIRD 6 for IPv6) is running and starts on reboot:
```
    service bird restart
    service bird6 restart
    chkconfig bird on
    chkconfig bird6 on
```
Create the /etc/calico/felix.cfg file by copying /etc/calico/felix.cfg.example. Ordinarily the default values should be used, but see Configuration for more details.
Restart the Felix service:
```
    systemctl restart calico-felix
```

Provide Feedback Get Help via Slack Training