Learn, Execute, Innovate

How to Setup etcd Cluster with TLS Encryption

Written by: Bagus Facsi Aginsa
Published at: 19 Aug 2021

If you are looking for external etcd cluster setup for kubernetes, this is the right place. Because kubernetes use etcd cluster with TLS Encryption.

To form a cluster, etcd require a minimum 3 etcd nodes. Don’t configure etcd with 2 nodes, it will not create a high availability. An etcd cluster needs a majority a quorum, to agree on updates to the cluster state. For a cluster with n nodes, quorum is (n/2)+1.

3 nodes etcd can handle 1 node failure, 5 nodes etcd can handle 2 node failure, and so on. A 5 nodes etcd cluster can tolerate 2 nodes failures, which is enough in most cases. Although larger clusters provide better fault tolerance, the write performance will suffers because data must be replicated across more machines.

Prerequisite

  1. Minimum of 3 servers
  2. Firewall open on port 2379 and 2380
  3. Ubuntu 18.04

Sudo Privileges

Before start we make sure that we will have no permission issue on the installation.

sudo su

Use Case

In this tutorial we will install etcd with this setup:

       ____________              ____________
      |            |            |            |
      |   etcd 1   |------------|   etcd 2   |
      |____________|     |      |____________|
                         |
                    _____|______
                   |            |
                   |   etcd 3   |
                   |____________|

etcd 1 ip address: 192.168.5.100

etcd 2 ip address: 192.168.5.101

etcd 3 ip address: 192.168.5.102

Create CA Cert & Keys

Do this step in your localhost (linux). If you are on windows, just use wsl or any vm.

Download Dependencies (cfssl & cfssljson) to create the CA certificate & key

wget -q --show-progress \
    https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssl \
    https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssljson

Move to /usr/bin and give execute permission

chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/bin

Create ca-config.json file

nano ca-config.json

Copy this script:

{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "etcd": {
                "expiry": "8760h",
                "usages": ["signing","key encipherment","server auth","client auth"]
            }
        }
    }
}

Create ca-csr.json file:

nano ca-csr.json

Copy this script:

{
  "CN": "etcd cluster",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "ID",
      "L": "Indonesia",
      "O": "Kubernetes",
      "OU": "ETCD-CA",
      "ST": "West Java"
    }
  ]
}

Execute this script to create CA certificate and key:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

We will get ca.pem and ca-key.pem file. This is the CA certificate and key file. We will need this in the next step.

Create etcd Cert & Keys

Create etcd-csr.json file:

nano etcd-csr.json

Copy this script:

{
  "CN": "etcd",
  "hosts": [
    "localhost",
    "127.0.0.1",
    "192.168.5.100", // edit this to match your etcd 1 ip address
    "192.168.5.101", // edit this to match your etcd 2 ip address
    "192.168.5.102" // edit this to match your etcd 3 ip address
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "ID",
      "L": "Indonesia",
      "O": "Kubernetes",
      "OU": "ETCD",
      "ST": "West Java"
    }
  ]
}

Don’t forget to edit your etcd ip to the file

Execute this script to get server certificate and key:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd

We will get etcd.pem and etcd-key.pem in our directory

Now upload ca.pem, etcd.pem and etcd-key.pem to all of your etcd server.

Installation

To download etcd (we will using the latest stable version 3.5.0), we can download the binary file from the github. Do this installation step on all 3 nodes.

wget -q --show-progress "https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz"

Extract the tar files

tar zxf etcd-v3.5.0-linux-amd64.tar.gz

Move the binary to /usr/bin and add executable permission to become executable

mv etcd-v3.5.0-linux-amd64/etcd* /usr/bin/
chmod +x /usr/bin/etcd*

Right now you already install etcd. You will have 2 important executable binary:

  1. etcd the binary to start the etcd it self
  2. etcdctl the binary for etcd client.

Start & Enable etcd service

First, we will create configuration file on all etcd nodes

nano /etc/etcd

Copy this script to /etc/etcd file on etcd 1:

ETCD_NAME=etcd1
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=https://192.168.5.100:2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://192.168.5.100:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.5.100:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.5.100:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.5.100:2380,etcd2=https://192.168.5.101:2380,etcd3=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/ca.pem         // edit this to match your ca.pem location
ETCD_CERT_FILE=/etc/ssl/etcd.pem             // edit this to match your etcd.pem location
ETCD_KEY_FILE=/etc/ssl/etcd-key.pem          // edit this to match your etcd-key.pem location
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/ca.pem    // edit this to match your ca.pem location
ETCD_PEER_CERT_FILE=/etc/ssl/etcd.pem        // edit this to match your etcd.pem location
ETCD_PEER_KEY_FILE=/etc/ssl/etcd-key.pem     // edit this to match your etcd-key.pem location

Copy this script to /etc/etcd file on etcd 2:

ETCD_NAME=etcd2
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=https://192.168.5.101:2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://192.168.5.101:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.5.101:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.5.101:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.5.100:2380,etcd2=https://192.168.5.101:2380,etcd3=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/ca.pem         // edit this to match your ca.pem location
ETCD_CERT_FILE=/etc/ssl/etcd.pem             // edit this to match your etcd.pem location
ETCD_KEY_FILE=/etc/ssl/etcd-key.pem          // edit this to match your etcd-key.pem location
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/ca.pem    // edit this to match your ca.pem location
ETCD_PEER_CERT_FILE=/etc/ssl/etcd.pem        // edit this to match your etcd.pem location
ETCD_PEER_KEY_FILE=/etc/ssl/etcd-key.pem     // edit this to match your etcd-key.pem location

Copy this script to /etc/etcd file on etcd 3:

ETCD_NAME=etcd3
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=https://192.168.5.102:2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://192.168.5.102:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.5.102:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.5.100:2380,etcd2=https://192.168.5.101:2380,etcd3=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/ca.pem         // edit this to match your ca.pem location
ETCD_CERT_FILE=/etc/ssl/etcd.pem             // edit this to match your etcd.pem location
ETCD_KEY_FILE=/etc/ssl/etcd-key.pem          // edit this to match your etcd-key.pem location
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/ca.pem    // edit this to match your ca.pem location
ETCD_PEER_CERT_FILE=/etc/ssl/etcd.pem        // edit this to match your etcd.pem location
ETCD_PEER_KEY_FILE=/etc/ssl/etcd-key.pem     // edit this to match your etcd-key.pem location

Create systemd file on all etcd nodes:

nano /etc/systemd/system/etcd.service

Copy this script to /etc/systemd/system/etcd.service file on all etcd notes:

[Unit]
Description=etcd

[Service]
Type=notify
EnvironmentFile=/etc/etcd
ExecStart=/usr/bin/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Enable the systemd etcd service on all etcd nodes so it will start on server boot

systemctl daemon-reload
systemctl enable etcd

Start etcd service on all 3 nodes:

service etcd start

Notes: You must start at least 2 etcd nodes whitin 50 seconds or it will return error because of the master election timeout.

Check the etcd service by running this command

service etcd status

Checking the installation

From 1 of the etcd server, run this command to verify the cluster status:

etcdctl \
  --endpoints=https://192.168.5.100:2379 \
  --cacert=ca.pem \
  --cert=etcd.pem \
  --key=etcd-key.pem \
  member list

Don’t forget to edit the cacert, cert, and key parameter to match your file location. This command should return the list of the etcd cluster member:

685732e85e851bdd, started, etcd1, https://192.168.5.100:2380, https://192.168.5.100:2379, false
8940390e3669b48e, started, etcd2, https://192.168.5.101:2380, https://192.168.5.101:2379, false
345721e85e456bzw, started, etcd3, https://192.168.5.102:2380, https://192.168.5.102:2379, false

Congrats! You have an etcd cluster running on your 3 servers with TLS Encryption.

Related Tutorials:

  • How to Setup etcd Cluster

    If you are looking for external etcd cluster setup for kubernetes, you can find my other tutorial: How to Setup etcd Cluster with TLS Encryption. To ...



Latest Tutorials:



Learn, Execute, Innovate

This website was developed with love by Bagus Facsi Aginsa, Illustration by Pixeltrue from Ouch!, and hosted on Cloudflare Pages. Please read through our Privacy Policy to ensure you that we value your privacy.