If you are looking for external etcd cluster setup for kubernetes, this is the right place. Because kubernetes use etcd cluster with TLS Encryption.
To form a cluster, etcd require a minimum 3 etcd nodes. Don’t configure etcd with 2 nodes, it will not create a high availability. An etcd cluster needs a majority a quorum, to agree on updates to the cluster state. For a cluster with n
nodes, quorum is (n/2)+1
.
3 nodes etcd can handle 1 node failure, 5 nodes etcd can handle 2 node failure, and so on. A 5 nodes etcd cluster can tolerate 2 nodes failures, which is enough in most cases. Although larger clusters provide better fault tolerance, the write performance will suffers because data must be replicated across more machines.
Prerequisite
- Minimum of 3 servers
- Firewall open on port
2379
and2380
- Ubuntu 18.04
Sudo Privileges
Before start we make sure that we will have no permission issue on the installation.
sudo su
Use Case
In this tutorial we will install etcd with this setup:
____________ ____________
| | | |
| etcd 1 |------------| etcd 2 |
|____________| | |____________|
|
_____|______
| |
| etcd 3 |
|____________|
etcd 1
ip address: 192.168.5.100
etcd 2
ip address: 192.168.5.101
etcd 3
ip address: 192.168.5.102
Create CA Cert & Keys
Do this step in your localhost (linux). If you are on windows, just use wsl or any vm.
Download Dependencies (cfssl & cfssljson) to create the CA certificate & key
wget -q --show-progress \
https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssl \
https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssljson
Move to /usr/bin
and give execute permission
chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/bin
Create ca-config.json
file
nano ca-config.json
Copy this script:
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"etcd": {
"expiry": "8760h",
"usages": ["signing","key encipherment","server auth","client auth"]
}
}
}
}
Create ca-csr.json
file:
nano ca-csr.json
Copy this script:
{
"CN": "etcd cluster",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "ID",
"L": "Indonesia",
"O": "Kubernetes",
"OU": "ETCD-CA",
"ST": "West Java"
}
]
}
Execute this script to create CA certificate and key:
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
We will get ca.pem
and ca-key.pem
file. This is the CA certificate and key file. We will need this in the next step.
Create etcd Cert & Keys
Create etcd-csr.json
file:
nano etcd-csr.json
Copy this script:
{
"CN": "etcd",
"hosts": [
"localhost",
"127.0.0.1",
"192.168.5.100", // edit this to match your etcd 1 ip address
"192.168.5.101", // edit this to match your etcd 2 ip address
"192.168.5.102" // edit this to match your etcd 3 ip address
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "ID",
"L": "Indonesia",
"O": "Kubernetes",
"OU": "ETCD",
"ST": "West Java"
}
]
}
Don’t forget to edit your etcd ip to the file
Execute this script to get server certificate and key:
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd
We will get etcd.pem
and etcd-key.pem
in our directory
Now upload ca.pem
, etcd.pem
and etcd-key.pem
to all of your etcd server.
Installation
To download etcd (we will using the latest stable version 3.5.0), we can download the binary file from the github. Do this installation step on all 3 nodes.
wget -q --show-progress "https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz"
Extract the tar files
tar zxf etcd-v3.5.0-linux-amd64.tar.gz
Move the binary to /usr/bin
and add executable permission to become executable
mv etcd-v3.5.0-linux-amd64/etcd* /usr/bin/
chmod +x /usr/bin/etcd*
Right now you already install etcd. You will have 2 important executable binary:
etcd
the binary to start the etcd it selfetcdctl
the binary for etcd client.
Start & Enable etcd service
First, we will create configuration file on all etcd nodes
nano /etc/etcd
Copy this script to /etc/etcd
file on etcd 1
:
ETCD_NAME=etcd1
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=https://192.168.5.100:2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://192.168.5.100:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.5.100:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.5.100:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.5.100:2380,etcd2=https://192.168.5.101:2380,etcd3=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/ca.pem // edit this to match your ca.pem location
ETCD_CERT_FILE=/etc/ssl/etcd.pem // edit this to match your etcd.pem location
ETCD_KEY_FILE=/etc/ssl/etcd-key.pem // edit this to match your etcd-key.pem location
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/ca.pem // edit this to match your ca.pem location
ETCD_PEER_CERT_FILE=/etc/ssl/etcd.pem // edit this to match your etcd.pem location
ETCD_PEER_KEY_FILE=/etc/ssl/etcd-key.pem // edit this to match your etcd-key.pem location
Copy this script to /etc/etcd
file on etcd 2
:
ETCD_NAME=etcd2
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=https://192.168.5.101:2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://192.168.5.101:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.5.101:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.5.101:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.5.100:2380,etcd2=https://192.168.5.101:2380,etcd3=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/ca.pem // edit this to match your ca.pem location
ETCD_CERT_FILE=/etc/ssl/etcd.pem // edit this to match your etcd.pem location
ETCD_KEY_FILE=/etc/ssl/etcd-key.pem // edit this to match your etcd-key.pem location
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/ca.pem // edit this to match your ca.pem location
ETCD_PEER_CERT_FILE=/etc/ssl/etcd.pem // edit this to match your etcd.pem location
ETCD_PEER_KEY_FILE=/etc/ssl/etcd-key.pem // edit this to match your etcd-key.pem location
Copy this script to /etc/etcd
file on etcd 3
:
ETCD_NAME=etcd3
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=https://192.168.5.102:2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://192.168.5.102:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.5.102:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.5.100:2380,etcd2=https://192.168.5.101:2380,etcd3=https://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ssl/ca.pem // edit this to match your ca.pem location
ETCD_CERT_FILE=/etc/ssl/etcd.pem // edit this to match your etcd.pem location
ETCD_KEY_FILE=/etc/ssl/etcd-key.pem // edit this to match your etcd-key.pem location
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/ca.pem // edit this to match your ca.pem location
ETCD_PEER_CERT_FILE=/etc/ssl/etcd.pem // edit this to match your etcd.pem location
ETCD_PEER_KEY_FILE=/etc/ssl/etcd-key.pem // edit this to match your etcd-key.pem location
Create systemd file on all etcd nodes:
nano /etc/systemd/system/etcd.service
Copy this script to /etc/systemd/system/etcd.service
file on all etcd notes:
[Unit]
Description=etcd
[Service]
Type=notify
EnvironmentFile=/etc/etcd
ExecStart=/usr/bin/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Enable the systemd etcd service on all etcd nodes so it will start on server boot
systemctl daemon-reload
systemctl enable etcd
Start etcd service on all 3 nodes:
service etcd start
Notes: You must start at least 2 etcd nodes whitin 50 seconds or it will return error because of the master election timeout.
Check the etcd service by running this command
service etcd status
Checking the installation
From 1 of the etcd server, run this command to verify the cluster status:
etcdctl \
--endpoints=https://192.168.5.100:2379 \
--cacert=ca.pem \
--cert=etcd.pem \
--key=etcd-key.pem \
member list
Don’t forget to edit the cacert
, cert
, and key
parameter to match your file location.
This command should return the list of the etcd cluster member:
685732e85e851bdd, started, etcd1, https://192.168.5.100:2380, https://192.168.5.100:2379, false
8940390e3669b48e, started, etcd2, https://192.168.5.101:2380, https://192.168.5.101:2379, false
345721e85e456bzw, started, etcd3, https://192.168.5.102:2380, https://192.168.5.102:2379, false
Congrats! You have an etcd cluster running on your 3 servers with TLS Encryption.