If you are looking for external etcd cluster setup for kubernetes, you can find my other tutorial: How to Setup etcd Cluster with TLS Encryption.
To form a cluster, etcd require a minimum 3 etcd nodes. Don’t configure etcd with 2 nodes, it will not create a high availability. An etcd cluster needs a majority a quorum, to agree on updates to the cluster state. For a cluster with n nodes, quorum is (n/2)+1.
3 nodes etcd can handle 1 node failure, 5 nodes etcd can handle 2 node failure, and so on. A 5 nodes etcd cluster can tolerate 2 nodes failures, which is enough in most cases. Although larger clusters provide better fault tolerance, the write performance will suffers because data must be replicated across more machines.
Prerequisite
- Minimum of 3 servers
- Firewall open on port
2379and2380 - Ubuntu 18.04
Sudo Privileges
Before start we make sure that we will have no permission issue on the installation.
sudo su
Use Case
In this tutorial we will install etcd with this setup:
____________ ____________
| | | |
| etcd 1 |------------| etcd 2 |
|____________| | |____________|
|
_____|______
| |
| etcd 3 |
|____________|
etcd 1 ip address: 192.168.5.100
etcd 2 ip address: 192.168.5.101
etcd 3 ip address: 192.168.5.102
Installation
To download etcd (we will using the latest stable version 3.5.0), we can download the binary file from the github. Do this installation step on all 3 nodes.
wget -q --show-progress "https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz"
Extract the tar files
tar zxf etcd-v3.5.0-linux-amd64.tar.gz
Move the binary to /usr/bin and add executable permission to become executable
mv etcd-v3.5.0-linux-amd64/etcd* /usr/bin/
chmod +x /usr/bin/etcd*
Right now you already install etcd. You will have 2 important executable binary:
etcdthe binary to start the etcd it selfetcdctlthe binary for etcd client.
Start & Enable etcd service
First, we will create configuration file on all etcd nodes
nano /etc/etcd
Copy this script to /etc/etcd file on etcd 1:
ETCD_NAME=etcd1
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=http://192.168.5.100:2379,http://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=http://192.168.5.100:2380
ETCD_ADVERTISE_CLIENT_URLS=http://192.168.5.100:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.5.100:2380
ETCD_INITIAL_CLUSTER=etcd1=http://192.168.5.100:2380,etcd2=http://192.168.5.101:2380,etcd3=http://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
Copy this script to /etc/etcd file on etcd 2:
ETCD_NAME=etcd2
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=http://192.168.5.101:2379,http://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=http://192.168.5.101:2380
ETCD_ADVERTISE_CLIENT_URLS=http://192.168.5.101:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.5.101:2380
ETCD_INITIAL_CLUSTER=etcd1=http://192.168.5.100:2380,etcd2=http://192.168.5.101:2380,etcd3=http://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
Copy this script to /etc/etcd file on etcd 3:
ETCD_NAME=etcd3
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=http://192.168.5.102:2379,http://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=http://192.168.5.102:2380
ETCD_ADVERTISE_CLIENT_URLS=http://192.168.5.102:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.5.102:2380
ETCD_INITIAL_CLUSTER=etcd1=http://192.168.5.100:2380,etcd2=http://192.168.5.101:2380,etcd3=http://192.168.5.102:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
Create systemd file on all etcd nodes:
nano /etc/systemd/system/etcd.service
Copy this script to /etc/systemd/system/etcd.service file on all etcd notes:
[Unit]
Description=etcd
[Service]
Type=notify
EnvironmentFile=/etc/etcd
ExecStart=/usr/bin/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Enable the systemd etcd service on all etcd nodes so it will start on server boot
systemctl daemon-reload
systemctl enable etcd
Start etcd service on all 3 nodes:
service etcd start
Notes: You must start at least 2 etcd nodes whitin 50 seconds or it will return error because of the master election timeout.
Check the etcd service by running this command
service etcd status
Checking the installation
From 1 of the etcd server, run this command to verify the cluster status:
etcdctl --endpoints=http://192.168.5.100:2379 member list
This command should return the list of the etcd cluster member:
685732e85e851bdd, started, etcd1, http://192.168.5.100:2380, http://192.168.5.100:2379, false
8940390e3669b48e, started, etcd2, http://192.168.5.101:2380, http://192.168.5.101:2379, false
345721e85e456bzw, started, etcd3, http://192.168.5.102:2380, http://192.168.5.102:2379, false
Congrats! You have an etcd cluster running on your 3 servers with High Availability.