How to Join a Node to the K3s Cluster

If you have a single k3s node and then you want to scale it to become a multi-node cluster, you came to the right tutorial. In this tutorial, I will assume that you already have a single node k3s. If you don’t, then you can check my previous tutorial to create a single node k3s cluster that is easy to scale here: “How To Install k3s With Calico And External MySQL Database”.

Prerequisite

1. Ubuntu 20.04 or later
2. k3s cluster already running
3. no firewall activated between the existing k3s cluster and the new server

Sudo Privileges

Before starting, we make sure that we will have no permission issues on the installation and configuration.

sudo su

Get Existing k3s Cluster Configuration

These steps below are done in the existing k3s server (master) node.

First, Go to your existing k3s cluster master node and then use this command to make sure the cluster is running

kubectl get node

If your k3s cluster is running, you will get this response

NAME           STATUS   ROLES                  AGE   VERSION
k3s-server-1   Ready    control-plane,master   30m   v1.28.4+k3s2

Next, get your k3s configuration by opening /etc/systemd/system/k3s.service file using nano

nano /etc/systemd/system/k3s.service

You can see your cluster configuration here

EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
        '--cluster-init' \
        '--tls-san=k3s_endpoint' \
        '--datastore-endpoint=mysql://k3s_user:K3S123@tcp(mysql_endpoint:3306)/k3s_database' \
        '--flannel-backend=none' \
        '--cluster-cidr=192.168.0.0/16' \
        '--disable-network-policy' \
        '--disable=traefik' \

The marked texts are all the cluster configurations that are configured at cluster initiation. Save the marked text as we will need them later to create the join command.

Notes!

The configuration k3s_endpoint and mysql_endpoint on the configuration is a domain name that must be resolvable. Depending on how your cluster is initiated, maybe it will be an IP address instead of a domain name.

To check the IP address, you can try doing a ping test to know the IP address.

root@k3s-server-1:/var/lib/rancher/k3s# ping k3s_endpoint
PING k3s_endpoint (127.0.0.1) 56(84) bytes of data.

root@k3s-server-1:/var/lib/rancher/k3s# ping mysql_endpoint
PING k3s_endpoint (127.0.0.1) 56(84) bytes of data.

You can see in my case the k3s_endpoint and mysql_endpoint is pointing to the localhost (127.0.0.1) of the k3s master node.

After that, get your cluster token on /var/lib/rancher/k3s/server/token file

cat /var/lib/rancher/k3s/server/token

You will get a response that contains your cluster token on the marked text below

K103ffe438192e539d52726e65e55d2a4eaa92023428ae522a2537709251a218339::server:K3SC11T

From this command, we get our cluster token: K3SC11T

Join k3s Server (Master) Node to Cluster

These steps below are done in the new k3s server node that wants to join the cluster.

To join a K3s server Node, you must write the cluster configuration on the join command. But first, we must define k3s_endpoint and mysql_endpoint IP addresses in /etc/hosts.

Go to your new node and open the /etc/hosts file using nano

nano /etc/hosts

Note! If your k3s_endpoint and mysql_endpoint are IP addresses, you don’t have to write anything in /etc/hosts and skip this step.

Add this line

10.57.149.209 k3s_endpoint
10.57.149.209 mysql_endpoint

Notes!

Note that 10.57.149.209 is my k3s master IP address. I’m using this IP address because previously we know that the k3s_endpoint and mysql_endpoint are pointing to the k3s master (using ping test on the k3s master node). If you have a different result, use that IP address instead.

You can use a ping test to confirm that your configuration is correct

root@k3s-server-2:/home/bagus# ping mysql_endpoint
PING mysql_endpoint (10.57.149.209) 56(84) bytes of data.

root@k3s-server-2:/home/bagus# ping k3s_endpoint
PING k3s_endpoint (10.57.149.209) 56(84) bytes of data.

You see that the domain is pointing to the correct ip address.

Next, make sure the new server can access the datastore (if the datastore using an external datastore) like in this case. You can check the connection by typing this command

nc -vz mysql_endpoint 3306

If failed, make sure these 2 things:

1. No firewall that blocks the connection between the datastore server & the new k3s server
2. Make sure the datastore does not listen to localhost only.

After that, use this command to join the cluster. Note that all of the cluster configuration and token that we get from previous steps are copied here.

curl -sfL https://get.k3s.io | sh -s - server \
  --token=K3SC11T \
  --tls-san=k3s_endpoint \
  --datastore-endpoint="mysql://k3s_user:K3S123@tcp(mysql_endpoint:3306)/k3s_database" \
  --flannel-backend=none \
  --cluster-cidr=192.168.0.0/16 \
  --disable-network-policy \
  --disable=traefik

Done! You can monitor until all control plane’s pods are in running status.

kubectl get pod --all-namespaces --watch

After some minutes, you can check your cluster is ready

root@k3s-server-2:/home/bagus# kubectl get node
NAME           STATUS   ROLES                  AGE     VERSION
k3s-server-1   Ready    control-plane,master   10h     v1.28.4+k3s2
k3s-server-2   Ready    control-plane,master   4m19s   v1.28.4+k3s2

Join k3s Agent (Worker) Node to Cluster

These steps below are done in the new k3s agent node that wants to join the cluster.

Same as the server node, we need to make sure that the agent node can point to the k3s_endpoint correctly. So,

To join an agent node, we don’t have to specify the cluster configuration, but we still need the cluster token. So, Go to your new node and open the /etc/hosts file using nano.

Note! If your k3s_endpoint is an IP address, you don’t have to write anything in /etc/hosts and skip this step.

nano /etc/hosts

Add this line

10.57.149.209 k3s_endpoint

Notes!

Note that 10.57.149.209 is my k3s master IP address. I’m using this IP address because previously we knew that the k3s_endpoint was pointing to the k3s master (using ping test on the k3s server (master) node). If you have a different result, use that IP address instead.

Next, use this command to join the cluster.

curl -sfL https://get.k3s.io | K3S_URL=https://k3s_endpoint:6443 K3S_TOKEN=K3SC11T sh -

Done! After some minutes, you can check your cluster is ready

root@k3s-server-1:/home/bagus# k get node
NAME           STATUS   ROLES                  AGE     VERSION
k3s-server-1   Ready    control-plane,master   11h     v1.28.4+k3s2
k3s-server-2   Ready    <none>                 2m31s   v1.28.4+k3s2