You might think that using a layer 4 load balancer prevents us from dividing traffic based on domain, as this is typically done by a layer 7 load balancer. However, this is not the case.
Nginx allows us to read the traffic domain without terminating the SSL, so we can still route traffic based on a domain at layer 4.
How can we do this? Let’s find out!
Prerequisite
- Ubuntu 20.04, 22.04
Sudo Privileges
Before starting, we make sure that we will have no permission issues on the configuration.
sudo su
Use Case
To demonstrate the load balancing configuration, will use this use case
____________
| |
-----> | App1 |
| |____________|
___________ | ____________
| | | | |
user -----------> | LB |----|-----> | App2 |
|___________| | |____________|
| ____________
| | |
-----> | App3 |
|____________|
There are 3 nodes of application that will be load balanced using Nginx Layer 4 LB (Load Balancer). These are the specifications of the nodes:
- LB Node
- IP: 10.11.12.13
- Port: 443
- App1 Node
- IP: 10.1.1.10
- Port: 443
- URL: https://app1.facsiaginsa.com
- App2 Node
- IP: 10.1.1.20
- Port: 443
- URL: https://app2.facsiaginsa.com
- App3 Node
- IP: 10.1.1.30
- Port: 443
- URL: https://app3.facsiaginsa.com
Install Dependencies
Run this command to install Nginx dependencies.
apt update -y && apt-get install git build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev libgd-dev libxml2 libxml2-dev uuid-dev
Download Nginx Source Code
Before you download the Nginx source code, you can visit http://nginx.org/en/download.html to see the Nginx version available now. After that, you can download them by running this command:
wget http://nginx.org/download/nginx-<version>.tar.gz
Now, the latest stable version is 1.26.2, so for me, I will download the nginx-1.26.2 version
wget http://nginx.org/download/nginx-1.26.2.tar.gz
Extract the downloaded file.
tar -zxvf nginx-1.26.2.tar.gz
Build & Install Nginx
After extracting the file, go to the nginx directory.
cd nginx-1.26.2
Now is the time to configure Nginx that suits your needs, this is where you put in the module you want to include in Nginx using the ./configure command. The full documentation is in here: Building Nginx from Sources. For now, I will give you the minimum configure option so you can build a good load balancer, reverse proxy, or web server. Run this command to configure Nginx:
./configure \
--prefix=/etc/nginx \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/run/nginx.pid \
--sbin-path=/usr/sbin/nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_stub_status_module \
--with-http_realip_module \
--with-file-aio \
--with-threads \
--with-stream \
--with-stream_ssl_preread_module
After that, run this command to build & install the Nginx.
make && make install
To verify the installation, you can check the Nginx version.
nginx -V
Also, notice that the Nginx folder will be created at /etc/nginx that provides the default nginx.conf.
Nginx Layer 4 Configuration
First, move our work directory to the Nginx configuration folder.
cd /etc/nginx
Backup the default nginx configuration file
mv nginx.conf nginx.conf.old
Create a new configuration file.
nano nginx.conf
Add this configuration to the configuration file
user www-data;
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;
events {
worker_connections 4096;
}
This will set up the basic global configuration for the Nginx. The important part is the worker_rlimit_nofile and worker_connections directives. You can set the number higher depending on the specification of your server. No one knows the best setup, you must do a load test yourself.
After that, create a stream block under the events block, and create a server block inside it.
stream {
server {
listen 443;
ssl_preread on;
proxy_pass $app_node;
}
}
This will tell the nginx to listen to port 3000, and then pass the TCP traffic to upstream called app_node. If you are using UDP traffic, just change the listen directive to:
listen 3000 udp;
The ssl_preread on directive will enable Nginx to extract information from the ClientHello message, get the SNI (Server Name Indication) information, and save it to $ssl_preread_server_name variable.
Now we must create a map block inside the stream block, and then also add some upstream block to list all of the app nodes behind the load balancer.
stream {
map $ssl_preread_server_name $app_node {
app1.facsiaginsa.com app1;
app2.facsiaginsa.com app2;
app3.facsiaginsa.com app3;
}
upstream app1 {
server 10.1.1.10:443;
}
upstream app2 {
server 10.1.1.20:443;
}
upstream app3 {
server 10.1.1.30:443;
}
server {
listen 443;
ssl_preread on;
proxy_pass $app_node;
}
}
Using this configuration, the Nginx will map the SNI information and the upstream server accordingly every time a request is coming.
So the full configuration of nginx.conf is like this.
user www-data;
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;
events {
worker_connections 4096;
}
stream {
map $ssl_preread_server_name $app_node {
app1.facsiaginsa.com app1;
app2.facsiaginsa.com app2;
app3.facsiaginsa.com app3;
}
upstream app1 {
server 10.1.1.10:443;
}
upstream app2 {
server 10.1.1.20:443;
}
upstream app3 {
server 10.1.1.30:443;
}
server {
listen 443;
ssl_preread on;
proxy_pass $app_node;
}
}
Notice that there are no TLS or https configurations in the Layer 4 Load Balancer because Nginx only reads the SSL/TLS information. The TLS termination must be done in the upstream server (the App node).
After the configuration is done, make sure that there is no error in the configuration by running this command.
nginx -t
Create Systemd File
To make Nginx easier to manage, we can build a systemd file. First, create a new file in the systemd folder:
nano /lib/systemd/system/nginx.service
Then copy & paste this config to the file.
[Unit]
Description=Nginx Custom From Source
After=syslog.target network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Reload the systemd
systemctl daemon-reload
Enable Nginx service so it will auto start when the server boot.
systemctl enable nginx
Now you can control Nginx using systemd, just like this:
service nginx start
service nginx stop
service nginx reload
service nginx restart
Create Logrotate File
Logrotate is useful for rotating the Nginx log so it will not write on a single file continuously. First, create a new file on the log rotate folder
nano /etc/logrotate.d/nginx
Copy & Paste this code
/var/log/nginx/*.log {
daily
missingok
rotate 7
compress
delaycompress
notifempty
create 0640 root root
sharedscripts
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi \
endscript
postrotate
invoke-rc.d nginx rotate >/dev/null 2>&1
endscript
}
There you go, now you have a layer 4 load balancer that can divide traffic based on the host domain.