How to Configure Nginx as Layer 4 Load Balancer

Nginx as Load Balancer is common use case for Nginx around the world. Load balancer usually not only for balancing the upstream server load, but also improve overall security of your application. There are 2 types of Load Balancer that can be built using Nginx. First is Layer 4 Load Balancer, the second one is Layer 7 Load Balancer. In this tutorial, we will do a Layer 4 Load Balancer configuration. If you need Layer 7 Load Balancer configuration, you can check my other tutorial: How to Configure Nginx as Layer 7 Load Balancer.

Prerequisite

1. Nginx Installed, if not installed yet, you can just use apt install nginx to install it.

Sudo Privileges

Before start we make sure that we will have no permission issue on the configuration.

sudo su

Use Case

To demonstrate the load balancing configuration, will use this usecase

                                                 ____________
                                                |            |
                                         -----> |    App1    |
                                        |       |____________|
                        ___________     |        ____________
                       |           |    |       |            |
     user -----------> |     LB    |----|-----> |    App2    |
                       |___________|    |       |____________|
                                        |        ____________
                                        |       |            |
                                         -----> |    App3    |
                                                |____________|

There are 3 nodes of application that will be load balanced using Nginx Layer 4 LB (Load Balancer). These are the specification of the nodes:

1. LB Node
   • IP: 10.11.12.13
   • Port: 3000
2. App1 Node
   • IP: 10.1.1.10
   • Port: 4000
3. App2 Node
   • IP: 10.1.1.20
   • Port: 4000
4. App3 Node
   • IP: 10.1.1.30
   • Port: 4000

In this use case, we will assume that the application is delivering TCP traffic on the Layer 4, and using TLS or https on the Layer 7.

Check Prerequisite

Before we start, we must ensure that nginx is installed with stream module. I believe that the default version of nginx in ubuntu / centOS already have this, but to make sure, you can

nginx -V

this command will output the nginx version and configured modules like this

configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-H4cN7P/nginx-1.14.0=. 
-fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' 
--with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx 
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log 
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid 
--modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body 
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy 
--http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug 
--with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module 
--with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module 
--with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module 
--with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module 
--with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic 
--with-mail_ssl_module

Make sure that --with-stream or --with-stream=dynamic is present. If the module is not present, then you have to install the nginx manually from source. Check out my other tutorial on How to Install Nginx from Source.

Nginx Layer 4 Configuration

First, move our work directory to Nginx configuration folder

cd /etc/nginx

Backup tne default nginx configuration file

mv nginx.conf nginx.conf.old

Create new configuration file

nano nginx.conf

Add this configuration to the configuration file

user www-data;
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;

# load_module /usr/lib/nginx/modules/ngx_stream_module.so;

events {
        worker_connections 4096;
}

If you have stream module installed as dynamic module (--with-stream=dynamic), you must uncomment the load_module directive. If you have stream module installed as static module (--with-stream), you can leave the directive commented or just delete it. For me, because the module present is --with-stream=dynamic, i have to uncomment it.

This will setup basic global configuration for the Nginx. The important part is the worker_rlimit_nofile and worker_connections directives. You can set the number higher depend on the specification of your server. No one know the best setup, you must do a load test yourself.

After that, create a stream block under the events block, and create a server block inside it.

stream {
    server {
        listen 3000;
        proxy_pass app_node;
    }
}

This will tell the nginx to listen to port 3000, and then pass the TCP traffic to upstream called app_node. If you are using UDP traffic, just change the listen directive to:

listen 3000 udp; 

Now we must create the upstream block named app_node inside the stream block, and then list all the app nodes behind the load balancer.

stream {
    upstream app_node {
        server 10.1.1.10:4000;
        server 10.1.1.20:4000;
        server 10.1.1.30:4000;
    }

    server {
        listen 3000;
        proxy_pass app_node;
    }
}

By default, NGINX uses the Round Robin algorithm to load balance incoming traffic, directing it sequentially to the servers in the configured upstream. Also Nginx will do a pasif health check to each server, so if a connection attempt times out or fails at least once in a 10‑second period, NGINX marks the server as unavailable for 10 seconds. To modify the algorithm and the health check parameter, please refer to Nginx documentation: TCP and UDP Load Balancing and TCP Pasif Health Checks.

So the full configuration of nginx.conf is like this

user www-data;
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

events {
        worker_connections 4096;
}

stream {
    upstream app_node {
        server 10.1.1.10:4000;
        server 10.1.1.20:4000;
        server 10.1.1.30:4000;
    }

    server {
        listen 3000;
        proxy_pass app_node;
    }
}

Notice that there are no TLS or https configuration in the Layer 4 Load Balancer, because Nginx only read Layer 4 data (TLS/SSL is using Layer 7). The TLS termination must be done in the upstream server (the App node).

After the configuration is done, make sure that there is no error in the configuration by running this command

nginx -t

Reload the nginx so it use the new configuration file

service nginx reload

You can check the nginx status by running this command

service nginx status

Thats it! Congratulation, now you can build your own Layer 4 Load Balancer using Nginx. I also make a tutorial on how to create a multi site/domain load balancer using just layer 4 Nginx. Check my tutorial here: Multi Domain Configuration in Layer 4 Nginx Load Balancer.