Multi-Domain Configuration in Layer 4 Nginx Load Balancer

Nginx as a load balancer or reverse proxy is a common use case for Nginx around the world. Load balancer usually not only for balancing the upstream server load, but also improve the overall security of your application. Usually, when we talk about layer 4 load balancer, we divide our traffic to a particular upstream by ip address or port. But in this tutorial, we will explain advanced configuration for layer 4 Nginx load balancer/reverse proxy so you can divide your traffic based on upstream domain.

This will make you able to have many sites/domains with 1 public ip address using only layer 4 load balancer/reverse proxy Nginx. Note that this will only work if the site use https. If your application is still using plain http, this configuration will not work.

Prerequisite

You need to have Nginx installed with --with-stream_ssl_preread_module. The default Nginx install using package manager doesn’t have this module. So you have to install Nginx from source. Check out my other tutorial on How to Install Nginx from Source.

Sudo Privileges

Before start, we make sure that we will have no permission issue on the configuration.

sudo su

Use Case

To demonstrate the load balancing / reverse proxy configuration, will use this use case

                                             ____________
                                            |            |
                                     -----> |    App1    |
                                    |       |____________|
                    ___________     |        ____________
                   |           |    |       |            |
     user -------> |     LB    |----|-----> |    App2    |
                   |___________|    |       |____________|
                                    |        ____________
                                    |       |            |
                                     -----> |    App3    |
                                            |____________|

There are 3 nodes of the application that will be load balanced using Nginx LB (Load Balancer). These are the specification of the nodes:

1. LB Node
   • IP: 110.2.2.10
   • Port: 443
2. App1 Node
   • Domain: app1.facsiaginsa.com
   • IP: 10.1.1.10
   • Port: 4000
3. App2 Node
   • Domain: app2.facsiaginsa.com
   • IP: 10.1.1.20
   • Port: 4000
4. App3 Node
   • Domain: app3.facsiaginsa.com
   • IP: 10.1.1.30
   • Port: 4000

In this use case, we have 3 different applications using 3 different domain names. The application is served using https protocol.

Nginx Layer 4 Configuration

First, move our work directory to the Nginx configuration folder

cd /etc/nginx

Backup the default Nginx configuration file

mv nginx.conf nginx.conf.old

Create a new configuration file

nano nginx.conf

Add this configuration to the configuration file

user www-data;
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;

events {
        worker_connections 4096;
}

This will set up a basic global configuration for the Nginx. The important part is the worker_rlimit_nofile and worker_connections directives. You can set the number higher depending on the specification of your server. No one knows the best setup, you must do a load test yourself.

After that, create a stream block under the events block, and create a map block inside it to map the app domain with the upstream

stream {
   map $ssl_preread_server_name $upstream {
      app1.facsiaginsa.com    app1;
      app2.facsiaginsa.com    app2;
      app3.facsiaginsa.com    app3;
   }
}

And then create a upstream block under the map block to define the location of each upstream.

stream {
   map $ssl_preread_server_name $upstream {
      app1.facsiaginsa.com    app1;
      app2.facsiaginsa.com    app2;
      app3.facsiaginsa.com    app3;
   }

   upstream app1 {
      server 10.1.1.10:4000;
   }

   upstream app2 {
      server 10.1.1.20:4000;
   }

   upstream app3 {
      server 10.1.1.30:4000;
   }
}

Lastly, add a server block under the upstream block to read the domain of the incoming request, and pass it to the right upstream.

stream {
   map $ssl_preread_server_name $upstream {
      app1.facsiaginsa.com    app1;
      app2.facsiaginsa.com    app2;
      app3.facsiaginsa.com    app3;
   }

   upstream app1 {
      server 10.1.1.10:4000;
   }

   upstream app2 {
      server 10.1.1.20:4000;
   }

   upstream app3 {
      server 10.1.1.30:4000;
   }

   server {
      listen 443;
      ssl_preread on;
      proxy_pass $upstream;
   }
}

This will tell the Nginx to listen to port 443, and then pass the TCP traffic to upstream that match the domain in the map block. ssl_preread on is the one that made this possible. Without this directive, your Nginx cannot read the domain on the incoming request.

Note that the ssl_preread on is not terminating the SSL/TLS. It just extracting information from the ClientHello. So the SSL/TLS termination should be done in the App1, App2, and App3nodes.

So the full configuration of nginx.conf is like this

user www-data;
worker_processes auto;
worker_rlimit_nofile 8192;
pid /run/nginx.pid;

events {
        worker_connections 4096;
}

stream {
   map $ssl_preread_server_name $upstream {
      app1.facsiaginsa.com    app1;
      app2.facsiaginsa.com    app2;
      app3.facsiaginsa.com    app3;
   }

   upstream app1 {
      server 10.1.1.10:4000;
   }

   upstream app2 {
      server 10.1.1.20:4000;
   }

   upstream app3 {
      server 10.1.1.30:4000;
   }

   server {
      listen 443;
      ssl_preread on;
      proxy_pass $upstream;
   }
}

After the configuration is done, make sure that there is no error in the configuration by running this command

nginx -t

Reload the Nginx so it uses the new configuration file

service nginx reload

Or start the Nginx if it is not started yet

service nginx start

You can check the Nginx status by running this command

service nginx status

That’s it! Now you can set up Nginx to serve multiple domains using only layer 4 load balancer/reverse proxy.