Once you have more than a couple of services running across a few machines, a familiar problem shows up: how does one service find another? Hardcoding IP addresses into config files works for a while, until a server gets replaced, a service moves to a different port, or you add a second instance for redundancy. Suddenly you are editing config files across every machine that talks to that service, hoping you did not miss one.
This is the problem service discovery solves, and HashiCorp Consul is one of the most widely used tools for it. Instead of hardcoding where a service lives, your applications ask Consul, “where is the api service right now?” and get back a live, health-checked answer. Consul also happens to be a fitting next step if you have already worked through Vault, since both tools come from the same HashiCorp family, share a very similar operational shape (a small Go binary, an HCL config directory, a systemd unit, and a web UI), and often run side by side in real infrastructure. If you have not yet, the Getting Started with HashiCorp Vault on Ubuntu guide covers the secrets management side of that pairing.
This tutorial is for developers, sysadmins, and DevOps engineers who run more than one service across more than one Ubuntu machine and want a real way to track what is alive and where, instead of a spreadsheet of IP addresses. By the end, you will have a three-node Consul cluster running, a real application registered as a service, health checks watching it, and two working ways to look it up: DNS and the HTTP API.
Conceptual Overview
Consul is, at its core, a distributed key/value store combined with a service catalog. Every Consul process is called an agent, and it runs in one of two modes.
A server agent stores the actual state of the cluster (the list of registered services, their health, and any key/value data) and participates in a consensus protocol called Raft to keep that state consistent and available even if a server fails. You typically run 3 or 5 server agents for production, always an odd number, so the cluster can tolerate losing a minority of servers while still having enough left to agree on a decision.
A client agent runs on every machine that hosts a service you want to track. It does not store the cluster state itself; instead, it forwards requests to the servers and, more importantly, runs the actual health checks for the services on its own machine, since it is the one in the best position to know if a local process is still alive.
A service in Consul is anything you register with a name, an address, a port, and optionally one or more health checks. Once registered, that service becomes discoverable by two mechanisms: a DNS interface, where Consul answers api.service.consul style queries the same way any DNS server would, and an HTTP API, where you query http://127.0.0.1:8500/v1/health/service/api and get back JSON with every healthy instance.
All of this runs over a gossip protocol (based on a library called Serial, implementing the SWIM protocol) on port 8301, which is how agents discover each other and detect failures quickly without every node needing to talk to a central coordinator constantly. Server-to-server consensus traffic uses port 8300, the HTTP API and UI listen on port 8500, and DNS queries go to port 8600.
Prerequisites
Before starting, make sure you have:
- Three Ubuntu 22.04 or 24.04 servers that can reach each other over the network. This guide uses:
consul-01at10.20.0.31(server)consul-02at10.20.0.32(server)consul-03at10.20.0.33(server)app-01at10.20.0.41(client, running our example application)
sudoaccess on every node- Basic familiarity with the Linux command line and editing text files with
nanoorvim curlandunzipinstalled (sudo apt install -y curl unzip)- No prior Consul experience needed; this guide builds everything from scratch
For simplicity, this tutorial runs Consul without TLS and ACLs, which is fine for a private lab network. Production clusters exposed to less trusted networks should enable both, mentioned briefly under Best Practices.
Step 1: Install Consul on Every Node
HashiCorp ships Consul through its own APT repository. Run this on all four machines (consul-01, consul-02, consul-03, and app-01):
sudo apt update
sudo apt install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y consul
Verify the install:
consul version
You should see output similar to:
Consul v1.19.2
Revision e778b8d9
Build Date 2024-06-17T18:12:52Z
Protocol 2 spoken by default, understanding 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
The APT package also creates a consul system user and a systemd unit at /lib/systemd/system/consul.service, which we will use later instead of running Consul manually in a terminal.
Step 2: Configure the First Server Agent
Consul reads its configuration from JSON or HCL files in /etc/consul.d/. On consul-01, create the base config:
sudo mkdir -p /etc/consul.d
sudo mkdir -p /opt/consul
sudo chown -R consul:consul /opt/consul
Generate a shared encryption key that all agents in the cluster will use to encrypt gossip traffic:
consul keygen
Copy the output (a short base64 string); you will reuse it on every node. Now create /etc/consul.d/consul.hcl:
sudo tee /etc/consul.d/consul.hcl > /dev/null <<'EOF'
datacenter = "dc1"
data_dir = "/opt/consul"
node_name = "consul-01"
server = true
bootstrap_expect = 3
bind_addr = "10.20.0.31"
client_addr = "0.0.0.0"
retry_join = ["10.20.0.31", "10.20.0.32", "10.20.0.33"]
encrypt = "PASTE_YOUR_KEYGEN_OUTPUT_HERE"
ui_config {
enabled = true
}
EOF
A few of these fields matter more than they look:
bootstrap_expect = 3tells this server how many server agents to wait for before electing a leader. Set it to the same number on all three servers so the cluster does not accidentally bootstrap itself with just one node.retry_joinlists the addresses agents should try when looking for the rest of the cluster, so a restarted node can rejoin automatically instead of needing a manualconsul join.bind_addris the address Consul advertises to other agents; it must be reachable from the other nodes, not127.0.0.1.encryptenables gossip encryption; every agent, server or client, needs the exact same key.
Now start Consul:
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul
Step 3: Bring Up the Other Two Servers
Repeat step 2 on consul-02 and consul-03, changing only node_name and bind_addr to match each machine (consul-02 / 10.20.0.32, and consul-03 / 10.20.0.33), while keeping retry_join, bootstrap_expect, and encrypt identical across all three.
Once all three are running, check cluster membership from any node:
consul members
Expected output:
Node Address Status Type Build Protocol DC Partition Segment
consul-01 10.20.0.31:8301 alive server 1.19.2 2 dc1 default <all>
consul-02 10.20.0.32:8301 alive server 1.19.2 2 dc1 default <all>
consul-03 10.20.0.33:8301 alive server 1.19.2 2 dc1 default <all>
You can also check which node holds the Raft leadership:
consul operator raft list-peers
If all three show alive and one is marked as the leader, your cluster has successfully bootstrapped.
Step 4: Install a Client Agent Alongside a Real Service
On app-01, install Consul the same way as step 1, then write a client config at /etc/consul.d/consul.hcl:
sudo tee /etc/consul.d/consul.hcl > /dev/null <<'EOF'
datacenter = "dc1"
data_dir = "/opt/consul"
node_name = "app-01"
server = false
bind_addr = "10.20.0.41"
client_addr = "0.0.0.0"
retry_join = ["10.20.0.31", "10.20.0.32", "10.20.0.33"]
encrypt = "PASTE_YOUR_KEYGEN_OUTPUT_HERE"
EOF
The only real differences from the server config are server = false and the absence of bootstrap_expect. Start it the same way:
sudo mkdir -p /opt/consul
sudo chown -R consul:consul /opt/consul
sudo systemctl enable consul
sudo systemctl start consul
Run consul members again from any node; you should now see app-01 listed with Type of client.
Now let’s put an actual service on this node. Assume you have a small Node.js API listening on port 3000 with a /health endpoint that returns HTTP 200 when the process is ready. Start it however you normally would, for example with a systemd unit or pm2, so it stays running:
curl -s http://localhost:3000/health
OK
Step 5: Register the Service
Consul services are defined in JSON files under /etc/consul.d/. Create /etc/consul.d/api.json on app-01:
sudo tee /etc/consul.d/api.json > /dev/null <<'EOF'
{
"service": {
"name": "api",
"port": 3000,
"tags": ["nodejs", "v1"],
"check": {
"http": "http://localhost:3000/health",
"interval": "10s",
"timeout": "2s"
}
}
}
EOF
This tells Consul: there is a service called api on this node, listening on port 3000, and to consider it healthy only if GET /health returns a 2xx response within 2 seconds, checked every 10 seconds. Reload Consul so it picks up the new service definition without a full restart:
consul reload
Confirm the service is registered and healthy:
consul catalog services
api
consul
curl -s http://localhost:8500/v1/health/service/api?passing | jq '.[0].Service'
{
"ID": "api",
"Service": "api",
"Tags": ["nodejs", "v1"],
"Address": "",
"Port": 3000
}
The ?passing filter is important: it only returns instances currently passing their health checks, so a stale or crashed instance never gets handed to a caller.
Step 6: Discover the Service Two Ways
Via DNS. Consul’s DNS interface listens on port 8600. From any node in the cluster, query it directly with dig:
sudo apt install -y dnsutils
dig @127.0.0.1 -p 8600 api.service.consul SRV
You will get back an SRV record pointing at app-01 on port 3000, along with an A record resolving to 10.20.0.41. This is exactly the kind of lookup you would configure an application or an Nginx upstream to perform instead of hardcoding an IP.
Via the HTTP API. This is what most application code and automation scripts actually use, since it returns structured JSON instead of DNS records:
curl -s http://10.20.0.31:8500/v1/catalog/service/api | jq
[
{
"Node": "app-01",
"Address": "10.20.0.41",
"ServiceName": "api",
"ServicePort": 3000,
"ServiceTags": ["nodejs", "v1"]
}
]
Via the web UI. Open http://10.20.0.31:8500/ui in a browser. You will see the api service listed under Services, its single healthy instance, and the health check passing in green. This UI is genuinely useful during an incident: you can see at a glance which instances of which service are currently failing their checks across the entire cluster.
Common Mistakes & Troubleshooting
Cluster never elects a leader. This almost always means bootstrap_expect does not match the actual number of servers, the encrypt key differs between nodes, or a firewall is blocking ports 8300 to 8302 (TCP and UDP) between the server nodes. Check with:
sudo journalctl -u consul -n 50 --no-pager
Look for repeated “no cluster leader” or “failed to sync remote state” messages.
Service shows up but is always “critical.” This usually means the health check URL is not reachable from the node running the check. Remember the check runs on the local client agent, so if your app is bound to 127.0.0.1 only, that is fine for a local check, but confirm the port and path actually respond with curl from that same node before assuming Consul is misconfigured.
consul reload does not pick up a new service file. Double-check the file is valid JSON (a trailing comma is a common culprit) and that it lives directly under /etc/consul.d/, not a subdirectory, unless you explicitly added that path with -config-dir.
Client agent never appears in consul members. Check that retry_join on the client lists reachable server IPs, and that UDP/TCP port 8301 is open between the client and every server, not just the first one in the list.
Gossip encryption key mismatch. If you regenerate the key with consul keygen after some agents are already running with an old key, every agent needs the update, not just the new ones. Mismatched keys silently prevent agents from joining rather than producing an obvious error.
Best Practices
- Run an odd number of server agents (3 or 5), never an even number, so the cluster can always establish a majority during a network partition.
- Enable TLS and ACLs before running Consul on anything but an isolated lab network. The setup in this guide has no authentication on the HTTP API or ACL system enabled, which means anyone who can reach port 8500 can register or deregister services. Production deployments should enable
acl { enabled = true, default_policy = "deny" }and issue scoped tokens per service. - Keep health checks cheap and specific. A check that hits a real
/healthendpoint reflecting actual readiness (database connection open, migrations applied) is far more useful than one that only confirms the process is running. - Separate server and client roles onto different machines in anything beyond a lab, so a runaway application on a client node cannot starve the CPU or disk that your consensus servers depend on.
- Back up Consul’s Raft state on a regular schedule with
consul snapshot save backup.snap, especially before upgrades, since restoring a snapshot is far faster than rebuilding cluster state and key/value data from scratch. - Monitor gossip health, not just Raft leadership. A cluster can have a healthy leader while individual client agents silently drop out of the gossip pool on a flaky network link, which
consul memberswill reveal but a leader-only check will not.
Conclusion
You now have a three-node Consul server cluster, a client agent running alongside a real application, and a service that is discoverable both over DNS and the HTTP API, backed by an actual health check instead of a hope that the process is still running. That is the foundation service discovery is built on: instead of every application maintaining its own list of where things live, they all ask the same source of truth, and that source of truth actively verifies the answer before handing it out.
From here, a natural next step is wiring Consul’s DNS interface directly into an Nginx or HAProxy upstream so load balancer configuration updates itself as instances come and go, or exploring Consul Connect for service-to-service mTLS without changing application code. If you are already running Vault alongside this cluster, Consul can also serve as Vault’s storage backend, tying the two tools together operationally as well as topically.