Pub/Sub vs Message Queue

Written by: Bagus Facsi Aginsa
Published at: 10 Sep 2022


If you are learning about microservices, you must have heard the term Pub/Sub or Message Queue. Pub/Sub and Message Queue are often used for alternative methods of communication between microservices other than HTTP.

When I first learned about Pub/Sub and Message Queues, I was very confused about the difference between the two. So here I made a small project using NodeJs and Redis to see the difference, and I will share it with all of you.

I hope you guys will understand better the difference between Pub/Sub and Message Queue.

Pub/Sub and Message Queue Overview

Well, I don’t know about you, but the definition does not matter that much to me as a developer. What I know is Pub/Sub and Message Queue is some kind of method to send a message/event asynchronously from 1 service to another service.

So, how is that different from HTTP? Well, HTTP can send messages directly from point A to B, while Pub/Sub and Message Queue need some intermediary. You can look at the diagram below:

This is HTTP communication

 ___                  ___
|   | ---- msg ----> |   |
| A |                | B |
|___| <--- ack ----- |___|

In HTTP communication, A directly send the message to B, and B send an acknowledge message back, so A knows for sure that B receive the message.

This is pub/sub communication

 ___               ___               ___
|   | --- msg --> |   |             |   |
| A | <-- ack --- | X | --- msg --> | B |
|___|             |___|             |___|

In Pub/Sub, A publish a message to X (a Pub/Sub service) in a specific channel, and then X will broadcast the message to anyone who subscribes to the channel. So if there is another service like C, D, E that subscribes to the channel, they will also get the same message as B. If no service subscribes to the channel, then X will drop the message.

In this communication, A doesn’t know who got the message and there is no guarantee that the message will be sent to other services.

This is Message Queue communication

 ___               ___               ___
|   | --- msg --> |   |             |   |
| A | <-- ack --- | X | <-- req --- | B |
|___|             |___| --- msg --> |___|

In Message Queue, A produce a message to X, and then X will wait for another service to request and consume the message. In this case, B send a request to X to see if there is a message. The X will then give the message to B. If no one request the message, then the X will keep the message until someone requests it.

In this communication, A doesn’t know who got the message, but it is guaranteed that the message will be sent to other services.

Setup the Demo

Let’s get to the demo, I already prepare the source code if you want to try and see it for yourself. You can clone the repo here: pubsub vs queue. This demo is using Nodejs Language and Redis as Pub/Sub and Message Queue systems.

You can look at the README.md file to set up the application.

Pub/Sub

This is the topology for the Pub/Sub:

Pub/Sub Topology Demo

In this demo, app1 is the publisher that will publish a message to Redis Pub/Sub each second, while app2 and app3 subscribe to Redis to get the message from app1. Whenever app2 and app3 got the message, they will do a post request to app1 to tell app1 that they got the message.

If we run all of the apps and monitor the app1 log, we will see something like this:

sending message: I want a pizza 22 to redis pubsub..
app2 handle: pizza 22
app3 handle: pizza 22
sending message: I want a pizza 23 to redis pubsub..
app2 handle: pizza 23
app3 handle: pizza 23
sending message: I want a pizza 24 to redis pubsub..
app3 handle: pizza 24
app2 handle: pizza 24

This explains that when app1 send a message: I want a pizza 22 to Redis, it will send the message to both app2 and app3.

Let’s see, what if we turn app2 and app3 off for a while, and then we turn them on again. You will see the log will be like this:

sending message: I want a pizza 51 to redis pubsub..
app2 handle: pizza 51
app3 handle: pizza 51
sending message: I want a pizza 52 to redis pubsub..
app3 handle: pizza 52                                 # app2 is turned off
sending message: I want a pizza 53 to redis pubsub..  # app3 is turned off
sending message: I want a pizza 54 to redis pubsub..
sending message: I want a pizza 55 to redis pubsub..
sending message: I want a pizza 56 to redis pubsub..
sending message: I want a pizza 57 to redis pubsub..
sending message: I want a pizza 58 to redis pubsub..
sending message: I want a pizza 59 to redis pubsub..
app2 handle: pizza 59                                 # app2 is turned on again
sending message: I want a pizza 60 to redis pubsub..
app2 handle: pizza 60
sending message: I want a pizza 61 to redis pubsub..
app2 handle: pizza 61
sending message: I want a pizza 62 to redis pubsub..
app2 handle: pizza 62
app3 handle: pizza 62                                 # app3 is turned on again
sending message: I want a pizza 63 to redis pubsub..
app2 handle: pizza 63
app3 handle: pizza 63

From here, we can see the are no app handling the pizza number 53 - 59. app1 keep publishing messages to Redis, but there is no other service subscribed to it (because we turn both apps off), so Redis drop the message. When we turn both apps up, Redis will once again broadcast the message to the available app at that time.

Message Queue

This is the topology for the queue:

Message Queue Topology Demo

Same with the Pub/Sub, app1 is the producer that will produce a message to the Redis queue each second, while app2 and app3do an infinite loop request to the Redis to get the message from app1. Whenever one of app2 or app3 got the message, they will do a post request to app1 to tell app1 that they got the message.

If we run all of the apps and monitor the app1 log, we will see something like this:

sending message: I want a pizza 39 to redis queue..
app2 handle: pizza 39
sending message: I want a pizza 40 to redis queue..
app3 handle: pizza 40
sending message: I want a pizza 41 to redis queue..
app2 handle: pizza 41
sending message: I want a pizza 42 to redis queue..
app3 handle: pizza 42

You can already see the difference between Message Queue and Pub/Sub here. There will be only one app that handles the pizza, either app2 or app3. Redis is not broadcasting the message, Redis will only give the message to 1 service at a time, and then delete it so no one gets the message again.

Now let’s see, what if we turn app2 and app3 off for a while, and then we turn them on again. You will see the log will be like this:

sending message: I want a pizza 8 to redis queue..
app2 handle: pizza 8
sending message: I want a pizza 9 to redis queue..
app3 handle: pizza 9
sending message: I want a pizza 10 to redis queue..
app2 handle: pizza 10
sending message: I want a pizza 11 to redis queue..
app3 handle: pizza 11                                # app2 is turned off
sending message: I want a pizza 12 to redis queue..
app3 handle: pizza 12
sending message: I want a pizza 13 to redis queue..  # app3 is turned off
sending message: I want a pizza 14 to redis queue..
sending message: I want a pizza 15 to redis queue..
sending message: I want a pizza 16 to redis queue..
app2 handle: pizza 13                                # app2 is turned on again
app2 handle: pizza 14
app2 handle: pizza 15
app2 handle: pizza 16
sending message: I want a pizza 17 to redis queue..
app2 handle: pizza 17
sending message: I want a pizza 18 to redis queue..
app2 handle: pizza 18
sending message: I want a pizza 19 to redis queue..
app2 handle: pizza 19
sending message: I want a pizza 20 to redis queue..
app3 handle: pizza 20                                # app3 is turned on again
sending message: I want a pizza 21 to redis queue..
app2 handle: pizza 21
sending message: I want a pizza 22 to redis queue..
app3 handle: pizza 22
sending message: I want a pizza 23 to redis queue..
app2 handle: pizza 23
sending message: I want a pizza 24 to redis queue..
app3 handle: pizza 24

From here, we can see that pizza number 13-16 is handled after 1 of the app is turned on. Different from Pub/Sub, Redis as a Message Queue system will save the message until someone requests to consume the message. As you can see that pizza number 13-16 is handled by app2 as soon as the app2 is turned on.

Conclusion

Let’s make a table to summarize the difference between Pub/Sub and Message Queue in this experiment:

No. Pub/Sub Message Queue
1 The one who sends the message is called Publisher The one who sends the message is called Producer
2 The one who receives the message is called Subscriber The one who receives the message is called Consumer
3 Pub/Sub will broadcast a message to everyone subscribing to the channel Queue only gives a message to the one who requests the message first
4 Pub/Sub will drop the message if there is no one subscribing to the channel at the time Queue will save the message and wait until someone requests the message
5 There is no guarantee the message will be received by someone It is guaranteed the message will be received by someone.
6 Pub/Sub can transmit multiple of the same message to the subscribers Queue can only transmit the same message once to one of the consumers

That’s it, now you know the practical differences between Pub/Sub and Message Queue. Hope you can choose correctly before implementing them in your use case.