Message Queue vs Pub/Sub

If you are learning about microservices, you must have heard the term Pub/Sub or Message Queue. Pub/Sub and Message Queue are often used for alternative methods of communication between microservices other than HTTP.

When I first learned about Pub/Sub and Message Queues, I was very confused about the difference between the two. So here I made a small project using NodeJs and Redis to see the difference, and I will share it with all of you.

I hope you guys will understand better the difference between Pub/Sub and Message Queue.

Pub/Sub and Message Queue Overview

Well, I don’t know about you, but the definition does not matter that much to me as a developer. What I know is Pub/Sub and Message Queue is some kind of method to send a message/event asynchronously from 1 service to another service.

So, how is that different from HTTP? Well, HTTP can send messages directly from point A to B, while Pub/Sub and Message Queue need some intermediary. You can look at the diagram below:

This is HTTP communication

 ___                  ___
|   | ---- msg ----> |   |
| A |                | B |
|___| <--- ack ----- |___|

In HTTP communication, A directly sends the message to B, and Bsends an acknowledge message back, so A knows for sure that Breceives the message.

This is pub/sub communication

 ___               ___               ___
|   | --- msg --> |   |             |   |
| A | <-- ack --- | X | --- msg --> | B |
|___|             |___|             |___|

In Pub/Sub, Apublishes a message to X (a Pub/Sub service) in a specific channel, and then X will broadcast the message to anyone who subscribes to the channel. So if there is another service like C, D, E that subscribes to the channel, they will also get the same message as B. If no service subscribes to the channel, then X will drop the message.

In this communication, A doesn’t know who got the message and there is no guarantee that the message will be sent to other services.

This is Message Queue communication

 ___               ___               ___
|   | --- msg --> |   |             |   |
| A | <-- ack --- | X | <-- req --- | B |
|___|             |___| --- msg --> |___|

In Message Queue, Aproduces a message to X, and then X will wait for another service to request and consume the message. In this case, Bsends a request to X to see if there is a message. The X will then give the message to B. If no one requests the message, then the X will keep the message until someone requests it.

In this communication, A doesn’t know who got the message, but it is guaranteed that the message will be sent to other services.

Setup the Demo

Let’s get to the demo, I already prepared the source code if you want to try and see it for yourself. You can clone the repo here: pubsub vs queue. This demo is using Nodejs Language and Redis as Pub/Sub and Message Queue systems.

You can look at the README.md file to set up the application.

Pub/Sub

This is the topology for the Pub/Sub:

In this demo, app1 is the publisher that will publish a message to Redis Pub/Sub each second, while app2 and app3 subscribe to Redis to get the message from app1. Whenever app2 and app3 got the message, they will do a post request to app1 to tell app1 that they got the message.

If we run all of the apps and monitor the app1 log, we will see something like this:

sending message: I want a pizza 22 to redis pubsub..
app2 handle: pizza 22
app3 handle: pizza 22
sending message: I want a pizza 23 to redis pubsub..
app2 handle: pizza 23
app3 handle: pizza 23
sending message: I want a pizza 24 to redis pubsub..
app3 handle: pizza 24
app2 handle: pizza 24

This explains that when app1 send a message: I want a pizza 22 to Redis, it will send the message to both app2 and app3.

Let’s see, what if we turn app2 and app3 off for a while, and then we turn them on again. You will see the log will be like this:

sending message: I want a pizza 51 to redis pubsub..
app2 handle: pizza 51
app3 handle: pizza 51
sending message: I want a pizza 52 to redis pubsub..
app3 handle: pizza 52                                 # app2 is turned off
sending message: I want a pizza 53 to redis pubsub..  # app3 is turned off
sending message: I want a pizza 54 to redis pubsub..
sending message: I want a pizza 55 to redis pubsub..
sending message: I want a pizza 56 to redis pubsub..
sending message: I want a pizza 57 to redis pubsub..
sending message: I want a pizza 58 to redis pubsub..
sending message: I want a pizza 59 to redis pubsub..
app2 handle: pizza 59                                 # app2 is turned on again
sending message: I want a pizza 60 to redis pubsub..
app2 handle: pizza 60
sending message: I want a pizza 61 to redis pubsub..
app2 handle: pizza 61
sending message: I want a pizza 62 to redis pubsub..
app2 handle: pizza 62
app3 handle: pizza 62                                 # app3 is turned on again
sending message: I want a pizza 63 to redis pubsub..
app2 handle: pizza 63
app3 handle: pizza 63

From here, we can see the are no apps handling pizza numbers 53 - 59. app1keeps publishing messages to Redis, but there is no other service subscribed to it (because we turn both apps off), so Redis drops the message. When we turn both apps up, Redis will once again broadcast the message to the available app at that time.

Message Queue

This is the topology for the queue:

Same with the Pub/Sub, app1 is the producer that will produce a message to the Redis queue each second, while app2 and app3do an infinite loop request to the Redis to get the message from app1. Whenever one of app2 or app3 got the message, they will do a post request to app1 to tell app1 that they got the message.

If we run all of the apps and monitor the app1 log, we will see something like this:

sending message: I want a pizza 39 to redis queue..
app2 handle: pizza 39
sending message: I want a pizza 40 to redis queue..
app3 handle: pizza 40
sending message: I want a pizza 41 to redis queue..
app2 handle: pizza 41
sending message: I want a pizza 42 to redis queue..
app3 handle: pizza 42

You can already see the difference between Message Queue and Pub/Sub here. There will be only one app that handles the pizza, either app2 or app3. Redis is not broadcasting the message, Redis will only give the message to 1 service at a time, and then delete it so no one gets the message again.

Now let’s see, what if we turn app2 and app3 off for a while, and then we turn them on again. You will see the log will be like this:

sending message: I want a pizza 8 to redis queue..
app2 handle: pizza 8
sending message: I want a pizza 9 to redis queue..
app3 handle: pizza 9
sending message: I want a pizza 10 to redis queue..
app2 handle: pizza 10
sending message: I want a pizza 11 to redis queue..
app3 handle: pizza 11                                # app2 is turned off
sending message: I want a pizza 12 to redis queue..
app3 handle: pizza 12
sending message: I want a pizza 13 to redis queue..  # app3 is turned off
sending message: I want a pizza 14 to redis queue..
sending message: I want a pizza 15 to redis queue..
sending message: I want a pizza 16 to redis queue..
app2 handle: pizza 13                                # app2 is turned on again
app2 handle: pizza 14
app2 handle: pizza 15
app2 handle: pizza 16
sending message: I want a pizza 17 to redis queue..
app2 handle: pizza 17
sending message: I want a pizza 18 to redis queue..
app2 handle: pizza 18
sending message: I want a pizza 19 to redis queue..
app2 handle: pizza 19
sending message: I want a pizza 20 to redis queue..
app3 handle: pizza 20                                # app3 is turned on again
sending message: I want a pizza 21 to redis queue..
app2 handle: pizza 21
sending message: I want a pizza 22 to redis queue..
app3 handle: pizza 22
sending message: I want a pizza 23 to redis queue..
app2 handle: pizza 23
sending message: I want a pizza 24 to redis queue..
app3 handle: pizza 24

From here, we can see that pizza numbers 13-16 is handled after 1 of the app is turned on. Different from Pub/Sub, Redis as a Message Queue system will save the message until someone requests to consume the message. As you can see pizza number 13-16 is handled by app2 as soon as the app2 is turned on.

Conclusion

Let’s make a table to summarize the difference between Pub/Sub and Message Queue in this experiment:

That’s it, now you know the practical differences between Message Queue and Pub/Sub. Hope you can choose correctly before implementing them in your use case.