If you ever had difficulties understanding message brokers, as I did, then this article is the right place to start your journey.
Stay with me ????Message BrokerIn general, a broker is a person who facilitates trades between a buyer and a seller.
An example could be a real estate agent or a stockbroker.
Similarly, if we want to trade messages between two distributed software components, we need a mediator.
This mediator is known as the message broker.
It receives incoming messages from a sender and sends them to a recipient.
This way the sender and receiver can be totally isolated.
Another analogy for a message broker can be a Post Office (see Figure 1).
Let’s take a scenario where you are going to send a letter to your cousin living in another city.
Then as per this analogy, you are a producer, your cousin is a consumer, and the post office is a message broker.
Figure 1: Analogy for message brokerRabbitMQ as message brokerNow we know that the purpose of a message broker is to route messages from a producer to a consumer.
Let’s examine one such message broker — RabbitMQ.
It’s one of the most extensively used message brokers these days.
The way RabbitMQ routes messages depends upon the messaging protocol it implements.
RabbitMQ supports multiple messaging protocols.
However, the one we are interested in is AMQP.
It is an acronym for Advanced Message Queuing Protocol.
So without any further ado let’s have a closer look at the AMQP protocol model.
Advanced Message Queuing ProtocolThe conceptual model of AMQP is quite simple and straightforward.
It has three entities:QueueBindingExchangeWhen a publisher pushes a message to RabbitMQ, it first arrives at an exchange.
The exchange then distributes copies of these messages to variously connected queues.
Finally, consumers receive these messages.
Consider a message as a piece of data.
It is necessarily a package with a payload and some meta-data.
The payload contains full data whereas meta-data are properties used by RabbitMQ.
Figure 2 depicts a graphical representation of the AMQP model.
Figure 2: AMQP modelAMQP is a programmable protocol.
Programmers have the liberty to use libraries to configure entities (exchange, binding, and queue) as per their own needs.
A RabbitMQ admin has no role in setting up these entities.
There are plenty of libraries available to work with RabbitMQ.
You can choose from Nodejs, Python, .
Net, Java, and many more.
QueuesThese queues are somehow similar to the queues from our data structure classes.
RabbitMQ queues also follow FIFO — First-In-First-Out methodology.
A queue is a place where RabbitMQ stores messages/data.
Programmers can configure queues through available programming libraries.
You can make a queue durable ( with the Durability property) to safeguard your data in case the broker crashes.
You can also provide a name(with the Name property) to a queue.
Other than Name and Durability, a queue has a few other properties like auto-delete, exclusive, and arguments.
Before moving any further, it’s important to understand who is a direct consumer of these queues.
Moreover, how many ways can a user consume messages from a queue?ConsumersConsumers are the ones who are going to use messages stored in a queue.
It is possible to connect more than one consumer to a queue at a time.
Consumers can either pull the message from the queue by pooling it or queues can even push the message to various connect consumers.
BindingsBindings are the rules that a queue defines while establishing a connection with an exchange.
You can have a queue connected to multiple exchanges.
Every queue is also connected to a default exchange.
An exchange will use these bindings to route messages to queues.
Exchanges and their typesAn Exchange is a gateway to RabbitMQ for your messages.
The distance the message has to travel inside RabbitMQ depends on the type of exchange.
Primarily there are four types.
DirectFanoutTopicHeaderDirectThe name explains it all! — A direct exchange delivers a message directly to the queues that satisfy the below condition:Routing key == Binding key A routing key is an attribute of the message.
On the other hand, a binding key is something you specify while creating a binding between a queue and an exchange.
Figure 3 is a visual explanation of how messages flow while using a direct exchange.
A message originates from a producer (green circle) with a routing key — img.
resize.
Once it reaches the exchange (Orange circle), the exchange will try to find all queues with binding key — img.
resize.
In case of a match, the message is pushed to all matched queues (resize in our case).
If there is no match found, the message can be sent back to the producer or can even be discarded.
We are lucky that we found a match in our example ????Figure 3: Direct exchange VisualizerOnce the message reaches the desired queue (resize in our case), they are distributed in round-robin fashion to all the connected consumers (resizer.
1/resizer.
2 in our case).
By distributing messages in a round-robin fashion, RabbitMQ makes sure that the messages are load balanced.
You must have noticed that the queue named crop is not receiving any messages.
Because the routing key in this example is img.
resize.
To send messages to this queue, we need to send messages with a routing key that would match the binding key (say img.
crop for instance).
FanoutA Fanout exchange ignores routing keys and distributes a message to all the connected queues.
No wonder it is called Fanout (blowing messages to all connected queues! ????).
One of the use cases for this type of exchange is message broadcast.
Figure 4: Fanout exchange VisualizerPlease note that RabbitMQ will still do round robin if there is more than one consumer of the queue.
TopicA topic exchange routes a message by matching routing key with a pattern in the binding key.
Routing key == Pattern in binding key.
RabbitMQ uses two wild card characters for pattern matching * and #.
Use a * to match 1 word and a # to match 0 or more words.
Figure 5 is a visual depiction of a topic exchange.
Messages with routing key — logs.
error will match patterns — logs.
error and logs.
*.
Hence these messages will end up in the queues — only error and all logs.
Whereas for the producer at the bottom-left, messages with routing key— logs.
success will match patterns of binding key #success and logs.
* .
Hence these messages will end up in the queues — all logs and only success .
Figure 5: Topic exchange VisualizerThis type of exchange has a vast range of use cases.
It can be used in the publish-subscribe pattern, distributing relevant data to desiring workers processes and many more.
HeaderA header is a particular type of exchange that routes messages based on keys present in the message header.
It overlooks routing key attribute of the message.
When creating bindings for a header exchange, it is possible to bind a queue to match more than one header.
In such a case, RabbitMQ should know from the producer if it should match all or any of these keys.
A producer/application can do this by providing an extra flag called ‘x-match’.
‘x-match’ can have any or all values.
The first one mandates that only one value should match while the latter mandates that all must match.
Message AcknowledgementOnce a message reaches its destination, the broker should delete the message from the queue.
It is necessary because a queue overflow can occur if it keeps accumulating messages.
Before deleting any message, the broker must have a delivery acknowledgment.
There are two possible ways to acknowledge message delivery.
Automatic acknowledgment: Once a consumer receives the messageExplicit acknowledgment: When a consumer sends back an acknowledgmentIn most cases, explicit acknowledgment is used as it makes sure that the consumer has consumed the message without any failover.
What’s nextRabbitMQ is a very mature and useful product.
This article is only a high-level introduction to RabbitMQ.
I simplified the concepts to provide a reference point for you to move further.
Visit the RabbitMQ website for more complex topics.
Hope you like the article, until next time, keep Queuing????.