With RabbitMQ we can scale-out our consumers by simply adding more, but we can also scale-out our queues. There are a few reasons why scaling out our queues might be preferential to simply adding more consumers to a single queue (competing consumers), one of those reasons is when using the work queue pattern.
Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1
This is the first post in a series that will look at bringing Kafka features to RabbitMQ. In this post we'll look at how we can partition a RabbitMQ queue into multiple queues, perform automatic queue assignment to consumers and perform automatic rebalancing as the number of queues and consumers change.
Event-Driven Architectures - Queue vs Log - A Case Study
In the previous post we looked at relative event ordering and the decoupling of publishers and consumers among other things. In this post we'll take those concepts and look at an example architecture. We'll look at the various modelling possibilities we have with RabbitMQ representing a queue based system, and Kafka representing a log based system.
Event-Driven Architectures - The Queue vs The Log
A messaging system is at the heart of most event-driven architectures and there are a plethora of different technologies in the space and they can be classified as either queue based or log based.
Queue based: RabbitMQ, ActiveMQ, MSMQ, AWS SQS, JMQ and many more.
Log based: Apache Kakfa, Apache Pulsar, AWS Kinesis, Azure Event Hubs and many more.
Each messaging system has different features but at the heart are their data structures: queue or log. In this post we'll take a look at how the underlying data structure affects your event-driven architecture.
RabbitMQ vs Kafka Part 4 - Message Delivery Semantics and Guarantees
Both RabbitMQ and Kafka offer durable messaging guarantees. Both offer at-most-once and at-least-once guarantees but kafka offers exactly-once guarantees in a very limited scenario.
Let's first understand what these guarantees mean:
- At-most-once delivery. This means that a message will never be delivered more than once but messages might be lost.
- At-least-once delivery. This means that we'll never lose a message but a message might end up being delivered to a consumer more than once.
- Exactly-once delivery. The holy grail of messaging. All messages will be delivered exactly one time.
Delivery is probably the wrong word for the above terms, instead Processing might be a better way of putting it. After all what we care about is whether a consumer can process a message and whether that is at-most-once, at-least-once or exactly-once. But using the word processing complicates things, exactly-once delivery makes less sense now as perhaps we need it to be delivered twice in order to be able to successfully process it once. If the consumer dies during processing, then we need that the message be delivered a second time for a new consumer.
RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns
In Part 2 we covered the patterns and topologies that RabbitMQ enables. In this part we'll look at Kafka and contrast it against RabbitMQ to get some perspective on their differences. Remember that this comparison is within the context of an event-driven application architecture rather than data processing pipelines, although the line between them can be a bit grey. Perhaps it is more like a continuum and this comparison focuses on the event-driven applications end of that continuum.
RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns
In this part we're going to forget about the low level details in the protocols and concentrate on the higher level patterns and message topologies that can be achieved in RabbitMQ. In Part 3 of the series we'll do the same for Apache Kafka.
First we'll cover the building blocks, or routing primitives, of RabbitMQ:
- Exchange types and bindings
- Queues
- Dead letter exchanges
- Ephemeral exchanges and queues
- Alternate Exchanges
- Priortity Queues
Then we'll combine them all into a set of example patterns.
RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging
In this part we'll explore what RabbitMQ and Apache Kafka are and their approach to messaging. Each technology has made very different decisions regarding every aspect of their design, each with strengths and weaknesses. We'll not come to any strong conclusions in this part, instead think of this as a primer on the technologies so we can dive deeper in subsequent parts of the series
RabbitMQ vs Kafka Series Introduction
Messaging is at the core of many architectures and two giants in the messaging space are RabbitMQ and Apache Kafka. In this series we'll be taking a deep look at RabbitMQ and Kafka within the context of real-time event-driven architectures. We're not talking about data processing and analytics pipelines which is where Kafka clearly shines, but as the messaging platform for business domain events.
Apache Kafka is ascendant right now and RabbitMQ is not talked about as much these days as it was. The hype has been centered on Kafka for good reason but RabbitMQ is still a great choice for messaging. One reason Kafka has stolen the limelight is the industry's obsession with scalability and clearly Kafka is more scalable than RabbitMQ but most of us don't deal with a scale where RabbitMQ has problems. Most of us aren't Google or Facebook. Most of us deal with daily message volumes of hundreds of thousands to hundreds of millions, not billions to trillions. Though I know that people have scaled RabbitMQ to billions of daily messages.
So in this series we're going to largely ignore extreme scalability and concentrate on the killer features that both messaging systems offer. What is so interesting is that they do both have fantastic features but they are so different. I may have written about RabbitMQ a fair amount in the past but I have no special affinity or bias towards it. I appreciate well made technology and both RabbitMQ and Kafka are mature, reliable and yes, scalable, messaging systems.
In this series we'll start at a high level and then start exploring different aspects of the two technologies. This series is for the messaging junkie or architect/engineer that wants to understand the lower level details and their implications. We'll not be writing code in this series, instead we'll focus on the functionality offered by both systems, the messaging patterns each enables and the decisions engineers and architects need to make.
We'll cover:
Part 1 - Two different takes on messaging (high level design comparison)
Part 5 - Fault tolerance and high availability with RabbitMQ
Also check out a couple of webinars on RabbitMQ vs Kafka I did with Erlang Solutions:
Finally, if you really want to see how RabbitMQ and Kafka cope with different failure scenarios then check out:
How to Make Your Messaging System Reliable and Keep Your Support Engineers Happy
When a messaging system lies at the heart of your application architecture you need to make it easy to respond to message processing failures. The more queues you have the more important a coherent incident response capability is. Unfortunately, it is all too common to see a chaotic policy or no policy at all for handling messages that cannot be processed successfully. Messages get delayed or lost on a regular basis and no-one is even sure how many.
The normal approach is to create a dead letter queue on a queue by queue basis and send messages there that cannot be processed. But what do you do from there? In this post we'll be looking at a message lifecycle baked into the messaging architecture that can solve this problem.