RabbitMQ vs Kafka Series Introduction

Messaging is at the core of many architectures and two giants in the messaging space are RabbitMQ and Apache Kafka. In this series we'll be taking a deep look at RabbitMQ and Kafka within the context of real-time event-driven architectures. We're not talking about data processing and analytics pipelines which is where Kafka clearly shines, but as the messaging platform for business domain events.

Apache Kafka is ascendant right now and RabbitMQ is not talked about as much these days as it was. The hype has been centered on Kafka for good reason but RabbitMQ is still a great choice for messaging. One reason Kafka has stolen the limelight is the industry's obsession with scalability and clearly Kafka is more scalable than RabbitMQ but most of us don't deal with a scale where RabbitMQ has problems. Most of us aren't Google or Facebook. Most of us deal with daily message volumes of hundreds of thousands to hundreds of millions, not billions to trillions. Though I know that people have scaled RabbitMQ to billions of daily messages.

So in this series we're going to largely ignore extreme scalability and concentrate on the killer features that both messaging systems offer. What is so interesting is that they do both have fantastic features but they are so different. I may have written about RabbitMQ a fair amount in the past but I have no special affinity or bias towards it. I appreciate well made technology and both RabbitMQ and Kafka are mature, reliable and yes, scalable, messaging systems.

In this series we'll start at a high level and then start exploring different aspects of the two technologies. This series is for the messaging junkie or architect/engineer that wants to understand the lower level details and their implications. We'll not be writing code in this series, instead we'll focus on the functionality offered by both systems, the messaging patterns each enables and the decisions engineers and architects need to make. 

We'll cover:

• Part 1 - Two different takes on messaging (high level design comparison)

• Part 2 - Messaging patterns and topologies with RabbitMQ

• Part 3 - Messaging patterns and topologies with Kafka

• Part 4 - Message delivery semantics and guarantees

• Part 5 - Fault tolerance and high availability with RabbitMQ

• Part 6 - Fault tolerance and high availability with Kafka

Also check out a couple of webinars on RabbitMQ vs Kafka I did with Erlang Solutions:

• RabbitMQ vs Kafka Part 1 - Erlang Solutions Webinar

• RabbitMQ vs Kafka Part 2 - Erlang Solutions Webinar

Finally, if you really want to see how RabbitMQ and Kafka cope with different failure scenarios then check out:

• How to Lose Messages on a RabbitMQ Cluster

• How to Lose Messages on a Kafka Cluster Part 1

• How to Lose Messages on a Kafka Cluster Part 2