More failure scenarios! See part 1 for the first seven scenarios.
How to Lose Messages on a Kafka Cluster - Part 1
In my previous post I used Blockade, Python and some Bash scripts to test a RabbitMQ cluster under various failure conditions such as failed nodes, network partitions, packet loss and a slow network. The aim was to find out how and when a RabbitMQ cluster loses messages. In this post we’ll do exactly the same but with a Kafka cluster. We’ll use our knowledge of the inside workings of Kafka and Zookeeper to produce various failure modes that produce message loss. Please read my post on Kafka fault tolerance as this post assumes you understand the basics of the acknowledgements and replication protocol.
How to Lose Messages on a RabbitMQ Cluster
In my RabbitMQ vs Kafka series Part 5 post I covered the theory of RabbitMQ clustering and some of the gotchas. In this post we'll demonstrate the message loss scenarios described in that post using Docker and Blockade. I recommend you read that post first as this post assumes understanding of the topics covered.
Blockade is a really easy way to test out how distributed systems cope with network partitions, flaky networks and slow networks. It was inspired by the Jepson series. In this post we'll either be killing off nodes, partitioning the cluster, introducing packet loss or slowing down the network. So with Blockade, some bash and python scripts we’ll test out some failure scenarios.
RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka
In the last post we took a look at the RabbitMQ clustering feature for fault tolerance and high availability. In this post we'll dig deep into Apache Kafka and its offering.
With Kafka the unit of replication is the partition. Each topic has one or more partitions and each partition has a leader and zero or more followers. When you create a topic you specify the number of partitions and the replication factor. A replication factor of three is common, this equates to one leader and two followers. Both leaders and followers can be referred to as replicas.
RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering
Fault tolerance and High Availability are big subjects and so we'll tackle RabbitMQ and Kafka in separate posts. In this post we'll look at RabbitMQ and in Part 6 we'll look at Kafka while making comparisons to RabbitMQ. This is a long post, even though we only look at RabbitMQ, so get comfortable.
In this post we'll look at the strategies for fault tolerance, consistency and high availability (HA) and the trade-offs each strategy makes. RabbitMQ can operate as a cluster of nodes and as such can be classed as a distributed system. When it comes to distributed data systems we often speak about consistency and availability.
We talk about consistency and availability with distributed systems because they describe how the system behaves under failure. A network link fails, a server fails, a hard disk fails, a server is temporarily unavailable due to GC or a network link is lossy or slow. All these things can cause outages, data loss or data conflicts. It turns out that it is generally not possible to provide a system that is ultimately consistent (no data loss, no data divergence) and available (will accept reads and writes) under all failure modes.
We'll see that consistency and availability are at two ends of a spectrum and you'll need to choose which of those you'll optimize for. The good news is that with RabbitMQ this is a choice that you can make. It gives you the nerd knobs required to tune it for greater consistency or greater availability.
In this post we'll be paying close attention to what configurations produce data loss of acknowledged writes. There is a chain of responsibility between producers, brokers and consumers. Once a message has been handed off to a broker, it is the broker's job not to lose that message. When the broker acknowledges receipt of a message to the publisher, we don't expect that message to be lost. But we'll see that this indeed can happen depending on your broker and publisher configuration.
RabbitMQ Work Queues: Avoiding Data Inconsistency with Rebalanser
With RabbitMQ we can scale-out our consumers by simply adding more, but we can also scale-out our queues. There are a few reasons why scaling out our queues might be preferential to simply adding more consumers to a single queue (competing consumers), one of those reasons is when using the work queue pattern.
Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1
This is the first post in a series that will look at bringing Kafka features to RabbitMQ. In this post we'll look at how we can partition a RabbitMQ queue into multiple queues, perform automatic queue assignment to consumers and perform automatic rebalancing as the number of queues and consumers change.
Event-Driven Architectures - Queue vs Log - A Case Study
In the previous post we looked at relative event ordering and the decoupling of publishers and consumers among other things. In this post we'll take those concepts and look at an example architecture. We'll look at the various modelling possibilities we have with RabbitMQ representing a queue based system, and Kafka representing a log based system.
Event-Driven Architectures - The Queue vs The Log
A messaging system is at the heart of most event-driven architectures and there are a plethora of different technologies in the space and they can be classified as either queue based or log based.
Queue based: RabbitMQ, ActiveMQ, MSMQ, AWS SQS, JMQ and many more.
Log based: Apache Kakfa, Apache Pulsar, AWS Kinesis, Azure Event Hubs and many more.
Each messaging system has different features but at the heart are their data structures: queue or log. In this post we'll take a look at how the underlying data structure affects your event-driven architecture.
RabbitMQ vs Kafka Part 4 - Message Delivery Semantics and Guarantees
Both RabbitMQ and Kafka offer durable messaging guarantees. Both offer at-most-once and at-least-once guarantees but kafka offers exactly-once guarantees in a very limited scenario.
Let's first understand what these guarantees mean:
- At-most-once delivery. This means that a message will never be delivered more than once but messages might be lost.
- At-least-once delivery. This means that we'll never lose a message but a message might end up being delivered to a consumer more than once.
- Exactly-once delivery. The holy grail of messaging. All messages will be delivered exactly one time.
Delivery is probably the wrong word for the above terms, instead Processing might be a better way of putting it. After all what we care about is whether a consumer can process a message and whether that is at-most-once, at-least-once or exactly-once. But using the word processing complicates things, exactly-once delivery makes less sense now as perhaps we need it to be delivered twice in order to be able to successfully process it once. If the consumer dies during processing, then we need that the message be delivered a second time for a new consumer.