This is a sister post to one I have written about multi-topic subscriptions with Apache Kafka that you can read on the CloudKarafka blog. I will provide a summary of those results before we get started with Apache Pulsar. The run the same tests in my tests of both technologies.

The objective is to get an understanding of what to expect from multi-topic subscriptions, specifically we are testing message ordering. Message ordering is a fundamental component of messaging systems and even though cross topic ordering is not guaranteed by Pulsar or Kafka, I find it interesting and useful to know what to expect.

First an Apache Kafka Summary

What I discovered in my Apache Kafka tests is that a Kafka consumer is capable of delivering messages in perfect order so long as the consumer is able to consume in real-time. I am not talking hand-wavy real-time but immediate consumption. Any jitter in the consumer processing time, or if the producer pulls ahead in anyway causes ordering to be lost. Likewise reading from the beginning of two topics produces no ordering at all.

It seems that a Kafka consumer will pull from its subscribed topics in a given order (logically speaking). This seems to hold for the duration of the consumer, for example, when it calls poll() it might be served from topic2, then from topic1.

When the consumer is keeping up with the producers and the unconsumed total is less than the max.poll.records, then we see pretty decent cross topic ordering, even perfect ordering.

But if you lose your strict real-time consumption, like when the unconsumed message total is greater than your max.poll.records, then you’ll see large stretches where only a single topic is being pulled from.

So for example, your consumer is fetching in the order:

topic2, partition 0
topic1, partition 0

So if there are 700 records in each topic, max.poll.records is 500 and we start at the earliest offset, then it will fetch:

poll() - 500 from topic2, partition 0
poll() - 200 from topic2, partition 0 and 300 from topic1, partition 0
poll() - 400 from topic1, partition 0

This means that if any of the below apply to you:

You can get periods where producers are much faster than consumers
Or you want to consume from the start of the topics
Or you want to reset your offset back a while
Or you have a process that wakes up every once and a while consumes from two or more topics

then don’t count on any ordering between topics at all.

No criticism of Apache Kafka, we’re just delving into its behaviour. It does state that there are no ordering guarantees across partitions.

A Dive into Apache Pulsar

So what’s the story with Apache Pulsar. Overall it is better at delivering messages in order across topics than Kafka, but still loses ordering when real-time consumption cannot be achieved, however, the way it buffers messages internally makes it less sensitive to processing time jitter and is able to deliver messages in the right order more often. Let’s have a look.

As with the Apache Kafka tests we have a single producer, producing to two topics sending a sequence of messages with two counters:

Global Counter, incremented on each message send
Topic Counter, each topic has its own counter, which is incremented when sending a message to that topic

The Global Counter will tell us if we get cross topic ordering, it creates a global sequence and the Topic Counter will tell us if each topic delivers messages in the right order. As with Kafka, we should ALWAYS get perfect topic ordering, but cross topic ordering will not always be achieved.

Spoiler, topic ordering was always perfect in all tests as expected.

So as an example, the producer sends out messages as follows:

Topic 1 -> { “GlobalCounter”: 1, “TopicCounter”: 1 }
Topic 2 -> { “GlobalCounter”: 2, “TopicCounter”: 1 }
Topic 1 -> { “GlobalCounter”: 3, “TopicCounter”: 2 }
Topic 2 -> { “GlobalCounter”: 4, “TopicCounter”: 2 }
Topic 1 -> { “GlobalCounter”: 5, “TopicCounter”: 3 }
Topic 2 -> { “GlobalCounter”: 6, “TopicCounter”: 3 }
Topic 1 -> { “GlobalCounter”: 7, “TopicCounter”: 4 }
Topic 2 -> { “GlobalCounter”: 8, “TopicCounter”: 4 }
Topic 1 -> { “GlobalCounter”: 9, “TopicCounter”: 5 }
Topic 2 -> { “GlobalCounter”: 10, “TopicCounter”: 5 }

That should leave topic1 and topic2 as follows:

Fig 1: The messages in each topic when sending in a round-robin fashion

Our test strategy

Let’s first clarify some terms I am going to use:

Tailing the topic vs Time-travel: A consumer that is tailing the topic is trying to consume messages in real-time as they arrive. There are more cases for topic tailing as perturbations to the real-timeness can affect ordering. Time-travel refers to consuming messages from the beginning of the topic, essentially turning back time to read messages from the past.
Equal vs unequal message arrival rate: Equal means a producer sends messages in a round robin fashion to each topic, meaning that each topic receives messages at the same rate. Unequal rate is where one topic receives messages at a different rate, for example for each messages that arrives at topic 1, ten arrive at topic 2
Zero processing time vs constant time vs jitter: How long a consumer takes to process each message.
Fast producer: The producer sends messages faster than the consumer can consume. Means that the consumer no longer achieves strict real-time consumption.

Having played around with this for a few days and had time to think it over, I see the following cases:

Tailing the topic, equal message arrival rate, zero processing time
Tailing the topic, equal message arrival rate, short constant processing time
Tailing the topic, equal message arrival rate, short processing time with jitter
Tailing the topic, equal message arrival rate, fast producer
Tailing the topic,unequal message arrival rate, zero processing time
Tailing the topic, unequal message arrival rate, short constant processing time
Tailing the topic, unequal message arrival rate, short processing time with jitter
Tailing the topic, unequal message arrival rate, fast producer
Time-travel, equal message arrival rate
Time-travel, unequal message arrival rate

Case 1 - Tailing the topic, equal message arrival rate, zero processing time

In this case we have the producer sending messages to the two topics as fast it can, synchronously. It is synchronous to doubly make sure that delivery is made in the expected order. The consumer has essentially zero processing time as all it does is check message ordering and nothing else.

20:31:25.230Z : Consumer : Subscribing
20:31:25.691Z : Consumer : Consuming
20:31:25.967Z : Producer : Topic: topic1, TC: 1, GC: 1
20:31:25.972Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
20:31:25.974Z : Producer : Topic: topic2, TC: 1, GC: 2
20:31:25.975Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 OK
20:31:25.979Z : Producer : Topic: topic1, TC: 2, GC: 3
20:31:25.981Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 OK
20:31:25.985Z : Producer : Topic: topic2, TC: 2, GC: 4
20:31:25.987Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 OK
20:31:25.991Z : Producer : Topic: topic1, TC: 3, GC: 5
20:31:25.992Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 OK
20:31:25.997Z : Producer : Topic: topic2, TC: 3, GC: 6
20:31:25.998Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 OK
20:31:26.002Z : Producer : Topic: topic1, TC: 4, GC: 7
20:31:26.003Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 OK
20:31:26.007Z : Producer : Topic: topic2, TC: 4, GC: 8
20:31:26.009Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 OK
20:31:26.013Z : Producer : Topic: topic1, TC: 5, GC: 9
20:31:26.014Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 OK
20:31:26.018Z : Producer : Topic: topic2, TC: 5, GC: 10
20:31:26.020Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 OK
20:31:26.023Z : Producer : Topic: topic1, TC: 6, GC: 11
20:31:26.025Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 OK
20:31:26.027Z : Producer : Topic: topic2, TC: 6, GC: 12
20:31:26.031Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 OK
20:31:26.033Z : Producer : Topic: topic1, TC: 7, GC: 13
20:31:26.036Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 OK
20:31:26.039Z : Producer : Topic: topic2, TC: 7, GC: 14
20:31:26.042Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 OK
20:31:26.044Z : Producer : Topic: topic1, TC: 8, GC: 15
20:31:26.047Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 OK
20:31:26.049Z : Producer : Topic: topic2, TC: 8, GC: 16
20:31:26.054Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 OK
20:31:26.054Z : Producer : Topic: topic1, TC: 9, GC: 17
20:31:26.058Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 OK
20:31:26.060Z : Producer : Topic: topic2, TC: 9, GC: 18
20:31:26.065Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 OK
20:31:26.065Z : Producer : Topic: topic1, TC: 10, GC: 19
20:31:26.069Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 OK
20:31:26.071Z : Producer : Topic: topic2, TC: 10, GC: 20
20:31:26.076Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 OK
20:31:26.076Z : Producer : Topic: topic1, TC: 11, GC: 21
20:31:26.080Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 OK
20:31:26.082Z : Producer : Topic: topic2, TC: 11, GC: 22
20:31:26.087Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 OK
20:31:26.087Z : Producer : Topic: topic1, TC: 12, GC: 23
20:31:26.090Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 OK
20:31:26.092Z : Producer : Topic: topic2, TC: 12, GC: 24
20:31:26.097Z : Producer : Topic: topic1, TC: 13, GC: 25
20:31:26.098Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 OK
20:31:26.102Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 OK
20:31:26.103Z : Producer : Topic: topic2, TC: 13, GC: 26
20:31:26.107Z : Producer : Topic: topic1, TC: 14, GC: 27
20:31:26.109Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 OK
20:31:26.112Z : Producer : Topic: topic2, TC: 14, GC: 28
20:31:26.113Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 OK
20:31:26.117Z : Producer : Topic: topic1, TC: 15, GC: 29
20:31:26.120Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 OK
20:31:26.122Z : Producer : Topic: topic2, TC: 15, GC: 30
20:31:26.123Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 OK
20:31:26.127Z : Producer : Topic: topic1, TC: 16, GC: 31
20:31:26.131Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 OK
20:31:26.132Z : Producer : Topic: topic2, TC: 16, GC: 32
20:31:26.135Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 OK
20:31:26.137Z : Producer : Topic: topic1, TC: 17, GC: 33
20:31:26.142Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 OK
20:31:26.142Z : Producer : Topic: topic2, TC: 17, GC: 34
20:31:26.145Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 OK
20:31:26.148Z : Producer : Topic: topic1, TC: 18, GC: 35
20:31:26.153Z : Producer : Topic: topic2, TC: 18, GC: 36
20:31:26.154Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 OK
20:31:26.154Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 JUMP FORWARD 1
20:31:26.157Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 JUMP BACKWARDS 1
20:31:26.158Z : Producer : Topic: topic1, TC: 19, GC: 37
20:31:26.163Z : Producer : Topic: topic2, TC: 19, GC: 38
20:31:26.165Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 JUMP FORWARD 2
20:31:26.168Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 JUMP BACKWARDS 1
20:31:26.168Z : Producer : Topic: topic1, TC: 20, GC: 39
20:31:26.173Z : Producer : Topic: topic2, TC: 20, GC: 40
20:31:26.176Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 JUMP FORWARD 2
20:31:26.178Z : Producer : Topic: topic1, TC: 21, GC: 41
20:31:26.178Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 JUMP BACKWARDS 1
20:31:26.179Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 JUMP FORWARD 1
20:31:26.185Z : Producer : Topic: topic2, TC: 21, GC: 42
20:31:26.187Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 OK
20:31:26.190Z : Producer : Topic: topic1, TC: 22, GC: 43
20:31:26.191Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 OK
20:31:26.196Z : Producer : Topic: topic2, TC: 22, GC: 44
20:31:26.198Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 OK
20:31:26.201Z : Producer : Topic: topic1, TC: 23, GC: 45
20:31:26.203Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 OK
20:31:26.206Z : Producer : Topic: topic2, TC: 23, GC: 46
20:31:26.209Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 OK
20:31:26.211Z : Producer : Topic: topic1, TC: 24, GC: 47
20:31:26.214Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 OK
20:31:26.216Z : Producer : Topic: topic2, TC: 24, GC: 48
20:31:26.221Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 OK
20:31:26.221Z : Producer : Topic: topic1, TC: 25, GC: 49
20:31:26.225Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 OK
20:31:26.227Z : Producer : Topic: topic2, TC: 25, GC: 50
20:31:26.232Z : Producer : Topic: topic1, TC: 26, GC: 51
20:31:26.232Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 OK
20:31:26.236Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 OK
20:31:26.238Z : Producer : Topic: topic2, TC: 26, GC: 52
20:31:26.243Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 OK
20:31:26.243Z : Producer : Topic: topic1, TC: 27, GC: 53
20:31:26.247Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 OK
20:31:26.249Z : Producer : Topic: topic2, TC: 27, GC: 54
20:31:26.254Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 OK
20:31:26.255Z : Producer : Topic: topic1, TC: 28, GC: 55
20:31:26.257Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 OK
20:31:26.260Z : Producer : Topic: topic2, TC: 28, GC: 56
20:31:26.265Z : Producer : Topic: topic1, TC: 29, GC: 57
20:31:26.265Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 OK
20:31:26.269Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 OK
20:31:26.272Z : Producer : Topic: topic2, TC: 29, GC: 58
20:31:26.276Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 OK
20:31:26.276Z : Producer : Topic: topic1, TC: 30, GC: 59
20:31:26.280Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 OK
20:31:26.282Z : Producer : Topic: topic2, TC: 30, GC: 60
20:31:26.287Z : Producer : Topic: topic1, TC: 31, GC: 61
20:31:26.287Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 OK
20:31:26.291Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 OK
20:31:26.293Z : Producer : Topic: topic2, TC: 31, GC: 62
20:31:26.298Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 OK
20:31:26.299Z : Producer : Topic: topic1, TC: 32, GC: 63
20:31:26.302Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 OK
20:31:26.305Z : Producer : Topic: topic2, TC: 32, GC: 64
20:31:26.310Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 OK
20:31:26.314Z : Producer : Topic: topic1, TC: 33, GC: 65
20:31:26.315Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 OK
20:31:26.320Z : Producer : Topic: topic2, TC: 33, GC: 66
20:31:26.321Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 OK
20:31:26.325Z : Producer : Topic: topic1, TC: 34, GC: 67
20:31:26.326Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 OK
20:31:26.331Z : Producer : Topic: topic2, TC: 34, GC: 68
20:31:26.332Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 OK
20:31:26.336Z : Producer : Topic: topic1, TC: 35, GC: 69
20:31:26.337Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 OK
20:31:26.341Z : Producer : Topic: topic2, TC: 35, GC: 70
20:31:26.342Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 OK
20:31:26.348Z : Producer : Topic: topic1, TC: 36, GC: 71
20:31:26.348Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 OK
20:31:26.354Z : Producer : Topic: topic2, TC: 36, GC: 72
20:31:26.354Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 OK
20:31:26.359Z : Producer : Topic: topic1, TC: 37, GC: 73
20:31:26.359Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 OK
20:31:26.364Z : Producer : Topic: topic2, TC: 37, GC: 74
20:31:26.365Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 OK
20:31:26.370Z : Producer : Topic: topic1, TC: 38, GC: 75
20:31:26.370Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 OK
20:31:26.375Z : Producer : Topic: topic2, TC: 38, GC: 76
20:31:26.376Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 OK
20:31:26.399Z : Producer : Topic: topic1, TC: 39, GC: 77
20:31:26.399Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 OK
20:31:26.406Z : Producer : Topic: topic2, TC: 39, GC: 78
20:31:26.407Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 OK
20:31:26.412Z : Producer : Topic: topic1, TC: 40, GC: 79
20:31:26.412Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 OK
20:31:26.417Z : Producer : Topic: topic2, TC: 40, GC: 80
20:31:26.418Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 OK
20:31:26.422Z : Producer : Topic: topic1, TC: 41, GC: 81
20:31:26.423Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 OK
20:31:26.427Z : Producer : Topic: topic2, TC: 41, GC: 82
20:31:26.429Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 OK
20:31:26.432Z : Producer : Topic: topic1, TC: 42, GC: 83
20:31:26.434Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 OK
20:31:26.437Z : Producer : Topic: topic2, TC: 42, GC: 84
20:31:26.440Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 OK
20:31:26.443Z : Producer : Topic: topic1, TC: 43, GC: 85
20:31:26.445Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 OK
20:31:26.449Z : Producer : Topic: topic2, TC: 43, GC: 86
20:31:26.451Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 OK
20:31:26.455Z : Producer : Topic: topic1, TC: 44, GC: 87
20:31:26.455Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 OK
20:31:26.460Z : Producer : Topic: topic2, TC: 44, GC: 88
20:31:26.462Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 OK
20:31:26.465Z : Producer : Topic: topic1, TC: 45, GC: 89
20:31:26.467Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 OK
20:31:26.470Z : Producer : Topic: topic2, TC: 45, GC: 90
20:31:26.473Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 OK
20:31:26.475Z : Producer : Topic: topic1, TC: 46, GC: 91
20:31:26.478Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 OK
20:31:26.479Z : Producer : Topic: topic2, TC: 46, GC: 92
20:31:26.484Z : Producer : Topic: topic1, TC: 47, GC: 93
20:31:26.484Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 OK
20:31:26.489Z : Producer : Topic: topic2, TC: 47, GC: 94
20:31:26.489Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 OK
20:31:26.494Z : Producer : Topic: topic1, TC: 48, GC: 95
20:31:26.495Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 OK
20:31:26.499Z : Producer : Topic: topic2, TC: 48, GC: 96
20:31:26.500Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 OK
20:31:26.504Z : Producer : Topic: topic1, TC: 49, GC: 97
20:31:26.506Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 OK
20:31:26.509Z : Producer : Topic: topic2, TC: 49, GC: 98
20:31:26.511Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 OK
20:31:26.513Z : Producer : Topic: topic1, TC: 50, GC: 99
20:31:26.517Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 OK
20:31:26.519Z : Producer : Topic: topic2, TC: 50, GC: 100
20:31:26.522Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 OK
20:31:26.528Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 OK

We see OK next to the global counter on 93/100 of consumed messages, showing we don’t get perfect ordering but close, no big jumps forwards or backwards in the global sequence.

Case 2 - Tailing the topic, equal message arrival rate, short constant processing time

In Case 1, the publisher sent messages as fast as it could synchronously and the consumer consumed as fast as it could. In this run we add a 10ms interval to both the publisher and the consumer.

20:38:35.722Z : Consumer C1 : Subscribing
20:38:36.183Z : Consumer C1 : Consuming
20:38:36.463Z : Producer : Topic: topic1, TC: 1, GC: 1
20:38:36.468Z : Consumer C1 : Topic: topic1, TC: 1 OK, GC: 1 OK
20:38:36.481Z : Producer : Topic: topic2, TC: 1, GC: 2
20:38:36.483Z : Consumer C1 : Topic: topic2, TC: 1 OK, GC: 2 OK
20:38:36.498Z : Producer : Topic: topic1, TC: 2, GC: 3
20:38:36.500Z : Consumer C1 : Topic: topic1, TC: 2 OK, GC: 3 OK
20:38:36.515Z : Producer : Topic: topic2, TC: 2, GC: 4
20:38:36.517Z : Consumer C1 : Topic: topic2, TC: 2 OK, GC: 4 OK
20:38:36.531Z : Producer : Topic: topic1, TC: 3, GC: 5
20:38:36.531Z : Consumer C1 : Topic: topic1, TC: 3 OK, GC: 5 OK
20:38:36.547Z : Producer : Topic: topic2, TC: 3, GC: 6
20:38:36.548Z : Consumer C1 : Topic: topic2, TC: 3 OK, GC: 6 OK
20:38:36.563Z : Producer : Topic: topic1, TC: 4, GC: 7
20:38:36.564Z : Consumer C1 : Topic: topic1, TC: 4 OK, GC: 7 OK
20:38:36.579Z : Producer : Topic: topic2, TC: 4, GC: 8
20:38:36.580Z : Consumer C1 : Topic: topic2, TC: 4 OK, GC: 8 OK
20:38:36.595Z : Producer : Topic: topic1, TC: 5, GC: 9
20:38:36.596Z : Consumer C1 : Topic: topic1, TC: 5 OK, GC: 9 OK
20:38:36.611Z : Producer : Topic: topic2, TC: 5, GC: 10
20:38:36.612Z : Consumer C1 : Topic: topic2, TC: 5 OK, GC: 10 OK
20:38:36.627Z : Producer : Topic: topic1, TC: 6, GC: 11
20:38:36.628Z : Consumer C1 : Topic: topic1, TC: 6 OK, GC: 11 OK
20:38:36.643Z : Producer : Topic: topic2, TC: 6, GC: 12
20:38:36.643Z : Consumer C1 : Topic: topic2, TC: 6 OK, GC: 12 OK
20:38:36.659Z : Producer : Topic: topic1, TC: 7, GC: 13
20:38:36.660Z : Consumer C1 : Topic: topic1, TC: 7 OK, GC: 13 OK
20:38:36.676Z : Producer : Topic: topic2, TC: 7, GC: 14
20:38:36.676Z : Consumer C1 : Topic: topic2, TC: 7 OK, GC: 14 OK
20:38:36.692Z : Producer : Topic: topic1, TC: 8, GC: 15
20:38:36.693Z : Consumer C1 : Topic: topic1, TC: 8 OK, GC: 15 OK
20:38:36.709Z : Producer : Topic: topic2, TC: 8, GC: 16
20:38:36.709Z : Consumer C1 : Topic: topic2, TC: 8 OK, GC: 16 OK
20:38:36.725Z : Producer : Topic: topic1, TC: 9, GC: 17
20:38:36.725Z : Consumer C1 : Topic: topic1, TC: 9 OK, GC: 17 OK
20:38:36.742Z : Producer : Topic: topic2, TC: 9, GC: 18
20:38:36.743Z : Consumer C1 : Topic: topic2, TC: 9 OK, GC: 18 OK
20:38:36.757Z : Producer : Topic: topic1, TC: 10, GC: 19
20:38:36.758Z : Consumer C1 : Topic: topic1, TC: 10 OK, GC: 19 OK
20:38:36.774Z : Producer : Topic: topic2, TC: 10, GC: 20
20:38:36.775Z : Consumer C1 : Topic: topic2, TC: 10 OK, GC: 20 OK
20:38:36.790Z : Producer : Topic: topic1, TC: 11, GC: 21
20:38:36.791Z : Consumer C1 : Topic: topic1, TC: 11 OK, GC: 21 OK
20:38:36.806Z : Producer : Topic: topic2, TC: 11, GC: 22
20:38:36.807Z : Consumer C1 : Topic: topic2, TC: 11 OK, GC: 22 OK
20:38:36.822Z : Producer : Topic: topic1, TC: 12, GC: 23
20:38:36.822Z : Consumer C1 : Topic: topic1, TC: 12 OK, GC: 23 OK
20:38:36.838Z : Producer : Topic: topic2, TC: 12, GC: 24
20:38:36.839Z : Consumer C1 : Topic: topic2, TC: 12 OK, GC: 24 OK
20:38:36.853Z : Producer : Topic: topic1, TC: 13, GC: 25
20:38:36.854Z : Consumer C1 : Topic: topic1, TC: 13 OK, GC: 25 OK
20:38:36.869Z : Producer : Topic: topic2, TC: 13, GC: 26
20:38:36.869Z : Consumer C1 : Topic: topic2, TC: 13 OK, GC: 26 OK
20:38:36.885Z : Producer : Topic: topic1, TC: 14, GC: 27
20:38:36.886Z : Consumer C1 : Topic: topic1, TC: 14 OK, GC: 27 OK
20:38:36.901Z : Producer : Topic: topic2, TC: 14, GC: 28
20:38:36.902Z : Consumer C1 : Topic: topic2, TC: 14 OK, GC: 28 OK
20:38:36.917Z : Producer : Topic: topic1, TC: 15, GC: 29
20:38:36.918Z : Consumer C1 : Topic: topic1, TC: 15 OK, GC: 29 OK
20:38:36.933Z : Producer : Topic: topic2, TC: 15, GC: 30
20:38:36.933Z : Consumer C1 : Topic: topic2, TC: 15 OK, GC: 30 OK
20:38:36.949Z : Producer : Topic: topic1, TC: 16, GC: 31
20:38:36.950Z : Consumer C1 : Topic: topic1, TC: 16 OK, GC: 31 OK
20:38:36.965Z : Producer : Topic: topic2, TC: 16, GC: 32
20:38:36.966Z : Consumer C1 : Topic: topic2, TC: 16 OK, GC: 32 OK
20:38:36.981Z : Producer : Topic: topic1, TC: 17, GC: 33
20:38:36.982Z : Consumer C1 : Topic: topic1, TC: 17 OK, GC: 33 OK
20:38:36.997Z : Producer : Topic: topic2, TC: 17, GC: 34
20:38:36.998Z : Consumer C1 : Topic: topic2, TC: 17 OK, GC: 34 OK
20:38:37.013Z : Producer : Topic: topic1, TC: 18, GC: 35
20:38:37.014Z : Consumer C1 : Topic: topic1, TC: 18 OK, GC: 35 OK
20:38:37.029Z : Producer : Topic: topic2, TC: 18, GC: 36
20:38:37.030Z : Consumer C1 : Topic: topic2, TC: 18 OK, GC: 36 OK
20:38:37.045Z : Producer : Topic: topic1, TC: 19, GC: 37
20:38:37.046Z : Consumer C1 : Topic: topic1, TC: 19 OK, GC: 37 OK
20:38:37.061Z : Producer : Topic: topic2, TC: 19, GC: 38
20:38:37.062Z : Consumer C1 : Topic: topic2, TC: 19 OK, GC: 38 OK
20:38:37.077Z : Producer : Topic: topic1, TC: 20, GC: 39
20:38:37.078Z : Consumer C1 : Topic: topic1, TC: 20 OK, GC: 39 OK
20:38:37.093Z : Producer : Topic: topic2, TC: 20, GC: 40
20:38:37.094Z : Consumer C1 : Topic: topic2, TC: 20 OK, GC: 40 OK
20:38:37.109Z : Producer : Topic: topic1, TC: 21, GC: 41
20:38:37.110Z : Consumer C1 : Topic: topic1, TC: 21 OK, GC: 41 OK
20:38:37.126Z : Producer : Topic: topic2, TC: 21, GC: 42
20:38:37.127Z : Consumer C1 : Topic: topic2, TC: 21 OK, GC: 42 OK
20:38:37.142Z : Producer : Topic: topic1, TC: 22, GC: 43
20:38:37.143Z : Consumer C1 : Topic: topic1, TC: 22 OK, GC: 43 OK
20:38:37.158Z : Producer : Topic: topic2, TC: 22, GC: 44
20:38:37.159Z : Consumer C1 : Topic: topic2, TC: 22 OK, GC: 44 OK
20:38:37.174Z : Producer : Topic: topic1, TC: 23, GC: 45
20:38:37.175Z : Consumer C1 : Topic: topic1, TC: 23 OK, GC: 45 OK
20:38:37.190Z : Producer : Topic: topic2, TC: 23, GC: 46
20:38:37.191Z : Consumer C1 : Topic: topic2, TC: 23 OK, GC: 46 OK
20:38:37.206Z : Producer : Topic: topic1, TC: 24, GC: 47
20:38:37.207Z : Consumer C1 : Topic: topic1, TC: 24 OK, GC: 47 OK
20:38:37.222Z : Producer : Topic: topic2, TC: 24, GC: 48
20:38:37.223Z : Consumer C1 : Topic: topic2, TC: 24 OK, GC: 48 OK
20:38:37.238Z : Producer : Topic: topic1, TC: 25, GC: 49
20:38:37.239Z : Consumer C1 : Topic: topic1, TC: 25 OK, GC: 49 OK
20:38:37.254Z : Producer : Topic: topic2, TC: 25, GC: 50
20:38:37.255Z : Consumer C1 : Topic: topic2, TC: 25 OK, GC: 50 OK
20:38:37.270Z : Producer : Topic: topic1, TC: 26, GC: 51
20:38:37.271Z : Consumer C1 : Topic: topic1, TC: 26 OK, GC: 51 OK
20:38:37.286Z : Producer : Topic: topic2, TC: 26, GC: 52
20:38:37.287Z : Consumer C1 : Topic: topic2, TC: 26 OK, GC: 52 OK
20:38:37.301Z : Producer : Topic: topic1, TC: 27, GC: 53
20:38:37.302Z : Consumer C1 : Topic: topic1, TC: 27 OK, GC: 53 OK
20:38:37.318Z : Producer : Topic: topic2, TC: 27, GC: 54
20:38:37.319Z : Consumer C1 : Topic: topic2, TC: 27 OK, GC: 54 OK
20:38:37.334Z : Producer : Topic: topic1, TC: 28, GC: 55
20:38:37.335Z : Consumer C1 : Topic: topic1, TC: 28 OK, GC: 55 OK
20:38:37.350Z : Producer : Topic: topic2, TC: 28, GC: 56
20:38:37.350Z : Consumer C1 : Topic: topic2, TC: 28 OK, GC: 56 OK
20:38:37.366Z : Producer : Topic: topic1, TC: 29, GC: 57
20:38:37.366Z : Consumer C1 : Topic: topic1, TC: 29 OK, GC: 57 OK
20:38:37.381Z : Producer : Topic: topic2, TC: 29, GC: 58
20:38:37.382Z : Consumer C1 : Topic: topic2, TC: 29 OK, GC: 58 OK
20:38:37.398Z : Producer : Topic: topic1, TC: 30, GC: 59
20:38:37.398Z : Consumer C1 : Topic: topic1, TC: 30 OK, GC: 59 OK
20:38:37.414Z : Producer : Topic: topic2, TC: 30, GC: 60
20:38:37.414Z : Consumer C1 : Topic: topic2, TC: 30 OK, GC: 60 OK
20:38:37.430Z : Producer : Topic: topic1, TC: 31, GC: 61
20:38:37.431Z : Consumer C1 : Topic: topic1, TC: 31 OK, GC: 61 OK
20:38:37.447Z : Producer : Topic: topic2, TC: 31, GC: 62
20:38:37.447Z : Consumer C1 : Topic: topic2, TC: 31 OK, GC: 62 OK
20:38:37.462Z : Producer : Topic: topic1, TC: 32, GC: 63
20:38:37.463Z : Consumer C1 : Topic: topic1, TC: 32 OK, GC: 63 OK
20:38:37.478Z : Producer : Topic: topic2, TC: 32, GC: 64
20:38:37.480Z : Consumer C1 : Topic: topic2, TC: 32 OK, GC: 64 OK
20:38:37.493Z : Producer : Topic: topic1, TC: 33, GC: 65
20:38:37.494Z : Consumer C1 : Topic: topic1, TC: 33 OK, GC: 65 OK
20:38:37.510Z : Producer : Topic: topic2, TC: 33, GC: 66
20:38:37.511Z : Consumer C1 : Topic: topic2, TC: 33 OK, GC: 66 OK
20:38:37.526Z : Producer : Topic: topic1, TC: 34, GC: 67
20:38:37.527Z : Consumer C1 : Topic: topic1, TC: 34 OK, GC: 67 OK
20:38:37.541Z : Producer : Topic: topic2, TC: 34, GC: 68
20:38:37.542Z : Consumer C1 : Topic: topic2, TC: 34 OK, GC: 68 OK
20:38:37.556Z : Producer : Topic: topic1, TC: 35, GC: 69
20:38:37.557Z : Consumer C1 : Topic: topic1, TC: 35 OK, GC: 69 OK
20:38:37.571Z : Producer : Topic: topic2, TC: 35, GC: 70
20:38:37.572Z : Consumer C1 : Topic: topic2, TC: 35 OK, GC: 70 OK
20:38:37.586Z : Producer : Topic: topic1, TC: 36, GC: 71
20:38:37.587Z : Consumer C1 : Topic: topic1, TC: 36 OK, GC: 71 OK
20:38:37.603Z : Producer : Topic: topic2, TC: 36, GC: 72
20:38:37.603Z : Consumer C1 : Topic: topic2, TC: 36 OK, GC: 72 OK
20:38:37.619Z : Producer : Topic: topic1, TC: 37, GC: 73
20:38:37.619Z : Consumer C1 : Topic: topic1, TC: 37 OK, GC: 73 OK
20:38:37.634Z : Producer : Topic: topic2, TC: 37, GC: 74
20:38:37.635Z : Consumer C1 : Topic: topic2, TC: 37 OK, GC: 74 OK
20:38:37.650Z : Producer : Topic: topic1, TC: 38, GC: 75
20:38:37.651Z : Consumer C1 : Topic: topic1, TC: 38 OK, GC: 75 OK
20:38:37.666Z : Producer : Topic: topic2, TC: 38, GC: 76
20:38:37.666Z : Consumer C1 : Topic: topic2, TC: 38 OK, GC: 76 OK
20:38:37.682Z : Producer : Topic: topic1, TC: 39, GC: 77
20:38:37.682Z : Consumer C1 : Topic: topic1, TC: 39 OK, GC: 77 OK
20:38:37.698Z : Producer : Topic: topic2, TC: 39, GC: 78
20:38:37.698Z : Consumer C1 : Topic: topic2, TC: 39 OK, GC: 78 OK
20:38:37.714Z : Producer : Topic: topic1, TC: 40, GC: 79
20:38:37.715Z : Consumer C1 : Topic: topic1, TC: 40 OK, GC: 79 OK
20:38:37.730Z : Producer : Topic: topic2, TC: 40, GC: 80
20:38:37.730Z : Consumer C1 : Topic: topic2, TC: 40 OK, GC: 80 OK
20:38:37.746Z : Producer : Topic: topic1, TC: 41, GC: 81
20:38:37.747Z : Consumer C1 : Topic: topic1, TC: 41 OK, GC: 81 OK
20:38:37.762Z : Producer : Topic: topic2, TC: 41, GC: 82
20:38:37.763Z : Consumer C1 : Topic: topic2, TC: 41 OK, GC: 82 OK
20:38:37.778Z : Producer : Topic: topic1, TC: 42, GC: 83
20:38:37.778Z : Consumer C1 : Topic: topic1, TC: 42 OK, GC: 83 OK
20:38:37.794Z : Producer : Topic: topic2, TC: 42, GC: 84
20:38:37.795Z : Consumer C1 : Topic: topic2, TC: 42 OK, GC: 84 OK
20:38:37.810Z : Producer : Topic: topic1, TC: 43, GC: 85
20:38:37.811Z : Consumer C1 : Topic: topic1, TC: 43 OK, GC: 85 OK
20:38:37.826Z : Producer : Topic: topic2, TC: 43, GC: 86
20:38:37.827Z : Consumer C1 : Topic: topic2, TC: 43 OK, GC: 86 OK
20:38:37.842Z : Producer : Topic: topic1, TC: 44, GC: 87
20:38:37.842Z : Consumer C1 : Topic: topic1, TC: 44 OK, GC: 87 OK
20:38:37.859Z : Producer : Topic: topic2, TC: 44, GC: 88
20:38:37.860Z : Consumer C1 : Topic: topic2, TC: 44 OK, GC: 88 OK
20:38:37.875Z : Producer : Topic: topic1, TC: 45, GC: 89
20:38:37.876Z : Consumer C1 : Topic: topic1, TC: 45 OK, GC: 89 OK
20:38:37.891Z : Producer : Topic: topic2, TC: 45, GC: 90
20:38:37.891Z : Consumer C1 : Topic: topic2, TC: 45 OK, GC: 90 OK
20:38:37.906Z : Producer : Topic: topic1, TC: 46, GC: 91
20:38:37.907Z : Consumer C1 : Topic: topic1, TC: 46 OK, GC: 91 OK
20:38:37.922Z : Producer : Topic: topic2, TC: 46, GC: 92
20:38:37.923Z : Consumer C1 : Topic: topic2, TC: 46 OK, GC: 92 OK
20:38:37.937Z : Producer : Topic: topic1, TC: 47, GC: 93
20:38:37.938Z : Consumer C1 : Topic: topic1, TC: 47 OK, GC: 93 OK
20:38:37.953Z : Producer : Topic: topic2, TC: 47, GC: 94
20:38:37.953Z : Consumer C1 : Topic: topic2, TC: 47 OK, GC: 94 OK
20:38:37.969Z : Producer : Topic: topic1, TC: 48, GC: 95
20:38:37.970Z : Consumer C1 : Topic: topic1, TC: 48 OK, GC: 95 OK
20:38:37.985Z : Producer : Topic: topic2, TC: 48, GC: 96
20:38:37.985Z : Consumer C1 : Topic: topic2, TC: 48 OK, GC: 96 OK
20:38:38Z : Producer : Topic: topic1, TC: 49, GC: 97
20:38:38.001Z : Consumer C1 : Topic: topic1, TC: 49 OK, GC: 97 OK
20:38:38.016Z : Producer : Topic: topic2, TC: 49, GC: 98
20:38:38.016Z : Consumer C1 : Topic: topic2, TC: 49 OK, GC: 98 OK
20:38:38.032Z : Producer : Topic: topic1, TC: 50, GC: 99
20:38:38.032Z : Consumer C1 : Topic: topic1, TC: 50 OK, GC: 99 OK
20:38:38.047Z : Producer : Topic: topic2, TC: 50, GC: 100
20:38:38.047Z : Consumer C1 : Topic: topic2, TC: 50 OK, GC: 100 OK

This shows that the Pulsar client is at least capable of processing messages in perfect ordering across topics - though if that extends to more real-world scenarios we’ll see. We see perfect message interleaving, i.e publish, consume, publish, consume = strict real-time consumption.

Case 3 - Tailing the topic, equal message arrival rate, short processing time with jitter

In the last test we saw that when we added some processing time we actually got perfect ordering.

If we add a bit of random jitter to consumer processing times then what do we see? It should eliminate that nice interleaving of publish-consume that we saw in Case 2.

With the publish interval at 25ms and consumption rate between 1ms and 50ms with peaks of 200ms, our Kafka consumer saw some loss of ordering. Kafka went from perfect ordering to quite a lot of small ordering problems, but with our Pulsar client we see:

20:33:12.808Z : Consumer : Subscribing
20:33:13.264Z : Consumer : Consuming
20:33:13.538Z : Producer : Topic: topic1, TC: 1, GC: 1
20:33:13.541Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
20:33:13.570Z : Producer : Topic: topic2, TC: 1, GC: 2
20:33:13.592Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 OK
20:33:13.602Z : Producer : Topic: topic1, TC: 2, GC: 3
20:33:13.632Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 OK
20:33:13.632Z : Producer : Topic: topic2, TC: 2, GC: 4
20:33:13.637Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 OK
20:33:13.664Z : Producer : Topic: topic1, TC: 3, GC: 5
20:33:13.678Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 OK
20:33:13.696Z : Producer : Topic: topic2, TC: 3, GC: 6
20:33:13.698Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 OK
20:33:13.727Z : Producer : Topic: topic1, TC: 4, GC: 7
20:33:13.728Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 OK
20:33:13.758Z : Producer : Topic: topic2, TC: 4, GC: 8
20:33:13.775Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 OK
20:33:13.790Z : Producer : Topic: topic1, TC: 5, GC: 9
20:33:13.797Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 OK
20:33:13.822Z : Producer : Topic: topic2, TC: 5, GC: 10
20:33:13.823Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 OK
20:33:13.853Z : Producer : Topic: topic1, TC: 6, GC: 11
20:33:13.854Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 OK
20:33:13.884Z : Producer : Topic: topic2, TC: 6, GC: 12
20:33:13.885Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 OK
20:33:13.916Z : Producer : Topic: topic1, TC: 7, GC: 13
20:33:13.931Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 OK
20:33:13.947Z : Producer : Topic: topic2, TC: 7, GC: 14
20:33:13.959Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 OK
20:33:13.979Z : Producer : Topic: topic1, TC: 8, GC: 15
20:33:14.009Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 OK
20:33:14.012Z : Producer : Topic: topic2, TC: 8, GC: 16
20:33:14.042Z : Producer : Topic: topic1, TC: 9, GC: 17
20:33:14.042Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 OK
20:33:14.045Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 OK
20:33:14.074Z : Producer : Topic: topic2, TC: 9, GC: 18
20:33:14.091Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 OK
20:33:14.106Z : Producer : Topic: topic1, TC: 10, GC: 19
20:33:14.120Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 OK
20:33:14.137Z : Producer : Topic: topic2, TC: 10, GC: 20
20:33:14.138Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 OK
20:33:14.169Z : Producer : Topic: topic1, TC: 11, GC: 21
20:33:14.181Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 OK
20:33:14.201Z : Producer : Topic: topic2, TC: 11, GC: 22
20:33:14.201Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 OK
20:33:14.231Z : Producer : Topic: topic1, TC: 12, GC: 23
20:33:14.235Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 OK
20:33:14.263Z : Producer : Topic: topic2, TC: 12, GC: 24
20:33:14.264Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 OK
20:33:14.294Z : Producer : Topic: topic1, TC: 13, GC: 25
20:33:14.300Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 OK
20:33:14.325Z : Producer : Topic: topic2, TC: 13, GC: 26
20:33:14.347Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 OK
20:33:14.357Z : Producer : Topic: topic1, TC: 14, GC: 27
20:33:14.380Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 OK
20:33:14.388Z : Producer : Topic: topic2, TC: 14, GC: 28
20:33:14.419Z : Producer : Topic: topic1, TC: 15, GC: 29
20:33:14.422Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 OK
20:33:14.447Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 OK
20:33:14.449Z : Producer : Topic: topic2, TC: 15, GC: 30
20:33:14.481Z : Producer : Topic: topic1, TC: 16, GC: 31
20:33:14.485Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 OK
20:33:14.495Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 OK
20:33:14.513Z : Producer : Topic: topic2, TC: 16, GC: 32
20:33:14.542Z : Producer : Topic: topic1, TC: 17, GC: 33
20:33:14.575Z : Producer : Topic: topic2, TC: 17, GC: 34
20:33:14.606Z : Producer : Topic: topic1, TC: 18, GC: 35
20:33:14.637Z : Producer : Topic: topic2, TC: 18, GC: 36
20:33:14.668Z : Producer : Topic: topic1, TC: 19, GC: 37
20:33:14.698Z : Producer : Topic: topic2, TC: 19, GC: 38
20:33:14.729Z : Producer : Topic: topic1, TC: 20, GC: 39
20:33:14.741Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 OK
20:33:14.745Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 OK
20:33:14.760Z : Producer : Topic: topic2, TC: 20, GC: 40
20:33:14.784Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 OK
20:33:14.791Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 OK
20:33:14.792Z : Producer : Topic: topic1, TC: 21, GC: 41
20:33:14.797Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 OK
20:33:14.817Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 OK
20:33:14.823Z : Producer : Topic: topic2, TC: 21, GC: 42
20:33:14.839Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 OK
20:33:14.855Z : Producer : Topic: topic1, TC: 22, GC: 43
20:33:14.867Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 OK
20:33:14.886Z : Producer : Topic: topic2, TC: 22, GC: 44
20:33:14.894Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 OK
20:33:14.917Z : Producer : Topic: topic1, TC: 23, GC: 45
20:33:14.925Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 OK
20:33:14.930Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 OK
20:33:14.946Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 OK
20:33:14.948Z : Producer : Topic: topic2, TC: 23, GC: 46
20:33:14.975Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 OK
20:33:14.979Z : Producer : Topic: topic1, TC: 24, GC: 47
20:33:14.983Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 OK
20:33:14.993Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 OK
20:33:15.011Z : Producer : Topic: topic2, TC: 24, GC: 48
20:33:15.017Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 OK
20:33:15.042Z : Producer : Topic: topic1, TC: 25, GC: 49
20:33:15.044Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 OK
20:33:15.070Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 OK
20:33:15.073Z : Producer : Topic: topic2, TC: 25, GC: 50
20:33:15.095Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 OK
20:33:15.104Z : Producer : Topic: topic1, TC: 26, GC: 51
20:33:15.135Z : Producer : Topic: topic2, TC: 26, GC: 52
20:33:15.167Z : Producer : Topic: topic1, TC: 27, GC: 53
20:33:15.199Z : Producer : Topic: topic2, TC: 27, GC: 54
20:33:15.230Z : Producer : Topic: topic1, TC: 28, GC: 55
20:33:15.262Z : Producer : Topic: topic2, TC: 28, GC: 56
20:33:15.293Z : Producer : Topic: topic1, TC: 29, GC: 57
20:33:15.324Z : Producer : Topic: topic2, TC: 29, GC: 58
20:33:15.338Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 OK
20:33:15.355Z : Producer : Topic: topic1, TC: 30, GC: 59
20:33:15.365Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 OK
20:33:15.386Z : Producer : Topic: topic2, TC: 30, GC: 60
20:33:15.414Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 OK
20:33:15.419Z : Producer : Topic: topic1, TC: 31, GC: 61
20:33:15.449Z : Producer : Topic: topic2, TC: 31, GC: 62
20:33:15.456Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 OK
20:33:15.480Z : Producer : Topic: topic1, TC: 32, GC: 63
20:33:15.503Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 OK
20:33:15.511Z : Producer : Topic: topic2, TC: 32, GC: 64
20:33:15.528Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 OK
20:33:15.539Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 OK
20:33:15.542Z : Producer : Topic: topic1, TC: 33, GC: 65
20:33:15.574Z : Producer : Topic: topic2, TC: 33, GC: 66
20:33:15.582Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 OK
20:33:15.605Z : Producer : Topic: topic1, TC: 34, GC: 67
20:33:15.627Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 OK
20:33:15.636Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 OK
20:33:15.636Z : Producer : Topic: topic2, TC: 34, GC: 68
20:33:15.642Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 OK
20:33:15.649Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 OK
20:33:15.668Z : Producer : Topic: topic1, TC: 35, GC: 69
20:33:15.691Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 OK
20:33:15.699Z : Producer : Topic: topic2, TC: 35, GC: 70
20:33:15.700Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 OK
20:33:15.729Z : Producer : Topic: topic1, TC: 36, GC: 71
20:33:15.746Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 OK
20:33:15.760Z : Producer : Topic: topic2, TC: 36, GC: 72
20:33:15.782Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 OK
20:33:15.792Z : Producer : Topic: topic1, TC: 37, GC: 73
20:33:15.794Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 OK
20:33:15.823Z : Producer : Topic: topic2, TC: 37, GC: 74
20:33:15.828Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 OK
20:33:15.854Z : Producer : Topic: topic1, TC: 38, GC: 75
20:33:15.861Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 OK
20:33:15.870Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 OK
20:33:15.885Z : Producer : Topic: topic2, TC: 38, GC: 76
20:33:15.889Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 OK
20:33:15.895Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 OK
20:33:15.916Z : Producer : Topic: topic1, TC: 39, GC: 77
20:33:15.936Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 OK
20:33:15.947Z : Producer : Topic: topic2, TC: 39, GC: 78
20:33:15.971Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 OK
20:33:15.978Z : Producer : Topic: topic1, TC: 40, GC: 79
20:33:16.010Z : Producer : Topic: topic2, TC: 40, GC: 80
20:33:16.010Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 OK
20:33:16.024Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 OK
20:33:16.041Z : Producer : Topic: topic1, TC: 41, GC: 81
20:33:16.058Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 OK
20:33:16.071Z : Producer : Topic: topic2, TC: 41, GC: 82
20:33:16.085Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 OK
20:33:16.103Z : Producer : Topic: topic1, TC: 42, GC: 83
20:33:16.133Z : Producer : Topic: topic2, TC: 42, GC: 84
20:33:16.166Z : Producer : Topic: topic1, TC: 43, GC: 85
20:33:16.197Z : Producer : Topic: topic2, TC: 43, GC: 86
20:33:16.229Z : Producer : Topic: topic1, TC: 44, GC: 87
20:33:16.259Z : Producer : Topic: topic2, TC: 44, GC: 88
20:33:16.290Z : Producer : Topic: topic1, TC: 45, GC: 89
20:33:16.307Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 OK
20:33:16.321Z : Producer : Topic: topic2, TC: 45, GC: 90
20:33:16.352Z : Producer : Topic: topic1, TC: 46, GC: 91
20:33:16.354Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 OK
20:33:16.378Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 OK
20:33:16.383Z : Producer : Topic: topic2, TC: 46, GC: 92
20:33:16.412Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 OK
20:33:16.413Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 OK
20:33:16.414Z : Producer : Topic: topic1, TC: 47, GC: 93
20:33:16.444Z : Producer : Topic: topic2, TC: 47, GC: 94
20:33:16.453Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 OK
20:33:16.476Z : Producer : Topic: topic1, TC: 48, GC: 95
20:33:16.506Z : Producer : Topic: topic2, TC: 48, GC: 96
20:33:16.538Z : Producer : Topic: topic1, TC: 49, GC: 97
20:33:16.570Z : Producer : Topic: topic2, TC: 49, GC: 98
20:33:16.601Z : Producer : Topic: topic1, TC: 50, GC: 99
20:33:16.632Z : Producer : Topic: topic2, TC: 50, GC: 100
20:33:16.656Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 OK
20:33:16.678Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 OK
20:33:16.718Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 OK
20:33:16.733Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 OK
20:33:16.768Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 OK
20:33:16.783Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 OK
20:33:16.802Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 OK
20:33:16.834Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 OK
20:33:16.852Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 OK
20:33:16.888Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 OK
20:33:16.898Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 OK
20:33:17.118Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 OK
20:33:17.155Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 OK
20:33:17.164Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 OK
20:33:17.168Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 OK
20:33:17.176Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 OK

Gone is the nice publish-consume interleaving but we still get perfect message ordering. But don’t forget the internal buffer with default size of 1000 messages. The Pulsar client was probably really tailing the producer perfectly, just the application code didn’t get to see it that way.

Case 4 - Tailing the topic, equal message arrival rate, fast producer

We can also have periods of time where the message arrival rate at a topic is higher than a consumer can cope with. How does that affect our ordering?

When we have a period that covers 1000 messages where the produce rate to both topics far outweighs the consumer processing rate, what do we see? In order to prevent that the Pulsar client from keeping up behind the scenes by leveraging its internal buffer, I set the maxReceiverQueue to 10. I did the same for the equivalent test with Kafka, setting the max.poll.records to 10 also. Kafka saw significant loss of ordering but with Pulsar we see:

20:34:12.056Z : Consumer : Subscribing
20:34:12.523Z : Consumer : Consuming
20:34:17.692Z : Producer : Topic: topic1, TC: 1, GC: 1
20:34:17.695Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
20:34:17.699Z : Producer : Topic: topic2, TC: 1, GC: 2
20:34:17.707Z : Producer : Topic: topic1, TC: 2, GC: 3
20:34:17.714Z : Producer : Topic: topic2, TC: 2, GC: 4
20:34:17.722Z : Producer : Topic: topic1, TC: 3, GC: 5
20:34:17.728Z : Producer : Topic: topic2, TC: 3, GC: 6
20:34:17.735Z : Producer : Topic: topic1, TC: 4, GC: 7
20:34:17.742Z : Producer : Topic: topic2, TC: 4, GC: 8
20:34:17.750Z : Producer : Topic: topic1, TC: 5, GC: 9
20:34:17.757Z : Producer : Topic: topic2, TC: 5, GC: 10
20:34:17.763Z : Producer : Topic: topic1, TC: 6, GC: 11
20:34:17.770Z : Producer : Topic: topic2, TC: 6, GC: 12
20:34:17.779Z : Producer : Topic: topic1, TC: 7, GC: 13
20:34:17.785Z : Producer : Topic: topic2, TC: 7, GC: 14
20:34:17.792Z : Producer : Topic: topic1, TC: 8, GC: 15
20:34:17.797Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 OK
20:34:17.798Z : Producer : Topic: topic2, TC: 8, GC: 16
20:34:17.806Z : Producer : Topic: topic1, TC: 9, GC: 17
20:34:17.813Z : Producer : Topic: topic2, TC: 9, GC: 18
20:34:17.819Z : Producer : Topic: topic1, TC: 10, GC: 19
20:34:17.827Z : Producer : Topic: topic2, TC: 10, GC: 20
20:34:17.833Z : Producer : Topic: topic1, TC: 11, GC: 21
20:34:17.841Z : Producer : Topic: topic2, TC: 11, GC: 22
20:34:17.847Z : Producer : Topic: topic1, TC: 12, GC: 23
20:34:17.855Z : Producer : Topic: topic2, TC: 12, GC: 24
20:34:17.862Z : Producer : Topic: topic1, TC: 13, GC: 25
20:34:17.868Z : Producer : Topic: topic2, TC: 13, GC: 26
20:34:17.876Z : Producer : Topic: topic1, TC: 14, GC: 27
20:34:17.882Z : Producer : Topic: topic2, TC: 14, GC: 28
20:34:17.890Z : Producer : Topic: topic1, TC: 15, GC: 29
20:34:17.896Z : Producer : Topic: topic2, TC: 15, GC: 30
20:34:17.898Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 OK
20:34:17.904Z : Producer : Topic: topic1, TC: 16, GC: 31
20:34:17.911Z : Producer : Topic: topic2, TC: 16, GC: 32
20:34:17.917Z : Producer : Topic: topic1, TC: 17, GC: 33
20:34:17.924Z : Producer : Topic: topic2, TC: 17, GC: 34
20:34:17.932Z : Producer : Topic: topic1, TC: 18, GC: 35
20:34:17.938Z : Producer : Topic: topic2, TC: 18, GC: 36
20:34:17.946Z : Producer : Topic: topic1, TC: 19, GC: 37
20:34:17.953Z : Producer : Topic: topic2, TC: 19, GC: 38
20:34:17.960Z : Producer : Topic: topic1, TC: 20, GC: 39
20:34:17.967Z : Producer : Topic: topic2, TC: 20, GC: 40
... [omitted 950 messages for brevity]
20:34:24.118Z : Producer : Topic: topic2, TC: 495, GC: 990
20:34:24.124Z : Producer : Topic: topic1, TC: 496, GC: 991
20:34:24.131Z : Producer : Topic: topic2, TC: 496, GC: 992
20:34:24.137Z : Producer : Topic: topic1, TC: 497, GC: 993
20:34:24.143Z : Producer : Topic: topic2, TC: 497, GC: 994
20:34:24.148Z : Producer : Topic: topic1, TC: 498, GC: 995
20:34:24.150Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 OK
20:34:24.155Z : Producer : Topic: topic2, TC: 498, GC: 996
20:34:24.162Z : Producer : Topic: topic1, TC: 499, GC: 997
20:34:24.168Z : Producer : Topic: topic2, TC: 499, GC: 998
20:34:24.174Z : Producer : Topic: topic1, TC: 500, GC: 999
20:34:24.181Z : Producer : Topic: topic2, TC: 500, GC: 1,000
20:34:24.251Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 OK
20:34:24.351Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 OK
20:34:24.452Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 OK
20:34:24.553Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 OK
20:34:24.654Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 OK
20:34:24.754Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 OK
20:34:24.856Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 OK
20:34:24.957Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 OK
20:34:25.058Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 OK
20:34:25.159Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 OK
20:34:25.260Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 OK
20:34:25.361Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 OK
20:34:25.462Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 OK
20:34:25.563Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 OK
20:34:25.664Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 OK
20:34:25.765Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 OK
20:34:25.866Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 OK
20:34:25.967Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 OK
20:34:26.068Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 OK
20:34:26.169Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 OK
20:34:26.270Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 OK
20:34:26.371Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 OK
20:34:26.472Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 OK
20:34:26.573Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 OK
20:34:26.674Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 OK
20:34:26.775Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 OK
20:34:26.876Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 OK
20:34:26.978Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 OK
20:34:27.079Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 OK
20:34:27.180Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 OK
20:34:27.281Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 OK
20:34:27.382Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 OK
20:34:27.483Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 OK
20:34:27.584Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 OK
20:34:27.685Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 OK
... [Omitted 850 messages for brevity, trust, they're all in order]
20:35:53.468Z : Consumer : Topic: topic2, TC: 475 OK, GC: 950 OK
20:35:53.570Z : Consumer : Topic: topic1, TC: 476 OK, GC: 951 OK
20:35:53.671Z : Consumer : Topic: topic2, TC: 476 OK, GC: 952 OK
20:35:53.772Z : Consumer : Topic: topic1, TC: 477 OK, GC: 953 OK
20:35:53.872Z : Consumer : Topic: topic2, TC: 477 OK, GC: 954 OK
20:35:53.973Z : Consumer : Topic: topic1, TC: 478 OK, GC: 955 OK
20:35:54.074Z : Consumer : Topic: topic2, TC: 478 OK, GC: 956 OK
20:35:54.176Z : Consumer : Topic: topic1, TC: 479 OK, GC: 957 OK
20:35:54.276Z : Consumer : Topic: topic2, TC: 479 OK, GC: 958 OK
20:35:54.378Z : Consumer : Topic: topic1, TC: 480 OK, GC: 959 OK
20:35:54.479Z : Consumer : Topic: topic2, TC: 480 OK, GC: 960 OK
20:35:54.580Z : Consumer : Topic: topic1, TC: 481 OK, GC: 961 OK
20:35:54.681Z : Consumer : Topic: topic2, TC: 481 OK, GC: 962 OK
20:35:54.782Z : Consumer : Topic: topic1, TC: 482 OK, GC: 963 OK
20:35:54.883Z : Consumer : Topic: topic2, TC: 482 OK, GC: 964 OK
20:35:54.984Z : Consumer : Topic: topic1, TC: 483 OK, GC: 965 OK
20:35:55.085Z : Consumer : Topic: topic2, TC: 483 OK, GC: 966 OK
20:35:55.186Z : Consumer : Topic: topic1, TC: 484 OK, GC: 967 OK
20:35:55.286Z : Consumer : Topic: topic2, TC: 484 OK, GC: 968 OK
20:35:55.387Z : Consumer : Topic: topic1, TC: 485 OK, GC: 969 OK
20:35:55.488Z : Consumer : Topic: topic2, TC: 485 OK, GC: 970 OK
20:35:55.590Z : Consumer : Topic: topic1, TC: 486 OK, GC: 971 OK
20:35:55.690Z : Consumer : Topic: topic2, TC: 486 OK, GC: 972 OK
20:35:55.791Z : Consumer : Topic: topic1, TC: 487 OK, GC: 973 OK
20:35:55.892Z : Consumer : Topic: topic2, TC: 487 OK, GC: 974 OK
20:35:55.993Z : Consumer : Topic: topic1, TC: 488 OK, GC: 975 OK
20:35:56.094Z : Consumer : Topic: topic2, TC: 488 OK, GC: 976 OK
20:35:56.195Z : Consumer : Topic: topic1, TC: 489 OK, GC: 977 OK
20:35:56.296Z : Consumer : Topic: topic2, TC: 489 OK, GC: 978 OK
20:35:56.397Z : Consumer : Topic: topic1, TC: 490 OK, GC: 979 OK
20:35:56.498Z : Consumer : Topic: topic2, TC: 490 OK, GC: 980 OK
20:35:56.599Z : Consumer : Topic: topic1, TC: 491 OK, GC: 981 OK
20:35:56.699Z : Consumer : Topic: topic2, TC: 491 OK, GC: 982 OK
20:35:56.800Z : Consumer : Topic: topic1, TC: 492 OK, GC: 983 OK
20:35:56.901Z : Consumer : Topic: topic2, TC: 492 OK, GC: 984 OK
20:35:57.002Z : Consumer : Topic: topic1, TC: 493 OK, GC: 985 OK
20:35:57.102Z : Consumer : Topic: topic2, TC: 493 OK, GC: 986 OK
20:35:57.203Z : Consumer : Topic: topic1, TC: 494 OK, GC: 987 OK
20:35:57.304Z : Consumer : Topic: topic2, TC: 494 OK, GC: 988 OK
20:35:57.405Z : Consumer : Topic: topic1, TC: 495 OK, GC: 989 OK
20:35:57.506Z : Consumer : Topic: topic2, TC: 495 OK, GC: 990 OK
20:35:57.607Z : Consumer : Topic: topic1, TC: 496 OK, GC: 991 OK
20:35:57.708Z : Consumer : Topic: topic2, TC: 496 OK, GC: 992 OK
20:35:57.809Z : Consumer : Topic: topic1, TC: 497 OK, GC: 993 OK
20:35:57.910Z : Consumer : Topic: topic2, TC: 497 OK, GC: 994 OK
20:35:58.011Z : Consumer : Topic: topic1, TC: 498 OK, GC: 995 OK
20:35:58.112Z : Consumer : Topic: topic2, TC: 498 OK, GC: 996 OK
20:35:58.213Z : Consumer : Topic: topic1, TC: 499 OK, GC: 997 OK
20:35:58.313Z : Consumer : Topic: topic2, TC: 499 OK, GC: 998 OK
20:35:58.414Z : Consumer : Topic: topic1, TC: 500 OK, GC: 999 OK
20:35:58.514Z : Consumer : Topic: topic2, TC: 500 OK, GC: 1,000 OK

We get perfect ordering again. This is a surprising result as I had expected that reducing the size of the buffer would prevent the client from delivering the messages in order.

So far we have the very unlikely scenario of the same message rates on both topics, round-robin style. Perhaps that explains this result. The result is definitely interesting, but likely not so relevant as in the real-word messages won’t necessarily arrive to both topics at the same rate according to a global sequence.

Case 5 - Tailing the topic,unequal message arrival rate, zero processing time

So far we have seen tailing consumers, consuming from two topics with the same message arrival rates. What happens when one topic gets ten times the arrival rate of the other? The below result shows zero processing time, like case 1.

21:02:15.193Z : Consumer : Subscribing
21:02:15.636Z : Consumer : Consuming
21:02:15.910Z : Producer : Topic: topic1, TC: 1, GC: 1
21:02:15.914Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
21:02:15.916Z : Producer : Topic: topic1, TC: 2, GC: 2
21:02:15.920Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
21:02:15.922Z : Producer : Topic: topic1, TC: 3, GC: 3
21:02:15.927Z : Producer : Topic: topic1, TC: 4, GC: 4
21:02:15.932Z : Producer : Topic: topic1, TC: 5, GC: 5
21:02:15.932Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
21:02:15.933Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
21:02:15.934Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
21:02:15.936Z : Producer : Topic: topic1, TC: 6, GC: 6
21:02:15.941Z : Producer : Topic: topic1, TC: 7, GC: 7
21:02:15.944Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
21:02:15.945Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
21:02:15.946Z : Producer : Topic: topic1, TC: 8, GC: 8
21:02:15.951Z : Producer : Topic: topic1, TC: 9, GC: 9
21:02:15.955Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
21:02:15.955Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
21:02:15.955Z : Producer : Topic: topic1, TC: 10, GC: 10
21:02:15.960Z : Producer : Topic: topic2, TC: 1, GC: 11
21:02:15.961Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 JUMP FORWARD 1
21:02:15.965Z : Producer : Topic: topic1, TC: 11, GC: 12
21:02:15.965Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 JUMP BACKWARDS 1
21:02:15.966Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 JUMP FORWARD 1
21:02:15.970Z : Producer : Topic: topic1, TC: 12, GC: 13
21:02:15.975Z : Producer : Topic: topic1, TC: 13, GC: 14
21:02:15.977Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
21:02:15.978Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
21:02:15.980Z : Producer : Topic: topic1, TC: 14, GC: 15
21:02:15.985Z : Producer : Topic: topic1, TC: 15, GC: 16
21:02:15.988Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
21:02:15.988Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
21:02:15.989Z : Producer : Topic: topic1, TC: 16, GC: 17
21:02:15.994Z : Producer : Topic: topic1, TC: 17, GC: 18
21:02:15.998Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
21:02:15.999Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 OK
21:02:15.999Z : Producer : Topic: topic1, TC: 18, GC: 19
21:02:16.004Z : Producer : Topic: topic1, TC: 19, GC: 20
21:02:16.008Z : Producer : Topic: topic1, TC: 20, GC: 21
21:02:16.010Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
21:02:16.010Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
21:02:16.010Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
21:02:16.013Z : Producer : Topic: topic2, TC: 2, GC: 22
21:02:16.014Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 OK
21:02:16.018Z : Producer : Topic: topic1, TC: 21, GC: 23
21:02:16.020Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 OK
21:02:16.023Z : Producer : Topic: topic1, TC: 22, GC: 24
21:02:16.027Z : Producer : Topic: topic1, TC: 23, GC: 25
21:02:16.032Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
21:02:16.032Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
21:02:16.032Z : Producer : Topic: topic1, TC: 24, GC: 26
21:02:16.037Z : Producer : Topic: topic1, TC: 25, GC: 27
21:02:16.041Z : Producer : Topic: topic1, TC: 26, GC: 28
21:02:16.043Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
21:02:16.043Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
21:02:16.043Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
21:02:16.046Z : Producer : Topic: topic1, TC: 27, GC: 29
21:02:16.051Z : Producer : Topic: topic1, TC: 28, GC: 30
21:02:16.054Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
21:02:16.054Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
21:02:16.055Z : Producer : Topic: topic1, TC: 29, GC: 31
21:02:16.060Z : Producer : Topic: topic1, TC: 30, GC: 32
21:02:16.065Z : Producer : Topic: topic2, TC: 3, GC: 33
21:02:16.065Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
21:02:16.066Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
21:02:16.066Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 OK
21:02:16.070Z : Producer : Topic: topic1, TC: 31, GC: 34
21:02:16.075Z : Producer : Topic: topic1, TC: 32, GC: 35
21:02:16.076Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 OK
21:02:16.076Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
21:02:16.080Z : Producer : Topic: topic1, TC: 33, GC: 36
21:02:16.085Z : Producer : Topic: topic1, TC: 34, GC: 37
21:02:16.087Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
21:02:16.087Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
21:02:16.090Z : Producer : Topic: topic1, TC: 35, GC: 38
21:02:16.095Z : Producer : Topic: topic1, TC: 36, GC: 39
21:02:16.098Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
21:02:16.098Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
21:02:16.100Z : Producer : Topic: topic1, TC: 37, GC: 40
21:02:16.105Z : Producer : Topic: topic1, TC: 38, GC: 41
21:02:16.109Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
21:02:16.109Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
21:02:16.111Z : Producer : Topic: topic1, TC: 39, GC: 42
21:02:16.118Z : Producer : Topic: topic1, TC: 40, GC: 43
21:02:16.120Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
21:02:16.121Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
21:02:16.123Z : Producer : Topic: topic2, TC: 4, GC: 44
21:02:16.124Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 OK
21:02:16.128Z : Producer : Topic: topic1, TC: 41, GC: 45
21:02:16.130Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 OK
21:02:16.133Z : Producer : Topic: topic1, TC: 42, GC: 46
21:02:16.138Z : Producer : Topic: topic1, TC: 43, GC: 47
21:02:16.142Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
21:02:16.142Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
21:02:16.142Z : Producer : Topic: topic1, TC: 44, GC: 48
21:02:16.148Z : Producer : Topic: topic1, TC: 45, GC: 49
21:02:16.153Z : Producer : Topic: topic1, TC: 46, GC: 50
21:02:16.153Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
21:02:16.153Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
21:02:16.153Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
21:02:16.157Z : Producer : Topic: topic1, TC: 47, GC: 51
21:02:16.162Z : Producer : Topic: topic1, TC: 48, GC: 52
21:02:16.164Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
21:02:16.164Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
21:02:16.168Z : Producer : Topic: topic1, TC: 49, GC: 53
21:02:16.172Z : Producer : Topic: topic1, TC: 50, GC: 54
21:02:16.175Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
21:02:16.175Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
21:02:16.177Z : Producer : Topic: topic2, TC: 5, GC: 55
21:02:16.178Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 OK
21:02:16.182Z : Producer : Topic: topic1, TC: 51, GC: 56
21:02:16.186Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 OK
21:02:16.187Z : Producer : Topic: topic1, TC: 52, GC: 57
21:02:16.192Z : Producer : Topic: topic1, TC: 53, GC: 58
21:02:16.197Z : Producer : Topic: topic1, TC: 54, GC: 59
21:02:16.198Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
21:02:16.199Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
21:02:16.202Z : Producer : Topic: topic1, TC: 55, GC: 60
21:02:16.207Z : Producer : Topic: topic1, TC: 56, GC: 61
21:02:16.208Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
21:02:16.208Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
21:02:16.209Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
21:02:16.212Z : Producer : Topic: topic1, TC: 57, GC: 62
21:02:16.217Z : Producer : Topic: topic1, TC: 58, GC: 63
21:02:16.220Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
21:02:16.220Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
21:02:16.221Z : Producer : Topic: topic1, TC: 59, GC: 64
21:02:16.226Z : Producer : Topic: topic1, TC: 60, GC: 65
21:02:16.232Z : Producer : Topic: topic2, TC: 6, GC: 66
21:02:16.232Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
21:02:16.232Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
21:02:16.232Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 OK
21:02:16.236Z : Producer : Topic: topic1, TC: 61, GC: 67
21:02:16.241Z : Producer : Topic: topic1, TC: 62, GC: 68
21:02:16.242Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 OK
21:02:16.243Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
21:02:16.246Z : Producer : Topic: topic1, TC: 63, GC: 69
21:02:16.252Z : Producer : Topic: topic1, TC: 64, GC: 70
21:02:16.253Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
21:02:16.253Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
21:02:16.257Z : Producer : Topic: topic1, TC: 65, GC: 71
21:02:16.263Z : Producer : Topic: topic1, TC: 66, GC: 72
21:02:16.264Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
21:02:16.264Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
21:02:16.267Z : Producer : Topic: topic1, TC: 67, GC: 73
21:02:16.271Z : Producer : Topic: topic1, TC: 68, GC: 74
21:02:16.275Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
21:02:16.275Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
21:02:16.275Z : Producer : Topic: topic1, TC: 69, GC: 75
21:02:16.280Z : Producer : Topic: topic1, TC: 70, GC: 76
21:02:16.286Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
21:02:16.286Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
21:02:16.286Z : Producer : Topic: topic2, TC: 7, GC: 77
21:02:16.287Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 OK
21:02:16.292Z : Producer : Topic: topic1, TC: 71, GC: 78
21:02:16.297Z : Producer : Topic: topic1, TC: 72, GC: 79
21:02:16.297Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 OK
21:02:16.301Z : Producer : Topic: topic1, TC: 73, GC: 80
21:02:16.306Z : Producer : Topic: topic1, TC: 74, GC: 81
21:02:16.307Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
21:02:16.308Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
21:02:16.308Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
21:02:16.311Z : Producer : Topic: topic1, TC: 75, GC: 82
21:02:16.316Z : Producer : Topic: topic1, TC: 76, GC: 83
21:02:16.319Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
21:02:16.319Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
21:02:16.321Z : Producer : Topic: topic1, TC: 77, GC: 84
21:02:16.325Z : Producer : Topic: topic1, TC: 78, GC: 85
21:02:16.330Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
21:02:16.331Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
21:02:16.331Z : Producer : Topic: topic1, TC: 79, GC: 86
21:02:16.336Z : Producer : Topic: topic1, TC: 80, GC: 87
21:02:16.341Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
21:02:16.341Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
21:02:16.341Z : Producer : Topic: topic2, TC: 8, GC: 88
21:02:16.342Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 OK
21:02:16.346Z : Producer : Topic: topic1, TC: 81, GC: 89
21:02:16.351Z : Producer : Topic: topic1, TC: 82, GC: 90
21:02:16.352Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 OK
21:02:16.352Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
21:02:16.356Z : Producer : Topic: topic1, TC: 83, GC: 91
21:02:16.360Z : Producer : Topic: topic1, TC: 84, GC: 92
21:02:16.363Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
21:02:16.363Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
21:02:16.365Z : Producer : Topic: topic1, TC: 85, GC: 93
21:02:16.369Z : Producer : Topic: topic1, TC: 86, GC: 94
21:02:16.374Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
21:02:16.374Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
21:02:16.374Z : Producer : Topic: topic1, TC: 87, GC: 95
21:02:16.379Z : Producer : Topic: topic1, TC: 88, GC: 96
21:02:16.383Z : Producer : Topic: topic1, TC: 89, GC: 97
21:02:16.385Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
21:02:16.385Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
21:02:16.385Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
21:02:16.388Z : Producer : Topic: topic1, TC: 90, GC: 98
21:02:16.392Z : Producer : Topic: topic2, TC: 9, GC: 99
21:02:16.393Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 JUMP FORWARD 1
21:02:16.396Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 JUMP BACKWARDS 1
21:02:16.397Z : Producer : Topic: topic1, TC: 91, GC: 100
21:02:16.407Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 JUMP FORWARD 1

6 messages out of the 100 arrived out of global order, which matches the result of case 1. So different message rates doesn’t seem to affect the cross topic ordering when consumed in strict real-time.

Case 6 - Tailing the topic, unequal message arrival rate, short constant processing time

Same as Case 2 but with 10 to 1 ratio of messages arriving at topic1. In Case 2 we saw perfect ordering.

21:04:52.488Z : Consumer : Subscribing
21:04:52.939Z : Consumer : Consuming
21:04:53.217Z : Producer : Topic: topic1, TC: 1, GC: 1
21:04:53.222Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
21:04:53.234Z : Producer : Topic: topic1, TC: 2, GC: 2
21:04:53.235Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
21:04:53.250Z : Producer : Topic: topic1, TC: 3, GC: 3
21:04:53.251Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
21:04:53.267Z : Producer : Topic: topic1, TC: 4, GC: 4
21:04:53.268Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
21:04:53.284Z : Producer : Topic: topic1, TC: 5, GC: 5
21:04:53.284Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
21:04:53.299Z : Producer : Topic: topic1, TC: 6, GC: 6
21:04:53.300Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
21:04:53.315Z : Producer : Topic: topic1, TC: 7, GC: 7
21:04:53.316Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
21:04:53.331Z : Producer : Topic: topic1, TC: 8, GC: 8
21:04:53.332Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
21:04:53.348Z : Producer : Topic: topic1, TC: 9, GC: 9
21:04:53.348Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
21:04:53.364Z : Producer : Topic: topic1, TC: 10, GC: 10
21:04:53.365Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
21:04:53.381Z : Producer : Topic: topic2, TC: 1, GC: 11
21:04:53.382Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 OK
21:04:53.397Z : Producer : Topic: topic1, TC: 11, GC: 12
21:04:53.398Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 OK
21:04:53.413Z : Producer : Topic: topic1, TC: 12, GC: 13
21:04:53.414Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
21:04:53.429Z : Producer : Topic: topic1, TC: 13, GC: 14
21:04:53.430Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
21:04:53.445Z : Producer : Topic: topic1, TC: 14, GC: 15
21:04:53.446Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
21:04:53.462Z : Producer : Topic: topic1, TC: 15, GC: 16
21:04:53.462Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
21:04:53.478Z : Producer : Topic: topic1, TC: 16, GC: 17
21:04:53.479Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
21:04:53.494Z : Producer : Topic: topic1, TC: 17, GC: 18
21:04:53.494Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 OK
21:04:53.509Z : Producer : Topic: topic1, TC: 18, GC: 19
21:04:53.509Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
21:04:53.524Z : Producer : Topic: topic1, TC: 19, GC: 20
21:04:53.525Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
21:04:53.540Z : Producer : Topic: topic1, TC: 20, GC: 21
21:04:53.541Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
21:04:53.557Z : Producer : Topic: topic2, TC: 2, GC: 22
21:04:53.557Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 OK
21:04:53.572Z : Producer : Topic: topic1, TC: 21, GC: 23
21:04:53.573Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 OK
21:04:53.588Z : Producer : Topic: topic1, TC: 22, GC: 24
21:04:53.589Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
21:04:53.605Z : Producer : Topic: topic1, TC: 23, GC: 25
21:04:53.605Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
21:04:53.621Z : Producer : Topic: topic1, TC: 24, GC: 26
21:04:53.621Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
21:04:53.637Z : Producer : Topic: topic1, TC: 25, GC: 27
21:04:53.637Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
21:04:53.653Z : Producer : Topic: topic1, TC: 26, GC: 28
21:04:53.653Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
21:04:53.669Z : Producer : Topic: topic1, TC: 27, GC: 29
21:04:53.670Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
21:04:53.686Z : Producer : Topic: topic1, TC: 28, GC: 30
21:04:53.687Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
21:04:53.703Z : Producer : Topic: topic1, TC: 29, GC: 31
21:04:53.703Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
21:04:53.718Z : Producer : Topic: topic1, TC: 30, GC: 32
21:04:53.719Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
21:04:53.735Z : Producer : Topic: topic2, TC: 3, GC: 33
21:04:53.735Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 OK
21:04:53.751Z : Producer : Topic: topic1, TC: 31, GC: 34
21:04:53.752Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 OK
21:04:53.767Z : Producer : Topic: topic1, TC: 32, GC: 35
21:04:53.767Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
21:04:53.783Z : Producer : Topic: topic1, TC: 33, GC: 36
21:04:53.784Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
21:04:53.799Z : Producer : Topic: topic1, TC: 34, GC: 37
21:04:53.800Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
21:04:53.814Z : Producer : Topic: topic1, TC: 35, GC: 38
21:04:53.815Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
21:04:53.831Z : Producer : Topic: topic1, TC: 36, GC: 39
21:04:53.832Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
21:04:53.846Z : Producer : Topic: topic1, TC: 37, GC: 40
21:04:53.847Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
21:04:53.863Z : Producer : Topic: topic1, TC: 38, GC: 41
21:04:53.863Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
21:04:53.879Z : Producer : Topic: topic1, TC: 39, GC: 42
21:04:53.879Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
21:04:53.894Z : Producer : Topic: topic1, TC: 40, GC: 43
21:04:53.895Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
21:04:53.910Z : Producer : Topic: topic2, TC: 4, GC: 44
21:04:53.911Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 OK
21:04:53.927Z : Producer : Topic: topic1, TC: 41, GC: 45
21:04:53.927Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 OK
21:04:53.942Z : Producer : Topic: topic1, TC: 42, GC: 46
21:04:53.943Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
21:04:53.958Z : Producer : Topic: topic1, TC: 43, GC: 47
21:04:53.959Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
21:04:53.974Z : Producer : Topic: topic1, TC: 44, GC: 48
21:04:53.975Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
21:04:53.991Z : Producer : Topic: topic1, TC: 45, GC: 49
21:04:53.991Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
21:04:54.007Z : Producer : Topic: topic1, TC: 46, GC: 50
21:04:54.008Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
21:04:54.022Z : Producer : Topic: topic1, TC: 47, GC: 51
21:04:54.023Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
21:04:54.039Z : Producer : Topic: topic1, TC: 48, GC: 52
21:04:54.041Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
21:04:54.056Z : Producer : Topic: topic1, TC: 49, GC: 53
21:04:54.057Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
21:04:54.072Z : Producer : Topic: topic1, TC: 50, GC: 54
21:04:54.073Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
21:04:54.089Z : Producer : Topic: topic2, TC: 5, GC: 55
21:04:54.090Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 OK
21:04:54.104Z : Producer : Topic: topic1, TC: 51, GC: 56
21:04:54.105Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 OK
21:04:54.120Z : Producer : Topic: topic1, TC: 52, GC: 57
21:04:54.121Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
21:04:54.136Z : Producer : Topic: topic1, TC: 53, GC: 58
21:04:54.137Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
21:04:54.154Z : Producer : Topic: topic1, TC: 54, GC: 59
21:04:54.154Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
21:04:54.169Z : Producer : Topic: topic1, TC: 55, GC: 60
21:04:54.170Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
21:04:54.185Z : Producer : Topic: topic1, TC: 56, GC: 61
21:04:54.186Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
21:04:54.201Z : Producer : Topic: topic1, TC: 57, GC: 62
21:04:54.202Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
21:04:54.217Z : Producer : Topic: topic1, TC: 58, GC: 63
21:04:54.218Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
21:04:54.234Z : Producer : Topic: topic1, TC: 59, GC: 64
21:04:54.234Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
21:04:54.250Z : Producer : Topic: topic1, TC: 60, GC: 65
21:04:54.250Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
21:04:54.266Z : Producer : Topic: topic2, TC: 6, GC: 66
21:04:54.266Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 OK
21:04:54.281Z : Producer : Topic: topic1, TC: 61, GC: 67
21:04:54.282Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 OK
21:04:54.299Z : Producer : Topic: topic1, TC: 62, GC: 68
21:04:54.300Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
21:04:54.315Z : Producer : Topic: topic1, TC: 63, GC: 69
21:04:54.315Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
21:04:54.331Z : Producer : Topic: topic1, TC: 64, GC: 70
21:04:54.332Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
21:04:54.347Z : Producer : Topic: topic1, TC: 65, GC: 71
21:04:54.347Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
21:04:54.363Z : Producer : Topic: topic1, TC: 66, GC: 72
21:04:54.363Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
21:04:54.378Z : Producer : Topic: topic1, TC: 67, GC: 73
21:04:54.379Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
21:04:54.393Z : Producer : Topic: topic1, TC: 68, GC: 74
21:04:54.394Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
21:04:54.408Z : Producer : Topic: topic1, TC: 69, GC: 75
21:04:54.409Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
21:04:54.424Z : Producer : Topic: topic1, TC: 70, GC: 76
21:04:54.425Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
21:04:54.441Z : Producer : Topic: topic2, TC: 7, GC: 77
21:04:54.442Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 OK
21:04:54.456Z : Producer : Topic: topic1, TC: 71, GC: 78
21:04:54.457Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 OK
21:04:54.471Z : Producer : Topic: topic1, TC: 72, GC: 79
21:04:54.472Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
21:04:54.487Z : Producer : Topic: topic1, TC: 73, GC: 80
21:04:54.488Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
21:04:54.503Z : Producer : Topic: topic1, TC: 74, GC: 81
21:04:54.504Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
21:04:54.519Z : Producer : Topic: topic1, TC: 75, GC: 82
21:04:54.520Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
21:04:54.536Z : Producer : Topic: topic1, TC: 76, GC: 83
21:04:54.536Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
21:04:54.551Z : Producer : Topic: topic1, TC: 77, GC: 84
21:04:54.552Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
21:04:54.568Z : Producer : Topic: topic1, TC: 78, GC: 85
21:04:54.569Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
21:04:54.586Z : Producer : Topic: topic1, TC: 79, GC: 86
21:04:54.587Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
21:04:54.602Z : Producer : Topic: topic1, TC: 80, GC: 87
21:04:54.603Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
21:04:54.617Z : Producer : Topic: topic2, TC: 8, GC: 88
21:04:54.618Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 OK
21:04:54.632Z : Producer : Topic: topic1, TC: 81, GC: 89
21:04:54.633Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 OK
21:04:54.648Z : Producer : Topic: topic1, TC: 82, GC: 90
21:04:54.649Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
21:04:54.664Z : Producer : Topic: topic1, TC: 83, GC: 91
21:04:54.665Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
21:04:54.680Z : Producer : Topic: topic1, TC: 84, GC: 92
21:04:54.681Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
21:04:54.696Z : Producer : Topic: topic1, TC: 85, GC: 93
21:04:54.697Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
21:04:54.712Z : Producer : Topic: topic1, TC: 86, GC: 94
21:04:54.712Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
21:04:54.727Z : Producer : Topic: topic1, TC: 87, GC: 95
21:04:54.728Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
21:04:54.743Z : Producer : Topic: topic1, TC: 88, GC: 96
21:04:54.745Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
21:04:54.759Z : Producer : Topic: topic1, TC: 89, GC: 97
21:04:54.760Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
21:04:54.775Z : Producer : Topic: topic1, TC: 90, GC: 98
21:04:54.776Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
21:04:54.791Z : Producer : Topic: topic2, TC: 9, GC: 99
21:04:54.792Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 OK
21:04:54.806Z : Producer : Topic: topic1, TC: 91, GC: 100
21:04:54.807Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 OK

Again, we get the perfect publish-consume interleaving producing perfect cross topic ordering. Again, real-time consumption makes perfect ordering possible.

Case 7 - Tailing the topic, unequal message arrival rate, short processing time with jitter

Same as Case 3 but with a 10 to 1 ratio of messages. Case 3 saw perfect ordering.

21:07:11.030Z : Consumer : Subscribing
21:07:11.497Z : Consumer : Consuming
21:07:11.766Z : Producer : Topic: topic1, TC: 1, GC: 1
21:07:11.770Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
21:07:11.796Z : Producer : Topic: topic1, TC: 2, GC: 2
21:07:11.820Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
21:07:11.827Z : Producer : Topic: topic1, TC: 3, GC: 3
21:07:11.835Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
21:07:11.858Z : Producer : Topic: topic1, TC: 4, GC: 4
21:07:11.858Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
21:07:11.888Z : Producer : Topic: topic1, TC: 5, GC: 5
21:07:11.889Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
21:07:11.919Z : Producer : Topic: topic1, TC: 6, GC: 6
21:07:11.931Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
21:07:11.951Z : Producer : Topic: topic1, TC: 7, GC: 7
21:07:11.982Z : Producer : Topic: topic1, TC: 8, GC: 8
21:07:12.013Z : Producer : Topic: topic1, TC: 9, GC: 9
21:07:12.045Z : Producer : Topic: topic1, TC: 10, GC: 10
21:07:12.076Z : Producer : Topic: topic2, TC: 1, GC: 11
21:07:12.106Z : Producer : Topic: topic1, TC: 11, GC: 12
21:07:12.137Z : Producer : Topic: topic1, TC: 12, GC: 13
21:07:12.144Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
21:07:12.168Z : Producer : Topic: topic1, TC: 13, GC: 14
21:07:12.176Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
21:07:12.200Z : Producer : Topic: topic1, TC: 14, GC: 15
21:07:12.220Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
21:07:12.223Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
21:07:12.228Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 OK
21:07:12.231Z : Producer : Topic: topic1, TC: 15, GC: 16
21:07:12.252Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 OK
21:07:12.260Z : Producer : Topic: topic1, TC: 16, GC: 17
21:07:12.292Z : Producer : Topic: topic1, TC: 17, GC: 18
21:07:12.297Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
21:07:12.323Z : Producer : Topic: topic1, TC: 18, GC: 19
21:07:12.345Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
21:07:12.354Z : Producer : Topic: topic1, TC: 19, GC: 20
21:07:12.365Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
21:07:12.385Z : Producer : Topic: topic1, TC: 20, GC: 21
21:07:12.409Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
21:07:12.417Z : Producer : Topic: topic2, TC: 2, GC: 22
21:07:12.432Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
21:07:12.446Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 OK
21:07:12.447Z : Producer : Topic: topic1, TC: 21, GC: 23
21:07:12.464Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
21:07:12.478Z : Producer : Topic: topic1, TC: 22, GC: 24
21:07:12.490Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
21:07:12.496Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
21:07:12.509Z : Producer : Topic: topic1, TC: 23, GC: 25
21:07:12.532Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 OK
21:07:12.540Z : Producer : Topic: topic1, TC: 24, GC: 26
21:07:12.542Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 OK
21:07:12.570Z : Producer : Topic: topic1, TC: 25, GC: 27
21:07:12.584Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
21:07:12.602Z : Producer : Topic: topic1, TC: 26, GC: 28
21:07:12.610Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
21:07:12.633Z : Producer : Topic: topic1, TC: 27, GC: 29
21:07:12.648Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
21:07:12.664Z : Producer : Topic: topic1, TC: 28, GC: 30
21:07:12.697Z : Producer : Topic: topic1, TC: 29, GC: 31
21:07:12.697Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
21:07:12.727Z : Producer : Topic: topic1, TC: 30, GC: 32
21:07:12.747Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
21:07:12.758Z : Producer : Topic: topic2, TC: 3, GC: 33
21:07:12.772Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
21:07:12.779Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
21:07:12.785Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
21:07:12.789Z : Producer : Topic: topic1, TC: 31, GC: 34
21:07:12.816Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
21:07:12.820Z : Producer : Topic: topic1, TC: 32, GC: 35
21:07:12.851Z : Producer : Topic: topic1, TC: 33, GC: 36
21:07:12.883Z : Producer : Topic: topic1, TC: 34, GC: 37
21:07:12.915Z : Producer : Topic: topic1, TC: 35, GC: 38
21:07:12.947Z : Producer : Topic: topic1, TC: 36, GC: 39
21:07:12.978Z : Producer : Topic: topic1, TC: 37, GC: 40
21:07:13.008Z : Producer : Topic: topic1, TC: 38, GC: 41
21:07:13.039Z : Producer : Topic: topic1, TC: 39, GC: 42
21:07:13.043Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 OK
21:07:13.071Z : Producer : Topic: topic1, TC: 40, GC: 43
21:07:13.077Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 OK
21:07:13.080Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
21:07:13.088Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
21:07:13.103Z : Producer : Topic: topic2, TC: 4, GC: 44
21:07:13.104Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
21:07:13.110Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
21:07:13.134Z : Producer : Topic: topic1, TC: 41, GC: 45
21:07:13.151Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
21:07:13.166Z : Producer : Topic: topic1, TC: 42, GC: 46
21:07:13.188Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
21:07:13.196Z : Producer : Topic: topic1, TC: 43, GC: 47
21:07:13.227Z : Producer : Topic: topic1, TC: 44, GC: 48
21:07:13.235Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
21:07:13.248Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
21:07:13.257Z : Producer : Topic: topic1, TC: 45, GC: 49
21:07:13.265Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
21:07:13.287Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 OK
21:07:13.290Z : Producer : Topic: topic1, TC: 46, GC: 50
21:07:13.320Z : Producer : Topic: topic1, TC: 47, GC: 51
21:07:13.322Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 OK
21:07:13.347Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
21:07:13.351Z : Producer : Topic: topic1, TC: 48, GC: 52
21:07:13.380Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
21:07:13.382Z : Producer : Topic: topic1, TC: 49, GC: 53
21:07:13.383Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
21:07:13.394Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
21:07:13.413Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
21:07:13.414Z : Producer : Topic: topic1, TC: 50, GC: 54
21:07:13.435Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
21:07:13.438Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
21:07:13.444Z : Producer : Topic: topic2, TC: 5, GC: 55
21:07:13.475Z : Producer : Topic: topic1, TC: 51, GC: 56
21:07:13.488Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
21:07:13.506Z : Producer : Topic: topic1, TC: 52, GC: 57
21:07:13.519Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
21:07:13.532Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 OK
21:07:13.538Z : Producer : Topic: topic1, TC: 53, GC: 58
21:07:13.567Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 OK
21:07:13.569Z : Producer : Topic: topic1, TC: 54, GC: 59
21:07:13.585Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
21:07:13.590Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
21:07:13.599Z : Producer : Topic: topic1, TC: 55, GC: 60
21:07:13.631Z : Producer : Topic: topic1, TC: 56, GC: 61
21:07:13.631Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
21:07:13.662Z : Producer : Topic: topic1, TC: 57, GC: 62
21:07:13.680Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
21:07:13.692Z : Producer : Topic: topic1, TC: 58, GC: 63
21:07:13.704Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
21:07:13.722Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
21:07:13.723Z : Producer : Topic: topic1, TC: 59, GC: 64
21:07:13.755Z : Producer : Topic: topic1, TC: 60, GC: 65
21:07:13.767Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
21:07:13.787Z : Producer : Topic: topic2, TC: 6, GC: 66
21:07:13.794Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
21:07:13.798Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
21:07:13.804Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 OK
21:07:13.818Z : Producer : Topic: topic1, TC: 61, GC: 67
21:07:13.829Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 OK
21:07:13.848Z : Producer : Topic: topic1, TC: 62, GC: 68
21:07:13.875Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
21:07:13.878Z : Producer : Topic: topic1, TC: 63, GC: 69
21:07:13.890Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
21:07:13.909Z : Producer : Topic: topic1, TC: 64, GC: 70
21:07:13.910Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
21:07:13.940Z : Producer : Topic: topic1, TC: 65, GC: 71
21:07:13.972Z : Producer : Topic: topic1, TC: 66, GC: 72
21:07:14.003Z : Producer : Topic: topic1, TC: 67, GC: 73
21:07:14.034Z : Producer : Topic: topic1, TC: 68, GC: 74
21:07:14.065Z : Producer : Topic: topic1, TC: 69, GC: 75
21:07:14.096Z : Producer : Topic: topic1, TC: 70, GC: 76
21:07:14.127Z : Producer : Topic: topic2, TC: 7, GC: 77
21:07:14.143Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
21:07:14.158Z : Producer : Topic: topic1, TC: 71, GC: 78
21:07:14.189Z : Producer : Topic: topic1, TC: 72, GC: 79
21:07:14.219Z : Producer : Topic: topic1, TC: 73, GC: 80
21:07:14.250Z : Producer : Topic: topic1, TC: 74, GC: 81
21:07:14.281Z : Producer : Topic: topic1, TC: 75, GC: 82
21:07:14.312Z : Producer : Topic: topic1, TC: 76, GC: 83
21:07:14.342Z : Producer : Topic: topic1, TC: 77, GC: 84
21:07:14.357Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
21:07:14.364Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
21:07:14.373Z : Producer : Topic: topic1, TC: 78, GC: 85
21:07:14.405Z : Producer : Topic: topic1, TC: 79, GC: 86
21:07:14.407Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
21:07:14.437Z : Producer : Topic: topic1, TC: 80, GC: 87
21:07:14.437Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
21:07:14.468Z : Producer : Topic: topic2, TC: 8, GC: 88
21:07:14.469Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
21:07:14.499Z : Producer : Topic: topic1, TC: 81, GC: 89
21:07:14.512Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 OK
21:07:14.530Z : Producer : Topic: topic1, TC: 82, GC: 90
21:07:14.534Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 OK
21:07:14.552Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
21:07:14.561Z : Producer : Topic: topic1, TC: 83, GC: 91
21:07:14.592Z : Producer : Topic: topic1, TC: 84, GC: 92
21:07:14.623Z : Producer : Topic: topic1, TC: 85, GC: 93
21:07:14.654Z : Producer : Topic: topic1, TC: 86, GC: 94
21:07:14.685Z : Producer : Topic: topic1, TC: 87, GC: 95
21:07:14.716Z : Producer : Topic: topic1, TC: 88, GC: 96
21:07:14.747Z : Producer : Topic: topic1, TC: 89, GC: 97
21:07:14.778Z : Producer : Topic: topic1, TC: 90, GC: 98
21:07:14.784Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
21:07:14.787Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
21:07:14.809Z : Producer : Topic: topic2, TC: 9, GC: 99
21:07:14.817Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
21:07:14.837Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
21:07:14.840Z : Producer : Topic: topic1, TC: 91, GC: 100
21:07:14.869Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
21:07:14.873Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
21:07:14.894Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
21:07:14.912Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
21:07:14.953Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 OK
21:07:14.959Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 OK
21:07:14.999Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
21:07:15.042Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
21:07:15.090Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
21:07:15.111Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
21:07:15.146Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
21:07:15.188Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
21:07:15.226Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
21:07:15.253Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
21:07:15.268Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
21:07:15.279Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 OK
21:07:15.296Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 OK

The default internal buffer size is 1000, so while my test code didn’t see perfect interleaving of publish-consume, the client itself will be achieving this. So arrival rate this hasn’t affected ordering when consuming in real-time.

Case 8 - Tailing the topic, unequal message arrival rate, fast producer

Like with Case 4, we’ll reduce the size of the internal buffer to prevent it from keeping up with the producer, despite my slow application code. We’ll set the buffer size to 10 again.

This is the first test where we lose our real-time consumption combined with unequal message rates to the topics.

21:13:51.601Z : Consumer : Subscribing
21:13:52.054Z : Consumer : Consuming
21:13:57.224Z : Producer : Topic: topic1, TC: 1, GC: 1
21:13:57.229Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 OK
21:13:57.232Z : Producer : Topic: topic1, TC: 2, GC: 2
21:13:57.237Z : Producer : Topic: topic1, TC: 3, GC: 3
21:13:57.243Z : Producer : Topic: topic1, TC: 4, GC: 4
21:13:57.248Z : Producer : Topic: topic1, TC: 5, GC: 5
21:13:57.255Z : Producer : Topic: topic1, TC: 6, GC: 6
21:13:57.263Z : Producer : Topic: topic1, TC: 7, GC: 7
21:13:57.269Z : Producer : Topic: topic1, TC: 8, GC: 8
21:13:57.275Z : Producer : Topic: topic1, TC: 9, GC: 9
21:13:57.282Z : Producer : Topic: topic1, TC: 10, GC: 10
21:13:57.288Z : Producer : Topic: topic2, TC: 1, GC: 11
21:13:57.295Z : Producer : Topic: topic1, TC: 11, GC: 12
21:13:57.301Z : Producer : Topic: topic1, TC: 12, GC: 13
21:13:57.308Z : Producer : Topic: topic1, TC: 13, GC: 14
21:13:57.315Z : Producer : Topic: topic1, TC: 14, GC: 15
21:13:57.321Z : Producer : Topic: topic1, TC: 15, GC: 16
21:13:57.327Z : Producer : Topic: topic1, TC: 16, GC: 17
21:13:57.331Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
21:13:57.333Z : Producer : Topic: topic1, TC: 17, GC: 18
21:13:57.339Z : Producer : Topic: topic1, TC: 18, GC: 19
21:13:57.345Z : Producer : Topic: topic1, TC: 19, GC: 20
21:13:57.351Z : Producer : Topic: topic1, TC: 20, GC: 21
21:13:57.359Z : Producer : Topic: topic2, TC: 2, GC: 22
21:13:57.366Z : Producer : Topic: topic1, TC: 21, GC: 23
21:13:57.372Z : Producer : Topic: topic1, TC: 22, GC: 24
21:13:57.379Z : Producer : Topic: topic1, TC: 23, GC: 25
21:13:57.385Z : Producer : Topic: topic1, TC: 24, GC: 26
21:13:57.391Z : Producer : Topic: topic1, TC: 25, GC: 27
21:13:57.398Z : Producer : Topic: topic1, TC: 26, GC: 28
21:13:57.404Z : Producer : Topic: topic1, TC: 27, GC: 29
21:13:57.411Z : Producer : Topic: topic1, TC: 28, GC: 30
21:13:57.417Z : Producer : Topic: topic1, TC: 29, GC: 31
21:13:57.423Z : Producer : Topic: topic1, TC: 30, GC: 32
21:13:57.430Z : Producer : Topic: topic2, TC: 3, GC: 33
21:13:57.432Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
21:13:57.437Z : Producer : Topic: topic1, TC: 31, GC: 34
21:13:57.444Z : Producer : Topic: topic1, TC: 32, GC: 35
21:13:57.451Z : Producer : Topic: topic1, TC: 33, GC: 36
21:13:57.458Z : Producer : Topic: topic1, TC: 34, GC: 37
21:13:57.465Z : Producer : Topic: topic1, TC: 35, GC: 38
21:13:57.471Z : Producer : Topic: topic1, TC: 36, GC: 39
21:13:57.479Z : Producer : Topic: topic1, TC: 37, GC: 40
21:13:57.486Z : Producer : Topic: topic1, TC: 38, GC: 41
21:13:57.492Z : Producer : Topic: topic1, TC: 39, GC: 42
21:13:57.499Z : Producer : Topic: topic1, TC: 40, GC: 43
21:13:57.505Z : Producer : Topic: topic2, TC: 4, GC: 44
21:13:57.512Z : Producer : Topic: topic1, TC: 41, GC: 45
21:13:57.518Z : Producer : Topic: topic1, TC: 42, GC: 46
21:13:57.525Z : Producer : Topic: topic1, TC: 43, GC: 47
21:13:57.532Z : Producer : Topic: topic1, TC: 44, GC: 48
21:13:57.532Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
21:13:57.539Z : Producer : Topic: topic1, TC: 45, GC: 49
21:13:57.547Z : Producer : Topic: topic1, TC: 46, GC: 50
21:13:57.553Z : Producer : Topic: topic1, TC: 47, GC: 51
21:13:57.560Z : Producer : Topic: topic1, TC: 48, GC: 52
21:13:57.567Z : Producer : Topic: topic1, TC: 49, GC: 53
21:13:57.573Z : Producer : Topic: topic1, TC: 50, GC: 54
21:13:57.579Z : Producer : Topic: topic2, TC: 5, GC: 55
21:13:57.586Z : Producer : Topic: topic1, TC: 51, GC: 56
21:13:57.592Z : Producer : Topic: topic1, TC: 52, GC: 57
21:13:57.599Z : Producer : Topic: topic1, TC: 53, GC: 58
21:13:57.606Z : Producer : Topic: topic1, TC: 54, GC: 59
21:13:57.613Z : Producer : Topic: topic1, TC: 55, GC: 60
21:13:57.619Z : Producer : Topic: topic1, TC: 56, GC: 61
21:13:57.625Z : Producer : Topic: topic1, TC: 57, GC: 62
21:13:57.632Z : Producer : Topic: topic1, TC: 58, GC: 63
21:13:57.634Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
21:13:57.639Z : Producer : Topic: topic1, TC: 59, GC: 64
21:13:57.646Z : Producer : Topic: topic1, TC: 60, GC: 65
21:13:57.652Z : Producer : Topic: topic2, TC: 6, GC: 66
21:13:57.659Z : Producer : Topic: topic1, TC: 61, GC: 67
21:13:57.666Z : Producer : Topic: topic1, TC: 62, GC: 68
21:13:57.672Z : Producer : Topic: topic1, TC: 63, GC: 69
21:13:57.679Z : Producer : Topic: topic1, TC: 64, GC: 70
21:13:57.686Z : Producer : Topic: topic1, TC: 65, GC: 71
21:13:57.692Z : Producer : Topic: topic1, TC: 66, GC: 72
21:13:57.699Z : Producer : Topic: topic1, TC: 67, GC: 73
21:13:57.705Z : Producer : Topic: topic1, TC: 68, GC: 74
21:13:57.712Z : Producer : Topic: topic1, TC: 69, GC: 75
21:13:57.719Z : Producer : Topic: topic1, TC: 70, GC: 76
21:13:57.726Z : Producer : Topic: topic2, TC: 7, GC: 77
21:13:57.733Z : Producer : Topic: topic1, TC: 71, GC: 78
21:13:57.735Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
21:13:57.740Z : Producer : Topic: topic1, TC: 72, GC: 79
21:13:57.747Z : Producer : Topic: topic1, TC: 73, GC: 80
21:13:57.753Z : Producer : Topic: topic1, TC: 74, GC: 81
21:13:57.759Z : Producer : Topic: topic1, TC: 75, GC: 82
21:13:57.766Z : Producer : Topic: topic1, TC: 76, GC: 83
21:13:57.771Z : Producer : Topic: topic1, TC: 77, GC: 84
21:13:57.776Z : Producer : Topic: topic1, TC: 78, GC: 85
21:13:57.783Z : Producer : Topic: topic1, TC: 79, GC: 86
21:13:57.788Z : Producer : Topic: topic1, TC: 80, GC: 87
21:13:57.795Z : Producer : Topic: topic2, TC: 8, GC: 88
21:13:57.802Z : Producer : Topic: topic1, TC: 81, GC: 89
21:13:57.809Z : Producer : Topic: topic1, TC: 82, GC: 90
21:13:57.816Z : Producer : Topic: topic1, TC: 83, GC: 91
21:13:57.822Z : Producer : Topic: topic1, TC: 84, GC: 92
21:13:57.828Z : Producer : Topic: topic1, TC: 85, GC: 93
21:13:57.835Z : Producer : Topic: topic1, TC: 86, GC: 94
21:13:57.837Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
21:13:57.842Z : Producer : Topic: topic1, TC: 87, GC: 95
21:13:57.848Z : Producer : Topic: topic1, TC: 88, GC: 96
21:13:57.855Z : Producer : Topic: topic1, TC: 89, GC: 97
21:13:57.863Z : Producer : Topic: topic1, TC: 90, GC: 98
21:13:57.870Z : Producer : Topic: topic2, TC: 9, GC: 99
21:13:57.877Z : Producer : Topic: topic1, TC: 91, GC: 100
21:13:57.938Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
21:13:58.039Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
21:13:58.141Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
21:13:58.242Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 OK
21:13:58.343Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 OK
21:13:58.444Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 JUMP FORWARD 9
21:13:58.545Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 JUMP BACKWARDS 9
21:13:58.646Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 JUMP FORWARD 19
21:13:58.747Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 JUMP BACKWARDS 19
21:13:58.848Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 JUMP FORWARD 29
21:13:58.949Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 JUMP BACKWARDS 29
21:13:59.050Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 JUMP FORWARD 39
21:13:59.150Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 JUMP BACKWARDS 39
21:13:59.252Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 JUMP FORWARD 49
21:13:59.353Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 JUMP BACKWARDS 49
21:13:59.454Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 JUMP FORWARD 59
21:13:59.555Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 JUMP BACKWARDS 59
21:13:59.656Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 JUMP FORWARD 69
21:13:59.757Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 JUMP BACKWARDS 69
21:13:59.858Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 JUMP FORWARD 79
21:13:59.959Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 JUMP BACKWARDS 79
21:14:00.060Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
21:14:00.161Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 JUMP FORWARD 1
21:14:00.262Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
21:14:00.363Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
21:14:00.464Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
21:14:00.565Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
21:14:00.666Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
21:14:00.767Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
21:14:00.868Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
21:14:00.968Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
21:14:01.070Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
21:14:01.171Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 JUMP FORWARD 1
21:14:01.272Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
21:14:01.373Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
21:14:01.474Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
21:14:01.574Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
21:14:01.675Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
21:14:01.776Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
21:14:01.877Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
21:14:01.979Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
21:14:02.080Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
21:14:02.181Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 JUMP FORWARD 1
21:14:02.282Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
21:14:02.383Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
21:14:02.484Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
21:14:02.585Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
21:14:02.686Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
21:14:02.787Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
21:14:02.889Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
21:14:02.989Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
21:14:03.090Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
21:14:03.192Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 JUMP FORWARD 1
21:14:03.293Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
21:14:03.394Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
21:14:03.495Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
21:14:03.596Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
21:14:03.697Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
21:14:03.798Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
21:14:03.899Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
21:14:04Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
21:14:04.101Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
21:14:04.202Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 JUMP FORWARD 1
21:14:04.303Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
21:14:04.404Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
21:14:04.505Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
21:14:04.606Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
21:14:04.707Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
21:14:04.808Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
21:14:04.909Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
21:14:05.010Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
21:14:05.112Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
21:14:05.213Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 JUMP FORWARD 1
21:14:05.314Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
21:14:05.415Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
21:14:05.516Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
21:14:05.617Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
21:14:05.717Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
21:14:05.818Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
21:14:05.920Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
21:14:06.021Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
21:14:06.122Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
21:14:06.223Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 JUMP FORWARD 1
21:14:06.324Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
21:14:06.425Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
21:14:06.526Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
21:14:06.627Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
21:14:06.728Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
21:14:06.829Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
21:14:06.930Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
21:14:07.031Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
21:14:07.133Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
21:14:07.233Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 JUMP FORWARD 1

Massive ordering issues this time. The consumer consumes from each topic at the same rate meaning that is quickly processes topic2, which only had 10 messages in it, causing large jumps forwards and back in the global sequence, then once the topic had been consumed, topic 1 got JUMP FORWARDS 1 every ten messages corresponding to the topic2 message now already consumed.

So we see that consumers need to consume in strict real-time in order to get decent ordering.
That leads us to the Time-Travel cases. Obviously not real-time so we should see ordering problems.

Case 9 - Time-travel, equal message arrival rate

This time the producer will send all 100 messages, then the consumer will start up and read from the beginning of the topics.

21:27:30.167Z : Producer : Topic: topic1, TC: 1, GC: 1
21:27:30.176Z : Producer : Topic: topic2, TC: 1, GC: 2
21:27:30.183Z : Producer : Topic: topic1, TC: 2, GC: 3
21:27:30.191Z : Producer : Topic: topic2, TC: 2, GC: 4
21:27:30.198Z : Producer : Topic: topic1, TC: 3, GC: 5
21:27:30.204Z : Producer : Topic: topic2, TC: 3, GC: 6
21:27:30.212Z : Producer : Topic: topic1, TC: 4, GC: 7
21:27:30.219Z : Producer : Topic: topic2, TC: 4, GC: 8
21:27:30.229Z : Producer : Topic: topic1, TC: 5, GC: 9
21:27:30.238Z : Producer : Topic: topic2, TC: 5, GC: 10
21:27:30.244Z : Producer : Topic: topic1, TC: 6, GC: 11
21:27:30.252Z : Producer : Topic: topic2, TC: 6, GC: 12
21:27:30.259Z : Producer : Topic: topic1, TC: 7, GC: 13
21:27:30.266Z : Producer : Topic: topic2, TC: 7, GC: 14
21:27:30.273Z : Producer : Topic: topic1, TC: 8, GC: 15
21:27:30.280Z : Producer : Topic: topic2, TC: 8, GC: 16
21:27:30.287Z : Producer : Topic: topic1, TC: 9, GC: 17
21:27:30.293Z : Producer : Topic: topic2, TC: 9, GC: 18
21:27:30.300Z : Producer : Topic: topic1, TC: 10, GC: 19
21:27:30.307Z : Producer : Topic: topic2, TC: 10, GC: 20
21:27:30.314Z : Producer : Topic: topic1, TC: 11, GC: 21
21:27:30.321Z : Producer : Topic: topic2, TC: 11, GC: 22
21:27:30.328Z : Producer : Topic: topic1, TC: 12, GC: 23
21:27:30.335Z : Producer : Topic: topic2, TC: 12, GC: 24
21:27:30.341Z : Producer : Topic: topic1, TC: 13, GC: 25
21:27:30.347Z : Producer : Topic: topic2, TC: 13, GC: 26
21:27:30.354Z : Producer : Topic: topic1, TC: 14, GC: 27
21:27:30.359Z : Producer : Topic: topic2, TC: 14, GC: 28
21:27:30.365Z : Producer : Topic: topic1, TC: 15, GC: 29
21:27:30.371Z : Producer : Topic: topic2, TC: 15, GC: 30
21:27:30.378Z : Producer : Topic: topic1, TC: 16, GC: 31
21:27:30.384Z : Producer : Topic: topic2, TC: 16, GC: 32
21:27:30.391Z : Producer : Topic: topic1, TC: 17, GC: 33
21:27:30.398Z : Producer : Topic: topic2, TC: 17, GC: 34
21:27:30.404Z : Producer : Topic: topic1, TC: 18, GC: 35
21:27:30.411Z : Producer : Topic: topic2, TC: 18, GC: 36
21:27:30.417Z : Producer : Topic: topic1, TC: 19, GC: 37
21:27:30.423Z : Producer : Topic: topic2, TC: 19, GC: 38
21:27:30.429Z : Producer : Topic: topic1, TC: 20, GC: 39
21:27:30.436Z : Producer : Topic: topic2, TC: 20, GC: 40
21:27:30.443Z : Producer : Topic: topic1, TC: 21, GC: 41
21:27:30.450Z : Producer : Topic: topic2, TC: 21, GC: 42
21:27:30.456Z : Producer : Topic: topic1, TC: 22, GC: 43
21:27:30.463Z : Producer : Topic: topic2, TC: 22, GC: 44
21:27:30.470Z : Producer : Topic: topic1, TC: 23, GC: 45
21:27:30.476Z : Producer : Topic: topic2, TC: 23, GC: 46
21:27:30.482Z : Producer : Topic: topic1, TC: 24, GC: 47
21:27:30.488Z : Producer : Topic: topic2, TC: 24, GC: 48
21:27:30.494Z : Producer : Topic: topic1, TC: 25, GC: 49
21:27:30.500Z : Producer : Topic: topic2, TC: 25, GC: 50
21:27:30.506Z : Producer : Topic: topic1, TC: 26, GC: 51
21:27:30.512Z : Producer : Topic: topic2, TC: 26, GC: 52
21:27:30.518Z : Producer : Topic: topic1, TC: 27, GC: 53
21:27:30.524Z : Producer : Topic: topic2, TC: 27, GC: 54
21:27:30.530Z : Producer : Topic: topic1, TC: 28, GC: 55
21:27:30.537Z : Producer : Topic: topic2, TC: 28, GC: 56
21:27:30.542Z : Producer : Topic: topic1, TC: 29, GC: 57
21:27:30.549Z : Producer : Topic: topic2, TC: 29, GC: 58
21:27:30.555Z : Producer : Topic: topic1, TC: 30, GC: 59
21:27:30.561Z : Producer : Topic: topic2, TC: 30, GC: 60
21:27:30.567Z : Producer : Topic: topic1, TC: 31, GC: 61
21:27:30.573Z : Producer : Topic: topic2, TC: 31, GC: 62
21:27:30.580Z : Producer : Topic: topic1, TC: 32, GC: 63
21:27:30.586Z : Producer : Topic: topic2, TC: 32, GC: 64
21:27:30.592Z : Producer : Topic: topic1, TC: 33, GC: 65
21:27:30.598Z : Producer : Topic: topic2, TC: 33, GC: 66
21:27:30.604Z : Producer : Topic: topic1, TC: 34, GC: 67
21:27:30.611Z : Producer : Topic: topic2, TC: 34, GC: 68
21:27:30.618Z : Producer : Topic: topic1, TC: 35, GC: 69
21:27:30.624Z : Producer : Topic: topic2, TC: 35, GC: 70
21:27:30.631Z : Producer : Topic: topic1, TC: 36, GC: 71
21:27:30.637Z : Producer : Topic: topic2, TC: 36, GC: 72
21:27:30.644Z : Producer : Topic: topic1, TC: 37, GC: 73
21:27:30.651Z : Producer : Topic: topic2, TC: 37, GC: 74
21:27:30.658Z : Producer : Topic: topic1, TC: 38, GC: 75
21:27:30.665Z : Producer : Topic: topic2, TC: 38, GC: 76
21:27:30.672Z : Producer : Topic: topic1, TC: 39, GC: 77
21:27:30.679Z : Producer : Topic: topic2, TC: 39, GC: 78
21:27:30.686Z : Producer : Topic: topic1, TC: 40, GC: 79
21:27:30.693Z : Producer : Topic: topic2, TC: 40, GC: 80
21:27:30.700Z : Producer : Topic: topic1, TC: 41, GC: 81
21:27:30.706Z : Producer : Topic: topic2, TC: 41, GC: 82
21:27:30.713Z : Producer : Topic: topic1, TC: 42, GC: 83
21:27:30.721Z : Producer : Topic: topic2, TC: 42, GC: 84
21:27:30.727Z : Producer : Topic: topic1, TC: 43, GC: 85
21:27:30.733Z : Producer : Topic: topic2, TC: 43, GC: 86
21:27:30.739Z : Producer : Topic: topic1, TC: 44, GC: 87
21:27:30.744Z : Producer : Topic: topic2, TC: 44, GC: 88
21:27:30.750Z : Producer : Topic: topic1, TC: 45, GC: 89
21:27:30.756Z : Producer : Topic: topic2, TC: 45, GC: 90
21:27:30.762Z : Producer : Topic: topic1, TC: 46, GC: 91
21:27:30.769Z : Producer : Topic: topic2, TC: 46, GC: 92
21:27:30.774Z : Producer : Topic: topic1, TC: 47, GC: 93
21:27:30.780Z : Producer : Topic: topic2, TC: 47, GC: 94
21:27:30.787Z : Producer : Topic: topic1, TC: 48, GC: 95
21:27:30.793Z : Producer : Topic: topic2, TC: 48, GC: 96
21:27:30.799Z : Producer : Topic: topic1, TC: 49, GC: 97
21:27:30.805Z : Producer : Topic: topic2, TC: 49, GC: 98
21:27:30.811Z : Producer : Topic: topic1, TC: 50, GC: 99
21:27:30.816Z : Producer : Topic: topic2, TC: 50, GC: 100
21:27:30.830Z : Consumer : Subscribing
21:27:30.937Z : Consumer : Consuming
21:27:30.961Z : Consumer : Topic: topic2, TC: 1 OK, GC: 2 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 2 OK, GC: 4 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 3 OK, GC: 6 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 4 OK, GC: 8 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 5 OK, GC: 10 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 6 OK, GC: 12 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 7 OK, GC: 14 JUMP FORWARD 1
21:27:30.966Z : Consumer : Topic: topic2, TC: 8 OK, GC: 16 JUMP FORWARD 1
21:27:30.967Z : Consumer : Topic: topic2, TC: 9 OK, GC: 18 JUMP FORWARD 1
21:27:30.968Z : Consumer : Topic: topic2, TC: 10 OK, GC: 20 JUMP FORWARD 1
21:27:30.968Z : Consumer : Topic: topic2, TC: 11 OK, GC: 22 JUMP FORWARD 1
21:27:30.968Z : Consumer : Topic: topic2, TC: 12 OK, GC: 24 JUMP FORWARD 1
21:27:30.968Z : Consumer : Topic: topic2, TC: 13 OK, GC: 26 JUMP FORWARD 1
21:27:30.968Z : Consumer : Topic: topic2, TC: 14 OK, GC: 28 JUMP FORWARD 1
21:27:30.968Z : Consumer : Topic: topic2, TC: 15 OK, GC: 30 JUMP FORWARD 1
21:27:30.969Z : Consumer : Topic: topic2, TC: 16 OK, GC: 32 JUMP FORWARD 1
21:27:30.969Z : Consumer : Topic: topic2, TC: 17 OK, GC: 34 JUMP FORWARD 1
21:27:30.969Z : Consumer : Topic: topic2, TC: 18 OK, GC: 36 JUMP FORWARD 1
21:27:30.969Z : Consumer : Topic: topic2, TC: 19 OK, GC: 38 JUMP FORWARD 1
21:27:30.970Z : Consumer : Topic: topic2, TC: 20 OK, GC: 40 JUMP FORWARD 1
21:27:30.970Z : Consumer : Topic: topic2, TC: 21 OK, GC: 42 JUMP FORWARD 1
21:27:30.970Z : Consumer : Topic: topic2, TC: 22 OK, GC: 44 JUMP FORWARD 1
21:27:30.971Z : Consumer : Topic: topic2, TC: 23 OK, GC: 46 JUMP FORWARD 1
21:27:30.971Z : Consumer : Topic: topic2, TC: 24 OK, GC: 48 JUMP FORWARD 1
21:27:30.971Z : Consumer : Topic: topic2, TC: 25 OK, GC: 50 JUMP FORWARD 1
21:27:30.971Z : Consumer : Topic: topic2, TC: 26 OK, GC: 52 JUMP FORWARD 1
21:27:30.971Z : Consumer : Topic: topic2, TC: 27 OK, GC: 54 JUMP FORWARD 1
21:27:30.971Z : Consumer : Topic: topic2, TC: 28 OK, GC: 56 JUMP FORWARD 1
21:27:30.972Z : Consumer : Topic: topic2, TC: 29 OK, GC: 58 JUMP FORWARD 1
21:27:30.972Z : Consumer : Topic: topic2, TC: 30 OK, GC: 60 JUMP FORWARD 1
21:27:30.972Z : Consumer : Topic: topic2, TC: 31 OK, GC: 62 JUMP FORWARD 1
21:27:30.972Z : Consumer : Topic: topic2, TC: 32 OK, GC: 64 JUMP FORWARD 1
21:27:30.972Z : Consumer : Topic: topic2, TC: 33 OK, GC: 66 JUMP FORWARD 1
21:27:30.972Z : Consumer : Topic: topic2, TC: 34 OK, GC: 68 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 35 OK, GC: 70 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 36 OK, GC: 72 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 37 OK, GC: 74 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 38 OK, GC: 76 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 39 OK, GC: 78 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 40 OK, GC: 80 JUMP FORWARD 1
21:27:30.973Z : Consumer : Topic: topic2, TC: 41 OK, GC: 82 JUMP FORWARD 1
21:27:30.974Z : Consumer : Topic: topic2, TC: 42 OK, GC: 84 JUMP FORWARD 1
21:27:30.974Z : Consumer : Topic: topic2, TC: 43 OK, GC: 86 JUMP FORWARD 1
21:27:30.974Z : Consumer : Topic: topic2, TC: 44 OK, GC: 88 JUMP FORWARD 1
21:27:30.974Z : Consumer : Topic: topic2, TC: 45 OK, GC: 90 JUMP FORWARD 1
21:27:30.975Z : Consumer : Topic: topic2, TC: 46 OK, GC: 92 JUMP FORWARD 1
21:27:30.975Z : Consumer : Topic: topic2, TC: 47 OK, GC: 94 JUMP FORWARD 1
21:27:30.975Z : Consumer : Topic: topic2, TC: 48 OK, GC: 96 JUMP FORWARD 1
21:27:30.975Z : Consumer : Topic: topic2, TC: 49 OK, GC: 98 JUMP FORWARD 1
21:27:30.975Z : Consumer : Topic: topic2, TC: 50 OK, GC: 100 JUMP FORWARD 1
21:27:30.975Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 JUMP BACKWARDS 99
21:27:30.975Z : Consumer : Topic: topic1, TC: 2 OK, GC: 3 JUMP FORWARD 1
21:27:30.976Z : Consumer : Topic: topic1, TC: 3 OK, GC: 5 JUMP FORWARD 1
21:27:30.976Z : Consumer : Topic: topic1, TC: 4 OK, GC: 7 JUMP FORWARD 1
21:27:30.976Z : Consumer : Topic: topic1, TC: 5 OK, GC: 9 JUMP FORWARD 1
21:27:30.977Z : Consumer : Topic: topic1, TC: 6 OK, GC: 11 JUMP FORWARD 1
21:27:30.977Z : Consumer : Topic: topic1, TC: 7 OK, GC: 13 JUMP FORWARD 1
21:27:30.977Z : Consumer : Topic: topic1, TC: 8 OK, GC: 15 JUMP FORWARD 1
21:27:30.977Z : Consumer : Topic: topic1, TC: 9 OK, GC: 17 JUMP FORWARD 1
21:27:30.977Z : Consumer : Topic: topic1, TC: 10 OK, GC: 19 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 11 OK, GC: 21 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 12 OK, GC: 23 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 13 OK, GC: 25 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 14 OK, GC: 27 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 15 OK, GC: 29 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 16 OK, GC: 31 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 17 OK, GC: 33 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 18 OK, GC: 35 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 19 OK, GC: 37 JUMP FORWARD 1
21:27:30.978Z : Consumer : Topic: topic1, TC: 20 OK, GC: 39 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 21 OK, GC: 41 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 22 OK, GC: 43 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 23 OK, GC: 45 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 24 OK, GC: 47 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 25 OK, GC: 49 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 26 OK, GC: 51 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 27 OK, GC: 53 JUMP FORWARD 1
21:27:30.979Z : Consumer : Topic: topic1, TC: 28 OK, GC: 55 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 29 OK, GC: 57 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 30 OK, GC: 59 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 31 OK, GC: 61 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 32 OK, GC: 63 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 33 OK, GC: 65 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 34 OK, GC: 67 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 35 OK, GC: 69 JUMP FORWARD 1
21:27:30.980Z : Consumer : Topic: topic1, TC: 36 OK, GC: 71 JUMP FORWARD 1
21:27:30.981Z : Consumer : Topic: topic1, TC: 37 OK, GC: 73 JUMP FORWARD 1
21:27:30.981Z : Consumer : Topic: topic1, TC: 38 OK, GC: 75 JUMP FORWARD 1
21:27:30.981Z : Consumer : Topic: topic1, TC: 39 OK, GC: 77 JUMP FORWARD 1
21:27:30.981Z : Consumer : Topic: topic1, TC: 40 OK, GC: 79 JUMP FORWARD 1
21:27:30.981Z : Consumer : Topic: topic1, TC: 41 OK, GC: 81 JUMP FORWARD 1
21:27:30.982Z : Consumer : Topic: topic1, TC: 42 OK, GC: 83 JUMP FORWARD 1
21:27:30.982Z : Consumer : Topic: topic1, TC: 43 OK, GC: 85 JUMP FORWARD 1
21:27:30.982Z : Consumer : Topic: topic1, TC: 44 OK, GC: 87 JUMP FORWARD 1
21:27:30.983Z : Consumer : Topic: topic1, TC: 45 OK, GC: 89 JUMP FORWARD 1
21:27:30.983Z : Consumer : Topic: topic1, TC: 46 OK, GC: 91 JUMP FORWARD 1
21:27:30.983Z : Consumer : Topic: topic1, TC: 47 OK, GC: 93 JUMP FORWARD 1
21:27:30.983Z : Consumer : Topic: topic1, TC: 48 OK, GC: 95 JUMP FORWARD 1
21:27:30.983Z : Consumer : Topic: topic1, TC: 49 OK, GC: 97 JUMP FORWARD 1
21:27:30.983Z : Consumer : Topic: topic1, TC: 50 OK, GC: 99 JUMP FORWARD 1

We saw that the consumer consumed all of topic2, then all of topic1. This is exactly like Kafka.

However…I ran this with 10000 messages instead of 100 and I saw that the consumer would process stretches of one topic, then shift to the other, then round-robin for a bit then long stretches of one topic etc. The jumps forwards and backwards in the global stream never exceeded 2000. This is better than Kafka, which would process an entire topic (single partition) first, even if it was huge, then the second one.

So if you find ordering between topics beneficial then depending on your specific situation you may be able to classify this as useful or not. If you have many thousands of messages per second, then it could be true that this gives good enough ordering. If you get a trickle, and related events are close together physically in the topic, then this might not be a useful level of ordering.

Case 10 - Time-travel, unequal message arrival rate

Same as Case 9 but with a 10 to 1 ratio on topic arrival rate.

21:36:16.014Z : Producer : Topic: topic1, TC: 1, GC: 1
21:36:16.024Z : Producer : Topic: topic1, TC: 2, GC: 2
21:36:16.031Z : Producer : Topic: topic1, TC: 3, GC: 3
21:36:16.038Z : Producer : Topic: topic1, TC: 4, GC: 4
21:36:16.045Z : Producer : Topic: topic1, TC: 5, GC: 5
21:36:16.052Z : Producer : Topic: topic1, TC: 6, GC: 6
21:36:16.058Z : Producer : Topic: topic1, TC: 7, GC: 7
21:36:16.066Z : Producer : Topic: topic1, TC: 8, GC: 8
21:36:16.073Z : Producer : Topic: topic1, TC: 9, GC: 9
21:36:16.080Z : Producer : Topic: topic1, TC: 10, GC: 10
21:36:16.088Z : Producer : Topic: topic2, TC: 1, GC: 11
21:36:16.095Z : Producer : Topic: topic1, TC: 11, GC: 12
21:36:16.101Z : Producer : Topic: topic1, TC: 12, GC: 13
21:36:16.107Z : Producer : Topic: topic1, TC: 13, GC: 14
21:36:16.115Z : Producer : Topic: topic1, TC: 14, GC: 15
21:36:16.121Z : Producer : Topic: topic1, TC: 15, GC: 16
21:36:16.129Z : Producer : Topic: topic1, TC: 16, GC: 17
21:36:16.135Z : Producer : Topic: topic1, TC: 17, GC: 18
21:36:16.141Z : Producer : Topic: topic1, TC: 18, GC: 19
21:36:16.148Z : Producer : Topic: topic1, TC: 19, GC: 20
21:36:16.155Z : Producer : Topic: topic1, TC: 20, GC: 21
21:36:16.162Z : Producer : Topic: topic2, TC: 2, GC: 22
21:36:16.168Z : Producer : Topic: topic1, TC: 21, GC: 23
21:36:16.174Z : Producer : Topic: topic1, TC: 22, GC: 24
21:36:16.181Z : Producer : Topic: topic1, TC: 23, GC: 25
21:36:16.188Z : Producer : Topic: topic1, TC: 24, GC: 26
21:36:16.193Z : Producer : Topic: topic1, TC: 25, GC: 27
21:36:16.200Z : Producer : Topic: topic1, TC: 26, GC: 28
21:36:16.206Z : Producer : Topic: topic1, TC: 27, GC: 29
21:36:16.212Z : Producer : Topic: topic1, TC: 28, GC: 30
21:36:16.220Z : Producer : Topic: topic1, TC: 29, GC: 31
21:36:16.226Z : Producer : Topic: topic1, TC: 30, GC: 32
21:36:16.233Z : Producer : Topic: topic2, TC: 3, GC: 33
21:36:16.239Z : Producer : Topic: topic1, TC: 31, GC: 34
21:36:16.246Z : Producer : Topic: topic1, TC: 32, GC: 35
21:36:16.252Z : Producer : Topic: topic1, TC: 33, GC: 36
21:36:16.259Z : Producer : Topic: topic1, TC: 34, GC: 37
21:36:16.266Z : Producer : Topic: topic1, TC: 35, GC: 38
21:36:16.272Z : Producer : Topic: topic1, TC: 36, GC: 39
21:36:16.279Z : Producer : Topic: topic1, TC: 37, GC: 40
21:36:16.285Z : Producer : Topic: topic1, TC: 38, GC: 41
21:36:16.292Z : Producer : Topic: topic1, TC: 39, GC: 42
21:36:16.299Z : Producer : Topic: topic1, TC: 40, GC: 43
21:36:16.306Z : Producer : Topic: topic2, TC: 4, GC: 44
21:36:16.312Z : Producer : Topic: topic1, TC: 41, GC: 45
21:36:16.318Z : Producer : Topic: topic1, TC: 42, GC: 46
21:36:16.326Z : Producer : Topic: topic1, TC: 43, GC: 47
21:36:16.333Z : Producer : Topic: topic1, TC: 44, GC: 48
21:36:16.339Z : Producer : Topic: topic1, TC: 45, GC: 49
21:36:16.345Z : Producer : Topic: topic1, TC: 46, GC: 50
21:36:16.351Z : Producer : Topic: topic1, TC: 47, GC: 51
21:36:16.357Z : Producer : Topic: topic1, TC: 48, GC: 52
21:36:16.364Z : Producer : Topic: topic1, TC: 49, GC: 53
21:36:16.371Z : Producer : Topic: topic1, TC: 50, GC: 54
21:36:16.377Z : Producer : Topic: topic2, TC: 5, GC: 55
21:36:16.383Z : Producer : Topic: topic1, TC: 51, GC: 56
21:36:16.391Z : Producer : Topic: topic1, TC: 52, GC: 57
21:36:16.398Z : Producer : Topic: topic1, TC: 53, GC: 58
21:36:16.404Z : Producer : Topic: topic1, TC: 54, GC: 59
21:36:16.411Z : Producer : Topic: topic1, TC: 55, GC: 60
21:36:16.418Z : Producer : Topic: topic1, TC: 56, GC: 61
21:36:16.425Z : Producer : Topic: topic1, TC: 57, GC: 62
21:36:16.432Z : Producer : Topic: topic1, TC: 58, GC: 63
21:36:16.439Z : Producer : Topic: topic1, TC: 59, GC: 64
21:36:16.446Z : Producer : Topic: topic1, TC: 60, GC: 65
21:36:16.452Z : Producer : Topic: topic2, TC: 6, GC: 66
21:36:16.458Z : Producer : Topic: topic1, TC: 61, GC: 67
21:36:16.465Z : Producer : Topic: topic1, TC: 62, GC: 68
21:36:16.471Z : Producer : Topic: topic1, TC: 63, GC: 69
21:36:16.478Z : Producer : Topic: topic1, TC: 64, GC: 70
21:36:16.483Z : Producer : Topic: topic1, TC: 65, GC: 71
21:36:16.489Z : Producer : Topic: topic1, TC: 66, GC: 72
21:36:16.496Z : Producer : Topic: topic1, TC: 67, GC: 73
21:36:16.502Z : Producer : Topic: topic1, TC: 68, GC: 74
21:36:16.508Z : Producer : Topic: topic1, TC: 69, GC: 75
21:36:16.515Z : Producer : Topic: topic1, TC: 70, GC: 76
21:36:16.522Z : Producer : Topic: topic2, TC: 7, GC: 77
21:36:16.528Z : Producer : Topic: topic1, TC: 71, GC: 78
21:36:16.535Z : Producer : Topic: topic1, TC: 72, GC: 79
21:36:16.542Z : Producer : Topic: topic1, TC: 73, GC: 80
21:36:16.548Z : Producer : Topic: topic1, TC: 74, GC: 81
21:36:16.553Z : Producer : Topic: topic1, TC: 75, GC: 82
21:36:16.559Z : Producer : Topic: topic1, TC: 76, GC: 83
21:36:16.565Z : Producer : Topic: topic1, TC: 77, GC: 84
21:36:16.570Z : Producer : Topic: topic1, TC: 78, GC: 85
21:36:16.576Z : Producer : Topic: topic1, TC: 79, GC: 86
21:36:16.582Z : Producer : Topic: topic1, TC: 80, GC: 87
21:36:16.589Z : Producer : Topic: topic2, TC: 8, GC: 88
21:36:16.596Z : Producer : Topic: topic1, TC: 81, GC: 89
21:36:16.602Z : Producer : Topic: topic1, TC: 82, GC: 90
21:36:16.609Z : Producer : Topic: topic1, TC: 83, GC: 91
21:36:16.615Z : Producer : Topic: topic1, TC: 84, GC: 92
21:36:16.621Z : Producer : Topic: topic1, TC: 85, GC: 93
21:36:16.627Z : Producer : Topic: topic1, TC: 86, GC: 94
21:36:16.634Z : Producer : Topic: topic1, TC: 87, GC: 95
21:36:16.640Z : Producer : Topic: topic1, TC: 88, GC: 96
21:36:16.647Z : Producer : Topic: topic1, TC: 89, GC: 97
21:36:16.652Z : Producer : Topic: topic1, TC: 90, GC: 98
21:36:16.659Z : Producer : Topic: topic2, TC: 9, GC: 99
21:36:16.666Z : Producer : Topic: topic1, TC: 91, GC: 100
21:36:16.679Z : Consumer : Subscribing
21:36:16.792Z : Consumer : Consuming
21:36:16.800Z : Consumer : Topic: topic2, TC: 1 OK, GC: 11 JUMP FORWARD 10
21:36:16.801Z : Consumer : Topic: topic2, TC: 2 OK, GC: 22 JUMP FORWARD 10
21:36:16.801Z : Consumer : Topic: topic2, TC: 3 OK, GC: 33 JUMP FORWARD 10
21:36:16.801Z : Consumer : Topic: topic2, TC: 4 OK, GC: 44 JUMP FORWARD 10
21:36:16.802Z : Consumer : Topic: topic2, TC: 5 OK, GC: 55 JUMP FORWARD 10
21:36:16.803Z : Consumer : Topic: topic2, TC: 6 OK, GC: 66 JUMP FORWARD 10
21:36:16.803Z : Consumer : Topic: topic2, TC: 7 OK, GC: 77 JUMP FORWARD 10
21:36:16.803Z : Consumer : Topic: topic2, TC: 8 OK, GC: 88 JUMP FORWARD 10
21:36:16.804Z : Consumer : Topic: topic2, TC: 9 OK, GC: 99 JUMP FORWARD 10
21:36:16.804Z : Consumer : Topic: topic1, TC: 1 OK, GC: 1 JUMP BACKWARDS 98
21:36:16.813Z : Consumer : Topic: topic1, TC: 2 OK, GC: 2 OK
21:36:16.813Z : Consumer : Topic: topic1, TC: 3 OK, GC: 3 OK
21:36:16.813Z : Consumer : Topic: topic1, TC: 4 OK, GC: 4 OK
21:36:16.814Z : Consumer : Topic: topic1, TC: 5 OK, GC: 5 OK
21:36:16.814Z : Consumer : Topic: topic1, TC: 6 OK, GC: 6 OK
21:36:16.814Z : Consumer : Topic: topic1, TC: 7 OK, GC: 7 OK
21:36:16.814Z : Consumer : Topic: topic1, TC: 8 OK, GC: 8 OK
21:36:16.814Z : Consumer : Topic: topic1, TC: 9 OK, GC: 9 OK
21:36:16.814Z : Consumer : Topic: topic1, TC: 10 OK, GC: 10 OK
21:36:16.815Z : Consumer : Topic: topic1, TC: 11 OK, GC: 12 JUMP FORWARD 1
21:36:16.816Z : Consumer : Topic: topic1, TC: 12 OK, GC: 13 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 13 OK, GC: 14 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 14 OK, GC: 15 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 15 OK, GC: 16 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 16 OK, GC: 17 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 17 OK, GC: 18 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 18 OK, GC: 19 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 19 OK, GC: 20 OK
21:36:16.817Z : Consumer : Topic: topic1, TC: 20 OK, GC: 21 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 21 OK, GC: 23 JUMP FORWARD 1
21:36:16.818Z : Consumer : Topic: topic1, TC: 22 OK, GC: 24 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 23 OK, GC: 25 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 24 OK, GC: 26 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 25 OK, GC: 27 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 26 OK, GC: 28 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 27 OK, GC: 29 OK
21:36:16.818Z : Consumer : Topic: topic1, TC: 28 OK, GC: 30 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 29 OK, GC: 31 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 30 OK, GC: 32 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 31 OK, GC: 34 JUMP FORWARD 1
21:36:16.819Z : Consumer : Topic: topic1, TC: 32 OK, GC: 35 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 33 OK, GC: 36 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 34 OK, GC: 37 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 35 OK, GC: 38 OK
21:36:16.819Z : Consumer : Topic: topic1, TC: 36 OK, GC: 39 OK
21:36:16.820Z : Consumer : Topic: topic1, TC: 37 OK, GC: 40 OK
21:36:16.820Z : Consumer : Topic: topic1, TC: 38 OK, GC: 41 OK
21:36:16.820Z : Consumer : Topic: topic1, TC: 39 OK, GC: 42 OK
21:36:16.820Z : Consumer : Topic: topic1, TC: 40 OK, GC: 43 OK
21:36:16.820Z : Consumer : Topic: topic1, TC: 41 OK, GC: 45 JUMP FORWARD 1
21:36:16.820Z : Consumer : Topic: topic1, TC: 42 OK, GC: 46 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 43 OK, GC: 47 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 44 OK, GC: 48 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 45 OK, GC: 49 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 46 OK, GC: 50 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 47 OK, GC: 51 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 48 OK, GC: 52 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 49 OK, GC: 53 OK
21:36:16.821Z : Consumer : Topic: topic1, TC: 50 OK, GC: 54 OK
21:36:16.822Z : Consumer : Topic: topic1, TC: 51 OK, GC: 56 JUMP FORWARD 1
21:36:16.822Z : Consumer : Topic: topic1, TC: 52 OK, GC: 57 OK
21:36:16.822Z : Consumer : Topic: topic1, TC: 53 OK, GC: 58 OK
21:36:16.822Z : Consumer : Topic: topic1, TC: 54 OK, GC: 59 OK
21:36:16.822Z : Consumer : Topic: topic1, TC: 55 OK, GC: 60 OK
21:36:16.823Z : Consumer : Topic: topic1, TC: 56 OK, GC: 61 OK
21:36:16.823Z : Consumer : Topic: topic1, TC: 57 OK, GC: 62 OK
21:36:16.823Z : Consumer : Topic: topic1, TC: 58 OK, GC: 63 OK
21:36:16.823Z : Consumer : Topic: topic1, TC: 59 OK, GC: 64 OK
21:36:16.823Z : Consumer : Topic: topic1, TC: 60 OK, GC: 65 OK
21:36:16.823Z : Consumer : Topic: topic1, TC: 61 OK, GC: 67 JUMP FORWARD 1
21:36:16.824Z : Consumer : Topic: topic1, TC: 62 OK, GC: 68 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 63 OK, GC: 69 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 64 OK, GC: 70 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 65 OK, GC: 71 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 66 OK, GC: 72 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 67 OK, GC: 73 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 68 OK, GC: 74 OK
21:36:16.824Z : Consumer : Topic: topic1, TC: 69 OK, GC: 75 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 70 OK, GC: 76 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 71 OK, GC: 78 JUMP FORWARD 1
21:36:16.825Z : Consumer : Topic: topic1, TC: 72 OK, GC: 79 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 73 OK, GC: 80 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 74 OK, GC: 81 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 75 OK, GC: 82 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 76 OK, GC: 83 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 77 OK, GC: 84 OK
21:36:16.825Z : Consumer : Topic: topic1, TC: 78 OK, GC: 85 OK
21:36:16.826Z : Consumer : Topic: topic1, TC: 79 OK, GC: 86 OK
21:36:16.826Z : Consumer : Topic: topic1, TC: 80 OK, GC: 87 OK
21:36:16.826Z : Consumer : Topic: topic1, TC: 81 OK, GC: 89 JUMP FORWARD 1
21:36:16.826Z : Consumer : Topic: topic1, TC: 82 OK, GC: 90 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 83 OK, GC: 91 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 84 OK, GC: 92 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 85 OK, GC: 93 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 86 OK, GC: 94 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 87 OK, GC: 95 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 88 OK, GC: 96 OK
21:36:16.827Z : Consumer : Topic: topic1, TC: 89 OK, GC: 97 OK
21:36:16.828Z : Consumer : Topic: topic1, TC: 90 OK, GC: 98 OK
21:36:16.828Z : Consumer : Topic: topic1, TC: 91 OK, GC: 100 JUMP FORWARD 1

No difference, it processed all of topic2, then topic 1.

Again, I ran this for 10000 messages and didn’t see leaps of more than 2000 messages in the global order.

Conclusions

What we’ve seen is that if you have a fast consumer that can consume messages in strict real-time, then Pulsar can deliver you your messages in order, across both topics. Don’t bet your house on that, there are no guarantees.

In this case I am defining strict real-time to be that the receive buffer (defaults to 1000 messages) doesn’t get totally full meaning that even when the application code lags a little, the client itself is able to be totally up to date, leveraging the buffer until the application code catches up.

However, once we lose real-time consumption, cross topic ordering is lost.

This leaves us the question: If I can achieve cross topic ordering but have no guarantees of that, should I leverage it?
If you want your application to be 100% consistent then no. If best-effort is enough for you, then perhaps you might explore it further. It is just like the problem with distributed systems that choose consistency under normal operations but choose availability when experiencing a problem such as a network partition - the application developer still needs to handle the inconsistencies of the data service in question, despite its consistency guarantees under normal conditions.

Alternatives to Multi-Topic Subscriptions When Ordering Is Desirable

Ordering may be desirable because the topics are related to each other. There are alternatives, review your options!

If you have related events that you want ordering for, then send them into the same topic. This will provide a 100% ordering guarantee. Let consumers filter out the ones they are not interested in. I don’t know what support Pulsar provides for multiple event types in a single topic when using their schema registry. An interesting link on this, regarding Kafka, is: https://www.confluent.io/blog/put-several-event-types-kafka-topic/
Check out my best-effort cross topic ordering solution under the next heading
Build your system in a way that is agnostic to the ordering of related events (could be hard)

Remove referential integrity in the target system
Try to use monotonicity: https://blog.acolyer.org/2019/03/06/keeping-calm-when-distributed-consistency-is-easy/
Please suggest patterns to me on Twitter!

The downside of mixing events in a single topic is that it is a decision that affects all consuming applications. A topic is shared infrastructure and mixing events becomes an architectural decision. This is not something that affects RabbitMQ as each application can bind a single queue to multiple exchanges and get custom mixing without impacting other applications, which is one killer feature of RabbitMQ. So think about RabbitMQ if you find it hard to model your events in Kafka or Pulsar.

A Best-Effort Strategy Based on Publisher Timestamps

We are now officially in the danger zone, ordering based on publisher timestamps. I include this as much to put you off the idea and as because it does achieve decent ordering if you don’t suffer any major clock skew… ever (I know).

The basic idea is for each topic you want to consume, create a consumer for it and put it in its own thread. Let it keep its internal buffer filled with messages and let your main thread peek at the first message and only process it if it has the lowest producer set timestamp. In Kafka we can use the LogAppendTime which is set by the broker, so less dangerous, though still dangerous.

So what can possibly wrong? Well, one producer might have a clock that is behind the others (like 1 hour or 1 year). That will cause those messages to be preferentially taken throwing off ordering quite a bit and could even block the consumption of other topics completely, if it is a fast producer. This is actually the same problem with priority queues, higher priority messages cause lower priority ones to get stuck under high load.

But if you control all your producers (i.e. you have control over their clocks), and don’t tend to get some producers who are super fast, then perhaps this is still a strategy. My test code achieves perfect ordering for both real-time and time-travel scenarios when clocks are perfectly synchronized. Again, don’t shout at me on Twitter about clock skew please, warnings have been clearly provided!

Last Words

So there you have it, useful or not I don’t know, but at least we have learned something about Pulsar today and what you can expect from multi-topic subscriptions. You can see the Apache Kafka analysis on the CloudKarafka blog.

Code used to generate these results.

Banner image credit: ESO