Processing Pipelines Series - Reactive Extensions (Rx.NET)

Whereas TPL Dataflow is all about passing messages between blocks, Reactive Extensions is about sequences. With sequences we can create projections, transformations and filters. We can combine multiple sequences into a single one. It is a very flexible and powerful paradigm but with such power comes extra complexity. I find TPL Dataflow easier to reason about due to its simple model. Reactive Extensions can get pretty complex and is not always intuitive, but you can create some elegant solutions with it. It will require some investment in time and tinkering to get a reasonable understanding of it.

Processing Pipelines Series - TPL Dataflow - Alternate Scenario

In the last post we built a TPL Dataflow pipeline based on the scenario from our first post in the series. Today we'll build another pipeline very similar to the first but with different requirements around latency and data loss.

In the first scenario we could not slow down the producer as slowing it down would cause data loss (it read from a bus that would not wait if you weren't there to consume the data). We also cared a lot about ensuring the first stage kept up with the producer and successfully wrote every message to disk. The rest was best effort, and we performed load-shedding so as not to slow down the producer.

Processing Pipelines Series - Concepts

In this series we'll look at few different .NET technologies we can use to process streams of data in processing pipelines and directed acyclic graphs (DAGs). This is not about distributed data platforms for big data but real-time processing and computation running on a single machine. We'll take a single scenario and build it out multiple times, each with a different technology. Each application will be built as a console application with .NET Core.

AgileTM vs Real Agility - The Constant Rush

Being in a rush is counter-productive. It is stressful for the people doing the work and the quality of the work suffers. In software development that translates into an ever increasing amount of spaghetti code and an architecture of quick fixes which then leads to a slow down. This slow down then leads to greater time pressure, and the rush gets worse.

Real deadlines do exist, and sometimes out of necessity a team has to pull out all the stops and make a short-lived dash. The big problem arises when this mad dash is the normal state of affairs.

AgileTM vs Real Agility - Truly Self-Organizing Teams

Empower people to act responsibly and they generally will. Exercise empathy and people will trust you. Give people the information they need and they will reciprocate. Agile development was an attempt at doing these things better, but it has been reduced to a collection of code words and empty rituals by managers who don’t realize that it is actually their philosophy about the purpose and nature of management itself that needs to change, and not merely their choice of techniques.

AgileTM vs Real Agility - The Fight Is On

This series is about agile. Not AgileTM, not Big A Agile, not the type sold in three day certificate programs, not the nice neat packaged set of rules Agile, not the type mandated from above, not the religious kind.

No... this is a series about real agility. An idea, a concept that is more than a set of rules and roles. This is my contribution to help my fellow software developers out there, struggling against the frankly terrible practices being carried out in our industry in the name of Agile. This is a series about taking the power back into the hands of the real experts, the software developers. That is where agile began after all.

RabbitMQ vs Kafka Part 4 - Message Delivery Semantics and Guarantees

Both RabbitMQ and Kafka offer durable messaging guarantees. Both offer at-most-once and at-least-once guarantees but kafka offers exactly-once guarantees in a very limited scenario.

Let's first understand what these guarantees mean:

  • At-most-once delivery. This means that a message will never be delivered more than once but messages might be lost.
  • At-least-once delivery. This means that we'll never lose a message but a message might end up being delivered to a consumer more than once.
  • Exactly-once delivery. The holy grail of messaging. All messages will be delivered exactly one time.

Delivery is probably the wrong word for the above terms, instead Processing might be a better way of putting it. After all what we care about is whether a consumer can process a message and whether that is at-most-once, at-least-once or exactly-once. But using the word processing complicates things, exactly-once delivery makes less sense now as perhaps we need it to be delivered twice in order to be able to successfully process it once. If the consumer dies during processing, then we need that the message be delivered a second time for a new consumer.

RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns

In Part 2 we covered the patterns and topologies that RabbitMQ enables. In this part we'll look at Kafka and contrast it against RabbitMQ to get some perspective on their differences. Remember that this comparison is within the context of an event-driven application architecture rather than data processing pipelines, although the line between them can be a bit grey. Perhaps it is more like a continuum and this comparison focuses on the event-driven applications end of that continuum.