In the last post we built a TPL Dataflow pipeline based on the scenario from our first post in the series. Today we'll build another pipeline very similar to the first but with different requirements around latency and data loss.

In the first scenario we could not slow down the producer as slowing it down would cause data loss (it read from a bus that would not wait if you weren't there to consume the data). We also cared a lot about ensuring the first stage kept up with the producer and successfully wrote every message to disk. The rest was best effort, and we performed load-shedding so as not to slow down the producer.

TPL Dataflow is a data processing library from Microsoft that came out years ago. It consists of different "blocks" that you compose together to make a pipeline. Blocks correspond to stages in your pipeline. If you didn't read the first post in the series then that might not be a bad idea before you read on.

In this series we'll look at few different .NET technologies we can use to process streams of data in processing pipelines and directed acyclic graphs (DAGs). This is not about distributed data platforms for big data but real-time processing and computation running on a single machine. We'll take a single scenario and build it out multiple times, each with a different technology. Each application will be built as a console application with .NET Core.