Event-Driven Architectures - The Queue vs The Log

A messaging system is at the heart of most event-driven architectures and there are a plethora of different technologies in the space and they can be classified as either queue based or log based.

Queue based: RabbitMQ, ActiveMQ, MSMQ, AWS SQS, JMQ and many more.

Log based: Apache Kakfa, Apache Pulsar, AWS Kinesis, Azure Event Hubs and many more.

Each messaging system has different features but at the heart are their data structures: queue or log. In this post we'll take a look at how the underlying data structure affects your event-driven architecture.

SQL Server CDC to Redshift Pipeline

In this post we'll take a look at what Change Data Capture (CDC) is and how we can use it to get data from SQL Server into Redshift in either a near real-time streaming fashion or more of a batched approach.

CDC is a SQL Server Enterprise feature and so not available to everyone. Also there are vendors that sell automated change data capture extraction and load into Redshift, such as Attunity and that may be your best option. But if you can't or don't want to pay for another tool on top of your SQL Server Enterprise license then this post may help you.

Processing Pipelines Series - Reactive Extensions (Rx.NET)

Whereas TPL Dataflow is all about passing messages between blocks, Reactive Extensions is about sequences. With sequences we can create projections, transformations and filters. We can combine multiple sequences into a single one. It is a very flexible and powerful paradigm but with such power comes extra complexity. I find TPL Dataflow easier to reason about due to its simple model. Reactive Extensions can get pretty complex and is not always intuitive, but you can create some elegant solutions with it. It will require some investment in time and tinkering to get a reasonable understanding of it.

Processing Pipelines Series - TPL Dataflow - Alternate Scenario

In the last post we built a TPL Dataflow pipeline based on the scenario from our first post in the series. Today we'll build another pipeline very similar to the first but with different requirements around latency and data loss.

In the first scenario we could not slow down the producer as slowing it down would cause data loss (it read from a bus that would not wait if you weren't there to consume the data). We also cared a lot about ensuring the first stage kept up with the producer and successfully wrote every message to disk. The rest was best effort, and we performed load-shedding so as not to slow down the producer.

Processing Pipelines Series - Concepts

In this series we'll look at few different .NET technologies we can use to process streams of data in processing pipelines and directed acyclic graphs (DAGs). This is not about distributed data platforms for big data but real-time processing and computation running on a single machine. We'll take a single scenario and build it out multiple times, each with a different technology. Each application will be built as a console application with .NET Core.

AgileTM vs Real Agility - The Constant Rush

Being in a rush is counter-productive. It is stressful for the people doing the work and the quality of the work suffers. In software development that translates into an ever increasing amount of spaghetti code and an architecture of quick fixes which then leads to a slow down. This slow down then leads to greater time pressure, and the rush gets worse.

Real deadlines do exist, and sometimes out of necessity a team has to pull out all the stops and make a short-lived dash. The big problem arises when this mad dash is the normal state of affairs.

AgileTM vs Real Agility - Truly Self-Organizing Teams

Empower people to act responsibly and they generally will. Exercise empathy and people will trust you. Give people the information they need and they will reciprocate. Agile development was an attempt at doing these things better, but it has been reduced to a collection of code words and empty rituals by managers who don’t realize that it is actually their philosophy about the purpose and nature of management itself that needs to change, and not merely their choice of techniques.

AgileTM vs Real Agility - The Fight Is On

This series is about agile. Not AgileTM, not Big A Agile, not the type sold in three day certificate programs, not the nice neat packaged set of rules Agile, not the type mandated from above, not the religious kind.

No... this is a series about real agility. An idea, a concept that is more than a set of rules and roles. This is my contribution to help my fellow software developers out there, struggling against the frankly terrible practices being carried out in our industry in the name of Agile. This is a series about taking the power back into the hands of the real experts, the software developers. That is where agile began after all.