Constraints breed innovation and so does tenacity

A number of years ago, I got a bit addicted to coding challenges on HackerRank. It was kind of intense, fun learning the algorithms, frustrating seeing my early attempts crash and burn but ultimately triumphant when I finally got the thing to run in under N number of seconds that was demanded.

Something that has always stuck with me from those days was how easy it would have been to settle if I hadn’t known it was possible. In real life, on each iteration, as I slowly improved the running time, I could have settled. But in a coding challenge, if the target was 4 seconds then I knew it was possible, and even though it seemed impossible after my initial attempts, I carried on. I carried on despite frustration and exasperation as my attempts continued to fail.

Another example of this was the 1 Billion Row Challenge. You could see other people’s results, so you knew you could do better.

In the real world it's so easy to settle. You write an algorithm, come up with an architecture, design a protocol and so on; it works and has reasonable performance, reasonable properties. In the real world sometimes that's enough and you move on. But sometimes perhaps it’s worth striving and not settling for your early ideas. What if, instead of accepting your design you ask yourself for something better, may be something out of left field? May be add a constraint that if you could pull off would be amazing. May be it requires some extra reading, like when I would dip into my various algorithms books for another way, another strategy.

It’s something I seriously think about whenever it comes to software design and implementation. It’s a sort of ongoing epiphany that surfaces anytime I need to design something - remember HackerRank. I ask, well that’s good but what if you had to come up with something better than this?

Table format comparisons - Streaming ingest of row-level operations

Table format comparisons - Streaming ingest of row-level operations

In the previous post, I covered append-only tables, a common table type in analytics used often for ingesting data into a data lake or modeling streams between stream processor jobs. I had promised to cover native support for changelog streams, aka change data capture (CDC), but before I do so, I think we should first look at how the table formats support the ingestion of data with row-level operations (insert, update, delete) rather than query-level operations that are commonly used in SQL batch commands. 

Table format comparisons - Append-only tables and incremental reads

Table format comparisons - Append-only tables and incremental reads

This post is about how the table formats support append-only tables and incremental reads. Streaming is becoming more and more important in the data analytics stack and the table formats all have some degree of support for streaming. One of the pillars of a streaming workload based on table formats is the append-only table. There are other pillars, such as changelog streams, and I’ll cover those in another post.

Incremental reads allow compute engines to perform repeated queries that return new records or changes to records that have occurred since the last query was executed. Basically, a table client polls the table on an interval, receiving the latest data on each occasion. Much like a Kafka consumer, albeit with a lot more end-to-end latency.

Table format comparisons - How do the table formats represent the canonical set of files?

Table format comparisons - How do the table formats represent the canonical set of files?

This is the first in a series of short comparisons of table format internals. While I have written in some detail about each, I think it’s interesting to look at what is the same or similar and what sets them apart from each other.

Question: How do the table formats represent the canonical list of data and delete files?

All the table formats store references to a canonical set of data and delete files within a set of metadata files. Each table format takes a slightly different approach but I’ll classify them into two categories:

  • The log of deltas approach (Hudi and Delta Lake)

  • The log of snapshots approach (Iceberg and Paimon)

,

A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems

,
A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems

Is it economical to build fault-tolerant transactional data systems directly on S3 Express One Zone, instead of using replication? Read on for an analysis.

Cloud object storage is becoming the universal storage layer for a wealth of cloud data systems. Some systems use object stores as the only storage layer, accepting the higher latency of object storage, and these tend to be analytics systems that can accept multi-second latencies. Transactional systems want single-digit millisecond latencies or latencies in the low tens of milliseconds and therefore don’t write to object stores directly. Instead, they land data on a fast replicated write-ahead-log (WAL) and offload data to an object store for read-optimized long-term, economical storage. Neon is a good example of this architecture. Writes hit a low-latency replicated write-ahead-log based on Multi-Paxos and data is eventually written to object storage.

Learning and reviewing system internals: tactics and psychology

Learning and reviewing system internals: tactics and psychology

Every now and then I get asked for advice on how to learn about distributed system internals and protocols. Over the course of my career I've picked up a learning and reviewing style that works pretty well for me.

To define these terms, learning and reviewing are similar but not the same:

  • Learning about how a system works is the easier of the two. By the means available to you (books, papers, blogs, code), you study the system to understand how it works and why it works that way.

  • Reviewing a system requires learning but also involves opinions, taking positions, making judgments. It is trickier to get right, more subjective, and often only time can show you if you were right or wrong about it and to what degree.

We all review systems to one degree or another, even if it's just a casual review where the results are some loosely held opinions shared by the coffee machine. But when it comes to sharing our opinions in more formal contexts, an architecture meeting, a blog post, a conference talk or a job interview, the stakes are higher and the risks are also greater. If you review a system and come to some conclusions, how do you know if you are right? What happens if you are wrong? Someone could point out your flawed arguments. You make a bad decision. Not only can reviewing complex systems be hard, it can be scary too.

Hybrid Transactional/Analytical Storage

Hybrid Transactional/Analytical Storage

Confluent has made two key feature announcements in the spring of 2024:

  • Freight Clusters, a new cluster type that writes directly to object storage. It is aimed at the “freight” of data streaming workloads, log ingestion, clickstreams, large-scale ETL and so on that can be cost-prohibitive using a low latency multi-AZ replication architecture in the cloud.

  • Tableflow, an automated feature that provides seamless materialization of Kafka topics as Apache Iceberg tables (and vice-versa in the future). 

This trend towards object storage is not just happening at Confluent but across the data ecosystem.

Tableflow: the stream/table, Kafka/Iceberg duality

Tableflow: the stream/table, Kafka/Iceberg duality

Confluent just announced Tableflow, the seamless materialization of Apache Kafka topics as Apache Iceberg tables. This announcement has to be the most impactful announcement I’ve witnessed while at Confluent. This post is about why Iceberg tables aren’t just another destination to sync data to; they fundamentally change the world of streaming. It’s also about the macro trends that have led us to this point and why Iceberg (and the other table formats) are so important to the future of streaming.

The beauty of writing

The beauty of writing

I woke up this morning, sleep deprived after my cat woke me up repeatedly last night and discovered I needed to write something about writing. Perhaps it's because I'm reading "Bird by bird" again by Anne Lamott. So here is another post about writing as a software engineer.

I love writing, I love the feel of knowing there's a seed of a story or an idea inside. With some writing projects I know something is there worth telling, but I don't always know enough about it yet to be sure. Sometimes it requires upfront research and sometimes I think I know enough to start writing and see the idea emerge as I type. The piece advances, in fits and starts, as I go back and research gaps, and often discarding the whole thing and rewriting it again... until it has formed itself into something worth while. Then I send it to people for review and I get more ideas and insights making the piece better again.