https://jack-vanlightly.com/blog daily 1.0 2025-12-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477847044851-IQXS4F9793SNFRV8K94Q/shutterstock_458475352.jpg Blog https://jack-vanlightly.com/blog/2025/12/10/the-three-durable-function-forms monthly 0.5 2025-12-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c5a9b747-bd0b-43d9-a424-910ef561a91c/behavior_state_continuum_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 1. (behavior only) -> (behavior with state) -> (state with behavior) -> (state-only) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4c44b23e-cc42-4980-9501-6546dd97e821/three_forms_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 2. The three function forms. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9a6a3917-36c9-4f79-92d7-01d942626175/properties_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 3. How the forms relate to different properties. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/529ad323-b53e-4510-abda-1902ddc4b433/stateless_function_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 4. A stateless function is invoked and runs until completion. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a9f7d233-3360-46fb-8963-9120979bbbf5/session_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 5. The session is kicked off by a main function and can be interacted with via secondary functions or message passing. It runs until completion. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/96a35280-f952-4e56-8bde-206487d2a469/actor_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 6. An actor offers multiple functions for state mutations and reactive logic. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/da538289-3b77-43e2-a604-d8c5a07eb7bc/temporal_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 7. A Temporal workflow as a session with the main function annotated with @workflow and secondary functions for interaction annotated with @signal. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4e179868-ac25-4f8a-b32a-3bcabbdee992/restate_basic_service_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 8. A basic service with four independent handlers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/75b704ea-38f8-4722-9715-6c8a47f9455e/restate_workflow_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 9. Restate workflow with its run handler and secondary handlers for interactivity. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6306a59f-9c65-456b-8f69-54bca0f59eb9/restate_virtual_object_small.png Blog - The Three Durable Function Forms - Make it stand out Fig 10. A Restate virtual object with handlers for mutating its state and possibly executing side effects. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f7b68aae-07f5-4798-92c4-ef29fac63e2f/dbos.png Blog - The Three Durable Function Forms - Make it stand out Fig 11. A DBOS workflow https://jack-vanlightly.com/blog/2025/12/4/the-durable-function-tree-part-2 monthly 0.5 2025-12-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a33ff7b9-df88-41cb-bc42-870e51215438/graph2_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 1. A tree of work kicked off by a root reliable trigger, for example a queue message kicks off a consumer that executes a tree of synchronous HTTP calls. Should any downstream nodes fail (despite in situ retries), the whole tree must be re-executed from the top. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6dd36c97-d0de-4e74-afde-5f614cb1ec70/graph3_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 2. Nodes A, B, C, and E form a synchronous flow of execution. Synchronous flows don’t benefit from balkanized responsibility boundaries. Typically, synchronous work involves a single responsibility boundary, where the root caller is the reliable trigger. Nodes D and F are kicked off by messages placed on queues, each functioning as a reliable trigger. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/74d5fc91-abc2-44dc-a2a8-e4eebd49d539/function_tree_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 3. A durable function tree from part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cff6a282-ad72-4050-b196-d9007f402ccc/function_tree_boundaries_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 4. A function tree consists of an outer responsibility boundary that wraps nested boundaries based on reliable triggers (one per durable function). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/67712171-e342-4765-a79c-6665291067ac/graph4_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 5. The entire execution graph is executed asynchronously, with each node existing in its own boundary with a Kafka topic or queue as its reliable trigger. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3999ef15-54e9-40af-9163-259dee609e8b/eda_boundaries_with_side_effects_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 6. Each consumer is invoked by a topic event (a reliable trigger) and executes a number of local-context side effects. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1f092f64-755c-409a-8d61-fa44f7df0230/orchestration_and_choreography_small.png Blog - The Durable Function Tree - Part 2 - Make it stand out Fig 7. Orchestration employed in bounded contexts (or just core business workflow) with events as the wider substrate. https://jack-vanlightly.com/blog/2025/12/4/the-durable-function-tree-part-1 monthly 0.5 2025-12-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/630dcaf2-b715-4dbe-934c-f05a704020b2/future_slash_promise_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 1. The promise/future as a container for a future value. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/74d5fc91-abc2-44dc-a2a8-e4eebd49d539/function_tree_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 2. A tree of function calls, returning durable promises. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7da823be-3020-4c4d-b55a-532a34de7d5f/function_tree_with_side_effects_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 3. Our durable function tree seen as a tree with local-context and remote-context side effects https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2e869e78-a460-4e56-96ca-e85fe1d1b38b/temporal_workflow_activity_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 4. Temporal’s two layer workflow→activity model. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6a1f1945-951e-46fe-a714-b97d18cef79d/temporal_polling_and_execution_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 5. Workers poll Temporal Server task queues for tasks, and then execute those tasks. Activity are invoked via commands which Temporal Server derives events and tasks from. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1edd39d0-4dff-4db6-9cf0-9f472ca02b42/restate_invocations_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 6. Invocations are driven by Restate Service. Functions will suspend when they await Restate governed remote side effects (and no local side effects). Restate detects when a suspended function should be resumed and invokes it. Note this diagram omits the notifications send from the Restate client back to Restate server related the start and end of each local side effect. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b05bf1da-6a67-46ef-b791-43e2484c22ac/dbos_small.png Blog - The Durable Function Tree - Part 1 - Make it stand out Fig 7. DBOS workers poll Postgres for work. Functions will suspend when they await a timer or another DBOS workflow. The logic is mostly housed in the DBOS client, where the polling logic can detect when to resume a suspended workflow. https://jack-vanlightly.com/blog/2025/11/24/demystifying-determinism-in-durable-execution monthly 0.5 2025-12-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/94565884-7d9c-4c8d-9111-4cf3e0bb9b4c/recovery_via_retry_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 1. A function is retried, using the results of the prior partial execution where available. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/420c6deb-fce3-4a9b-9931-fce6e80778d8/control_flow_vs_side_effects_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 2. Control flow and side effects https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/80eb30bd-f7e0-4119-83f6-e3a6d60c024d/process_order_flow_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 3. process_order function as a mix of control flow (green) and side effects (grey) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fa313df8-5c99-436a-b997-51601f3575e2/double_charge_bug_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 4. Double charge bug because of a non-deterministic if/else https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/df69b55d-8ae2-48d1-9074-1a4280547347/non_deterministic_flow_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 5. A non-deterministic control flow causes a different branch to execute during the function retry. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4e7423ab-84d2-4db1-a9a9-016c72f1f75d/double_charge_fixed_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 6. Make the customer retrieval deterministic if the control flow depends on it. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f60cef71-ec58-4511-b04d-d3a2ce0f73a2/tree_small.png Blog - Demystifying Determinism in Durable Execution - Make it stand out Fig 7. A tree of function calls, with control-flow in each function. https://jack-vanlightly.com/blog/2025/11/19/have-your-iceberg-cubed-not-sorted-meet-qbeast-the-otree-spatial-index monthly 0.5 2025-11-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0af2514d-4bef-47a4-85f7-cefa612a95f2/lexicographical_order_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 1. Lexicographical order of two dimensions, which follows the “sort by x then by y” order. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c3bca513-378f-48f1-940b-52d46cd99569/z_order_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 2. Z-order uses bit mixing to produce a single scalar sort key, which determines the order, which resembles a z-shaped space-filling curve. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/656e282f-f2ac-4de2-85ed-4c5b7eea9660/cube_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 3. A table indexed on three columns leads to a 3-dimensional (normalized) space (more on that later). In this figure, the original cube has subdivided into 8 subcubes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/50d531af-d3e4-4eab-b735-ebeb7aeb2281/cube_subdivide_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 4. A cube subdivides into 8 subcubes (in a 3-dimensional space) corresponding to three indexes columns. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/df4af1cb-5704-4527-8d10-c743c59c84a2/closeness_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 5. Two dimensional space with normalized domains https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5b577fb6-e3ff-422b-b19f-fd65d0479467/2d_space_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 6. Cubes adaptively subdivide recursively based on multidimensional spatial density. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/75754fe1-869e-4418-a80d-8274e678a990/otree1_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 7. The OTree representation of the cubes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b400f623-aeda-4d2e-9408-51c17ecccfbb/otree_progress_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 8. The OTree over time. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b7885cb7-2770-49ee-bd53-7758d6807f3e/otree_blocks_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 9. Each node of the OTree contains one or more blocks, where each block is a data file (such as Parquet). In this example, the root cube reached capacity with three files and split along 2 dimensions. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c7fe0052-6fe0-42cd-b857-27e51bea27eb/otree_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 10. The data point maps onto the nodes: root, 3, 30 and 302. A query whose filter predicates cover this point may end up reading each of these files (it depends on the column stats). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/48114b19-2083-4a7f-a278-9259688d3474/iceberg_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 11. A very simplified representation of four Iceberg commits which add one Parquet file per commit. The root cube splits in the 3rd snapshot, writing to one subcube, and another subcube in snapshot 4. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/04aafdcd-f118-44b7-9c97-b5836833f941/delta_log_small.png Blog - Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index - Make it stand out Fig 12. A very simplified representation of the Delta log with four add_files operations. Each added file is mapped to a cube id (where the tree structure is encoded into the cube ids). https://jack-vanlightly.com/blog/2025/11/5/how-would-you-like-your-iceberg-sir-stream-or-batch-ordered monthly 0.5 2025-11-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/77d1ff1f-07b9-4e6a-b952-7a5e6c990507/streaming_bootstrap_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 1. Bootstrap a streaming Flink job from historical then switch to real-time. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d6a478ef-8232-4885-a090-048786ec2fae/streaming_bootstrap_sort_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 2. Sort batch-ordered historical data in the Flink source task. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a1aff765-735f-4521-9129-9351e6b11a6d/fluss_stream_order_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 3. Fluss stores real-time and historical data in stream-order. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ca6c6bbc-fec3-4e3d-9c89-f539b3ec807b/iceberg_partition_spec_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 4. An example of the Iceberg partition spec and sort order https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5bdbb327-165e-48e6-a59f-cc0fef137f40/fluss_log_table_as_iceberg_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 5. The partitions of an Iceberg table visualized. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/887cb6fe-8f48-4707-8142-8a0de4e7af92/flink_split_reading_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 6. Flink source bootstraps from Iceberg visualized https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/242d1c94-cc0c-4912-b95f-ee14ba74b2ee/unpartitioned_iceberg_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 7. Unpartitioned primary key table with 6 buckets. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7bc3c46d-4cea-4688-adcc-36b17d51d3ff/confluent_materialization_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 8. Tableflow continuously materializes a copy of a Kafka topic as an Iceberg table. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/21125fc4-9319-435a-a7d3-c294543bfa79/internal_tiering_and_materialization_small.png Blog - How Would You Like Your Iceberg Sir? Stream or Batch Ordered? - Make it stand out Fig 9. Copy 1 (original): Kora maintains stream-ordered live and historical Kafka data. Copy 2 (derived): Tableflow continuously materializes Kafka topics as Iceberg tables. https://jack-vanlightly.com/blog/2025/10/22/a-fork-in-the-road-deciding-kafkas-diskless-future monthly 0.5 2025-10-24 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6e9c1e18-db5d-4882-96e4-79f52ed0b139/WarpStreamArchitectureSmall.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig WS-A. The WarpStream stateless/stateful split architecture. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e986fbc0-4580-4dc0-a5ca-da06a9416056/ws_write_path_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig WS-B. The WarpStream write path. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ed339c4d-5320-4fae-a6ea-fa5c6269128f/ws_shared_read_cache_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig WS-C. Shared per-zone read cache to reduce S3 throughput and request costs. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cbf19d52-699e-4c38-833a-f6efd948e792/ws_roles_and_groups_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig WS-D. Agent roles and groups can be used for shaping traffic, independent scaling and deploying different workloads into separate VPCs. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fc614a0c-0d44-4d2f-9a77-9e78d3ed8271/kip_1150_rev1_write_path_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev1-A. Leaderless brokers upload shared objects, then commit and sequence them via the Batch Coordinator. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/34443a20-32c6-4ee7-8f9c-99467eb950ce/kip_1150_rev1_read_path_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev1-B. The consume path of KIP-1150 (ignoring any caching logic here). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5fed48b9-c579-4817-88b8-1bcb056279c7/kip_1150_with_roles_and_groups_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev1-C. Broker roles would bring the stateless/stateful separation that would unlock elasticity in the high throughput workload of many Direct-to-S3 deployments. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/daa016c7-aa50-4e1a-86a2-6f9a803b61b0/inkless_write_path_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig Inkless-A. The write path. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2237734f-ec4d-41f1-a376-fad51aa50c9b/inkless_read_path_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig Inkless-B. The read path. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2fe2abba-c1cd-4c93-9887-39af8f4619f7/inkless_leaderless_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig Inkless-C. Direct-to-S3 traffic uses a leaderless broker architecture with coordination owned by Postgres. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/06196b3e-0310-44cd-a44c-286c222db5f5/inkless_postgres_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig Inkless-D. Postgres data model. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5a086f7c-c614-4e62-b6a6-69e56b441ce0/slack_kip_1176_architecture_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig Slack-KIP-1176-A. Leader-based architecture retained, replication replaced by an S3 WAL. Tiered storage manages long-term data. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/535b6937-e6ad-413d-a92d-19d2ba52b313/slack_kip_1176_active_tiering_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig Slack-KIP-1176-B. Produce batches are written to the page cache as usual, but active log segment files are aggressively tiered to S3 (possibly Express One Zone) in combined log segment files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3807896a-127f-4df6-929e-e02f7d6bf929/kip_1150_rev2_stage1_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev2-A. Write path stage 1 (the synchronous part). Leaderless brokers upload multi-topic WAL Segments, then commit and sequence them via the Batch Coordinator. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d965a86a-edc8-4500-a082-e97ef637ac54/kip_1150_rev2_stage2_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev2-B. Stage 2 of the write-path where assigned brokers download WAL segments and append to local log segments. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5ff56369-1e08-45ee-b5e0-6ca4565382db/kip_1150_rev2_stage3_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev2-C. Tiered storage works as-is, based on local log segment files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3fc1a686-9bd8-4d70-a440-4dae8c846b3a/kip_1150_rev3_write_path_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev3-A. Replicated partitions continue to exist, acting as per-partition sequencers and metadata stores (replacing batch coordinators). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e249d9f9-e3ac-4f4a-a0bc-fafbfd550db6/kip_1150_rev3_wal_cleanup_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Fig KIP-1150-rev3-B. The WAL File Manager is responsible for WAL Segment clean up https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ff2f5e25-9e67-413e-86bc-34a5c4a8b558/continuum2_small.png Blog - A Fork in the Road: Deciding Kafka’s Diskless Future - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/10/15/why-im-not-a-fan-of-zero-copy-apache-kafka-apache-iceberg monthly 0.5 2025-10-20 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7fbf4926-ac74-4204-8d78-73cd0097f28b/tiering_materialization_small.png Blog - Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg - Make it stand out Fig 1. Tiering vs materialization https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5140b923-5966-4541-9f46-f693e302575d/internal_shared_tiering_materialization_small.png Blog - Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg - Make it stand out Fig 2. Shared tiering makes the long-term storage a shared storage layer for two or more systems. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3d8963c3-8a97-4c8a-80c9-d6aa9592ad5f/internal_tiering_materialization_small.png Blog - Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg - Make it stand out Fig 3. Internal Tiering + Materialization https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8f150648-f9b4-4471-9667-f169ef251316/shared_tiering_small.png Blog - Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg - Make it stand out Fig 4. Kafka tiers data directly to Iceberg. Iceberg is shared between Kafka and analytics. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/706530e7-b673-4afa-91aa-2ee818e35bba/by_offset_small.png Blog - Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg - Make it stand out Fig 5. Column statistics and Iceberg partitions clearly tell Kafka where a specific topic partition offset range lives. But these pruning statistics are not useful for analytics queries. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/524d56ba-6ed9-4c68-8b14-fd400a8b4e07/by_bis_dimension_small.png Blog - Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg - Make it stand out Fig 6. Column statistics and partitions clearly tell analytics queries that want to slice and dice by EventType and Region how to prune Parquet files. But these pruning statistics are not useful for Kafka at all, which must do a costly table scan (from a Kafka broker) to find a specific offset range to rebuild a log segment. https://jack-vanlightly.com/blog/2025/10/8/beyond-indexes-how-open-table-formats-optimize-query-performance monthly 0.5 2025-10-08 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9143890b-a337-4fe4-8c34-44b5987f8c3d/B-tree-clustered-index-small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 1. The clustered index with a primary key of UserId as an auto-incrementing int. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bf590b5b-e322-4950-8d78-0e34a4fa2f91/secondary_indexes_small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 2. Secondary indexes map columns to the clustering key (primary key) so the query engine can perform a clustered index seek. Note that in small tables, secondary indexes are not used as it's quicker to scan the table. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e9769c14-0b8e-405d-b940-50fab745d6d6/iceberg_tree_small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 3. An Iceberg table is queried based on a single snapshot, which forms a tree of metadata and data files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7ec1487a-9d96-49f6-b80c-29489ba47618/Iceberg_compaction_partitioning_sort_order_small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 4. A table with three partitions by Country and sort order by Surname. Sort order is preserved, but less effective on newly written small files. Compactions can sort rows of a large group of files into fewer larger files with ordering preserved across them. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/34e7c7cc-2ad4-4ee3-8d06-d5a49aba3026/parquet_rowgroups_small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 4. Parquet splits up the data into one or more row groups, and within each row group, each column is stored contiguously. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2d777797-5d6f-4dc1-be06-f4a560973e79/materialized_views_small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 5. Materialized views help support diverse queries that the base table, with its one layout, cannot. Making such materialized views transparent to the user, by allowing the query planner to use the MV instead of the base table is a powerful approach, akin to the use of secondary indexes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0c1de095-f1c0-46c9-a296-b4bd40c0f2f5/star_schema_small.png Blog - Beyond Indexes: How Open Table Formats Optimize Query Performance - Make it stand out Fig 6. The central fact table contains information such as sales, with foreign keys to the dimension tables and facts such as sale amount, quantity, etc. Dimensions may include products, customers, vendors, dates and so on. A product dimension will have a ProductId and a number of columns describing aspects of the product such as manufacturer, SKU, etc. https://jack-vanlightly.com/blog/2025/9/2/understanding-apache-fluss monthly 0.5 2025-09-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/32967327-23bc-4922-9ffc-8fbc96f84c5d/fluss_architecture_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 1. Fluss architecture components https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bb810253-e2de-486e-be82-041d722e310f/Log_PK_Table.png Blog - Understanding Apache Fluss - Make it stand out Fig 2. Fluss logical model https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cd5c6e5e-4961-4eaa-871a-aef0be12a9e8/fluss_paimon_flink_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 3. Fluss and Paimon forming real-time/historical table storage for Flink https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aef339e6-4335-4a44-ba2f-1025c28ebc0a/flink_job_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 4. Flink as a changelog engine https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5e9b8e40-8df0-4a71-bc46-5acbcc13dd9a/flink_table_store_evolution_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 5. Flink table storage project evolution https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/65801f6b-e5c2-4c40-ad64-b7a8ab5109ae/fluss_job_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 6. Fluss PK Table offloads MV/changelog state management from Flink and provides a shared data primitive for other Flink jobs, with lookup (lookup join) support https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/70d6482c-ca13-4b71-ae0a-815361248fdf/columnar_files_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 7. The Fluss client serializes record batches into columnar Arrow IPC batches which are appended to segment files by the tablet servers. On the other end, the client that reads the batches converts the columnar Arrow into records again. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c05d71c8-ce15-4096-9cca-00b7619aa84a/realtime_historical_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 8. Fluss real-time and historical data https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/10eebba5-f2d2-4b22-8d29-3846f0ce21a2/fluss_core_architecture_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 9. Fluss core architecture (without lakehouse or Flink) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9849c256-50c4-4b2f-992d-d309859448ea/fluss_partitions_buckets_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 10. Log table sharding scheme. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bff15d37-0c06-4484-999b-ba170b06d34f/partition_bucket_selection_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 11. Table partition and table bucket selection when writing. Fluss clients write to table buckets according to the partition columns(s) to choose the table partition, then a bucket scheme, such as round-robin, sticky, hash-based (like Kafka producers choosing partitions). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0d1f0255-4bc8-4c2e-9f7a-44526a35db4d/log_tablet_replication_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 12. Log Tablet replication is based on Kafka partition replication. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a7a042e8-c05b-4f9d-92b8-703868463d68/log_table_with_tiering_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 13. Log tablet with internal segment tiering and replication. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f89bdb32-edfc-4dd1-987d-8e6a21c92e3b/log_table_flink_source_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 14. Fluss integration of Log Table via Flink Source API (without lakehouse integration). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/285e507d-0533-4571-8ff7-962127097d14/pk_table_replication_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 15. A KV tablet and its replicated child log tablet for the changelog. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c10c9ecf-1ae1-4ef4-a16b-d6b8900995d2/pk_table_tiering_replication_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 16. KV Tablet (and child Log Tablet) read, write, tiering and replication. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ea5946be-8a45-4e59-96f3-74c047104cf2/pk_table_flink_source_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 17. Fluss integration to the Flink Source API. The difference between log table and primary key table source are highlighted in green. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ed9cc5f5-fd48-4846-8643-c364400cbd1c/fluss_logical_physical_table_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 18. Logical table maps onto a Fluss Table (hosted by Fluss cluster) and optionally, a Lakehouse Table https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/657b9c0e-a45b-444e-a8ec-b8fa5370c684/lakehouse_tiering_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 19. Flink-based lakehouse tiering components. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b679d907-64a0-433b-a21f-047bc1cf402a/lakehouse_tiering_flink_job_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 20. Internal topology lakehouse tiering Flink job. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7cfcc026-f306-4544-8669-13c09c24404d/tiering_states_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 21. Table tiering job state transitions, managed by Fluss coordinators. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bf614772-19c0-4d48-9214-561aecccdf3a/client_storage_unification_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 22. Fluss uses client-side stitching at two layers of the client-side abstractions. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a91a827b-1480-4f5b-aed1-ffd032753c1c/primary_storage_small.png Blog - Understanding Apache Fluss - Make it stand out Fig 22. Fluss cluster storage and Paimon storage form the primary storage for the primary system (Flink). https://jack-vanlightly.com/blog/2025/8/21/a-conceptual-model-for-storage-unification monthly 0.5 2025-09-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/12957aab-4f21-43c3-a24e-7caa7b3fafee/logical_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3370e82d-afda-4c9f-aef3-91a97b95e735/virtualization_layers_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d9c0c036-8a5c-4470-a598-787b15f7e97e/tiering_materialization_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5140b923-5966-4541-9f46-f693e302575d/internal_shared_tiering_materialization_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/728d8d63-36dc-4f1d-a593-d2f6e6299069/bidirectional_conversion_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7a6c5875-dad4-454b-96ea-f6a1318db0ee/server_side_stitching_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5347d916-37bd-40b2-909c-049643402b38/client_side_stitching_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f54af551-b1ad-43d9-ad53-c4e991275c93/flink_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9029a104-f912-4bd8-970d-01da39bc27b2/integrated_or_external_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e7ee2a3d-67bf-4404-9131-40d0d04d9a06/direct_vs_api_access_tier1_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4c9d0092-8aeb-48f3-bdac-6ce36d116637/direct_vs_api_access_tier2_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b9b5edbb-024f-415d-a545-be58bcef4b9c/flink_does_not_care_small.png Blog - A Conceptual Model for Storage Unification - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/7/28/remediation-what-happens-after-ai-goes-wrong monthly 0.5 2025-07-28 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6566d765-0330-4fa6-bfbc-a90f8b259e15/ai_agents_what_can_go_wrong.png Blog - Remediation: What happens after AI goes wrong? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/7/22/the-cost-of-being-wrong monthly 0.5 2025-07-22 https://jack-vanlightly.com/blog/2025/7/15/responsibility-boundaries-in-the-coordinated-progress-model monthly 0.5 2025-07-17 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6dfada9f-2aa4-4ab2-8f9b-9cdaf7d384e0/reliable_trigger_prog_work.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 1. Reliable progress is made via reliable triggers and progressable work. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/22b2749d-4bae-44e1-a5b6-181d6c8f7cdc/placement_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 2. Different ways of relying on reliable triggers to make downstream work reliable. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/66cef27e-a837-4616-8395-d65f3a03cdfd/graph1_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 3. Synchronous execution graph without a reliable trigger. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a33ff7b9-df88-41cb-bc42-870e51215438/graph2_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 4. Synchronous execution graph with a root reliable trigger, constituting a responsibility boundary. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6dd36c97-d0de-4e74-afde-5f614cb1ec70/graph3_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 5. Synchronous and asynchronous work in the execution graph is separated into different responsibility boundaries. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/67712171-e342-4765-a79c-6665291067ac/graph4_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 6. The entire execution graph is executed asynchronously, with node (service) existing in its own boundary. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9419cdb5-8827-4f63-858c-91cd1bcde621/graph5_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 7. The same execution graph from fig 6, where each two-way communication acts like two one-way communications. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cb57bf85-93aa-4b58-ad4e-e9ebc3983163/dee_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 8. How a Durable Execution Engine (DEE) drives a work, with a polling and caching approach. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/10ca4983-8b5a-4a29-b6f4-b9c9c38aa928/dee_boundaries_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Fig 9. Orchestrated flows consist of an outer responsibility boundary, that wraps nested boundaries based on reliable triggers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/46cf6306-7db4-4e49-8bbb-7dfdc6bf4318/mix_small.png Blog - Responsibility Boundaries in the Coordinated Progress model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/6/11/coordinated-progress-part-4-a-loose-decision-framework monthly 0.5 2025-06-11 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8e79d363-19ad-4508-a392-35a2efdfc778/decisions.png Blog - Coordinated Progress – Part 4 – A Loose Decision Framework - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c3868c67-ca0d-4e1d-880e-ed0c64089037/graph2_small.png Blog - Coordinated Progress – Part 4 – A Loose Decision Framework - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6dfada9f-2aa4-4ab2-8f9b-9cdaf7d384e0/reliable_trigger_prog_work.png Blog - Coordinated Progress – Part 4 – A Loose Decision Framework - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/6/11/coordinated-progress-part-3-coupling-synchrony-and-complexity monthly 0.5 2025-06-11 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9c87072d-050d-4f73-a3c0-333b886332b0/e-commerce_events.png Blog - Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aa12d0e5-bbed-49a7-ab3a-dab317be538e/e-commerce_events_compensations_small.png Blog - Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8c505376-18a8-4b06-ab25-005cdfb70b30/continuum.png Blog - Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1349414d-f079-4ef3-8d44-25cb608c2c0d/resumability_rpc_queues_small.png Blog - Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e5bcdd79-16ad-4ee3-8af3-795bedf22199/resumability_dee_small.png Blog - Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/6/11/coordinated-progress-part-2-making-progress-reliable monthly 0.5 2025-06-11 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bec31493-b6cb-402a-964f-d5fa98027acf/graph2_small.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ab2ac544-5651-4ca8-901a-afcafac9b76f/reliable_trigger_prog_work.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9356c68e-1e43-496f-b664-4f0680342cd8/flink_reliability.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1ebc4bc5-d8df-419e-9434-8c227cf04f4b/reliable_rpc_small.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3565d33c-d698-49e2-894a-a368f2c34b96/event_driven_reliable_progress.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d8a7c17c-896c-411e-b9a7-f09353663677/orchestration_reliable_progress.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/914753e6-8d7b-44be-a6e0-6e5937f80077/hybrid1.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/863561e7-3a65-4f66-b5cc-cedc97ae7be0/hybrid2_small.png Blog - Coordinated Progress – Part 2 – Making Progress Reliable - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/6/11/coordinated-progress-part-1 monthly 0.5 2025-06-11 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/785f05b4-2554-4bf9-a8de-9d656887e94b/graph2_small.png Blog - Coordinated Progress – Part 1 – Seeing the System: The Graph - Make it stand out The Graph. Nodes connected by edges. Workflows as sub-graphs. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2c786169-0bc9-4c76-be86-5441c7421d0a/graph_choreography.png Blog - Coordinated Progress – Part 1 – Seeing the System: The Graph - Make it stand out Coordination via publish-subscribe semantics. The entire impact of an upstream event can spread far, and is dynamic over time. The boundaries of any given workflow within that wider event flow can be soft and hard to define (but with low coupling). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ed847f14-0984-4fc2-83fb-f5f83f3a3058/graph_orchestration.png Blog - Coordinated Progress – Part 1 – Seeing the System: The Graph - Make it stand out Coordination via procedural orchestration semantics. The entire impact of an upstream workflow can spread far, as individual nodes can still emit events. The boundaries of a given workflow are clearly encoded in the orchestration code, albeit at the cost of increased coupling. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a58e664a-1320-4288-ac07-4e61cfb3c716/choreography_flow_small.png Blog - Coordinated Progress – Part 1 – Seeing the System: The Graph - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/3/13/log-replication-disaggregation-survey-apache-pulsar-and-bookkeeper monthly 0.5 2025-03-13 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ed94a4ec-e5ee-442f-acfe-9bca4ef0fb99/LogletChainSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 1. The shared log, formed by a chain of loglets. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/739dc33d-59b9-4389-bd28-12d11785094e/Reconfiguration.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 2. Reconfiguration moves the system from one steady state configuration to another in response to failures, policy triggers or other factors such as load balancing. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d94ab2ce-2f61-4fb5-a21b-8d7983f6de56/QuorumReplicationReconfigurationSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 3. Depicts a quorum replication Loglet implementation. Loglet 3 experiences a failed server, notifies the VirtualLog which seals Loglet 3 and then appends Loglet 4 with a new set of storage servers (without the failed one). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b2574771-e1ae-48ad-9d69-36f3540e24ab/VirtualConsensusRolesSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 4. The three disaggregated roles of Virtual Consensus (client, Loglet storage, metastore). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c246a991-ec51-48af-9d17-dda35a99f554/PlusarAndBKSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 5. Pulsar brokers, using BookKeeper as its log storage sub-system. All components depend on a common metadata store. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2a267bad-edaa-445d-8e03-81841ed7d3cc/BKRolesSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 6. The BookKeeper roles of client, bookie and metadata store. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/435cc0d5-c48d-4d8b-94af-f12e936b7535/LedgersSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 7. Depicts the metadata that represents a Pulsar topic partition, and the final ledger, which has three fragments. The range of the final ledger and final fragment is not set in metadata yet, as it is still open and accepting writes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9c84a5c2-0bd2-4cc8-ac9b-dd0398160121/LedgerMetadataSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 8. Ledger metadata https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/989e2296-e94a-445f-999a-55c03871d1cb/BkProposerAcceptorSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 9. The client that opens a ledger is its distinguished proposer, also known as a leader. The bookie ensemble are the acceptors. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a22ba76a-1fc8-460e-b1be-54ae0564ed22/EnsembleChangeSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 10. A bookie fails, causing an ensemble change. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/eb327ea0-3dc5-4913-ad8a-181d12f4c4d5/CloseLedgerSmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 11. Closing a ledger, by the leader client, is a metadata op, setting the status and last entry id of the ledger. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/22067ba1-0128-4659-8601-d594d4e25a33/LedgerRecoverySmall.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 12. Ledger recovery. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/739dc33d-59b9-4389-bd28-12d11785094e/Reconfiguration.png Blog - Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper - Make it stand out Fig 12. Each fragment is a steady state configuration, an ensemble change is a reconfiguration that leads to a new steady-state fragment. https://jack-vanlightly.com/blog/2025/2/21/log-replication-disaggregation-survey-kafka-replication-protocol monthly 0.5 2025-02-21 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/739dc33d-59b9-4389-bd28-12d11785094e/Reconfiguration.png Blog - Log Replication Disaggregation Survey - Kafka Replication Protocol - Make it stand out Fig 1. Reconfiguration moves the system from one steady state configuration to another in response to failures, policy triggers or other factors such as load balancing. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/546a61ed-5e58-49bf-a458-01272f36e82d/KafkaRolesSmall.png Blog - Log Replication Disaggregation Survey - Kafka Replication Protocol - Make it stand out Fig 2. The roles in the Kafka Replication Protocol https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/187c73a9-1784-4c25-8a2b-be4f7cae6cba/DataPlaneLogicalPhysicalModelSmall.png Blog - Log Replication Disaggregation Survey - Kafka Replication Protocol - Make it stand out Fig 3. We usually talk and think about replication using the logical model, but the work is really carried out at the broker level using multiple fetch sessions for each broker pair. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fb573fe2-b383-4bc0-a884-172090333832/RoleCommsSmall.png Blog - Log Replication Disaggregation Survey - Kafka Replication Protocol - Make it stand out Fig 4. The control plane←→ data plane communication. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f9865e24-d1e6-4d76-97c6-c305dbe408b2/MembershipChangesSmall.png Blog - Log Replication Disaggregation Survey - Kafka Replication Protocol - Make it stand out Fig 5. Membership changes are executed as a linear series of metadata ops, ensuring minimum ISR size for safety. https://jack-vanlightly.com/blog/2025/2/19/log-replication-disaggregation-survey-neon-and-multipaxos monthly 0.5 2025-02-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e5bf1650-1210-4c32-b0f4-a18b496867ae/pg_and_sk.png Blog - Log Replication Disaggregation Survey - Neon and MultiPaxos - Make it stand out Fig 1. The main components of the Neon architecture. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3d82239d-6447-4d2c-b16d-68419cfc0c8f/multi_paxos_roles.png Blog - Log Replication Disaggregation Survey - Neon and MultiPaxos - Make it stand out Fig 2. Neon components mapping onto MultiPaxos roles. https://jack-vanlightly.com/blog/2025/2/17/towards-composable-data-platforms monthly 0.5 2025-02-18 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/87df8e0b-52e8-41ed-ac48-0526a2b95b98/table_virtualization_small.png Blog - Towards composable data platforms - Make it stand out Fig 1. Table virtualization, enabled by shared storage and open table formats https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9014aaf4-d6c8-45c8-89cf-a98dd55f4a46/TakesTwoSmall.png Blog - Towards composable data platforms - Make it stand out Fig 2. It takes two platforms to tango. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/82cd35e1-f178-476e-8211-711eea4d3995/NativeExternalSmall.png Blog - Towards composable data platforms - Make it stand out Fig 3. Native and external tables, still a powerful model for collaboration. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e7200d09-fee1-4ee9-9948-7e1f751a3823/StreamTableMaterializationSmall.png Blog - Towards composable data platforms - Make it stand out Fig 4. Stream/table materialization working in combination with table virtualization. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4328f240-9e7d-4aad-9530-e8f563c6df48/GraphMindsetSmall.png Blog - Towards composable data platforms - Make it stand out Fig 5. From the the ELT mindset (extract from left side to deposit on right side), towards the graph mindset (a graph of derived data sets). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6997803c-649e-4e11-9fc7-5ef5a60c088b/ConfluentDatabricksSmall.png Blog - Towards composable data platforms - Make it stand out Fig 6. Planned bidirectional flow between Confluent and Databricks. Kafka topics exposed as Delta tables and vice versa. https://jack-vanlightly.com/blog/2025/2/10/how-to-disaggregate-a-log-replication-protocol monthly 0.5 2025-02-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0f59ac55-c56b-4114-a56a-1e885d3c7628/MultiPaxosRolesSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 1. The Paxos roles, seen here in the context of MultiPaxos. Note that the number of each role are not limited to this depiction. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5d0ee473-0e32-4dac-9fa3-50edfec02518/ControlDataPlanesSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 2. The protocol is split into control plane and data plane. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3ce68987-fca1-4f99-9d8d-88ae60ea8d2b/LogSegmentsSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 3. The log is formed from a chain of logical log segments. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/98fb15b1-cd54-4ded-b77c-337586e7c957/MappingsSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 4. A log can store complete log entries, or can store only log entry metadata, pointing to another storage service that hosts the data payload. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4a83883a-2c1c-486f-9594-e2546905ac3b/PointerBasedLogSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 5. Data is separated from ordering. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/085b2878-9cd7-4016-b9c0-be96ff93fd8c/SequencerSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 6. Leaderless servers perform the IO and ordering is achieved by a simple sequencer component. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1cfc2aec-9727-40fe-8253-5a7bccb23fe6/LeaderlessProxiesSmall.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 7. Leaderless proxies can hide leader-based log replication protocols https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cb858a0e-60c4-4691-bab3-a558767b197a/LeaderlessProxies2Small.png Blog - How to disaggregate a log replication protocol - Make it stand out Fig 8. Data and ordering separated. https://jack-vanlightly.com/blog/2025/2/6/steady-on-separating-failure-free-ordering-from-fault-tolerant-consensus monthly 0.5 2025-02-06 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/95ae68a4-7d16-4591-92fb-4076e8ea5727/ReconfigurationSmall.png Blog - Steady on! Separating Failure-Free Ordering from Fault-Tolerant Consensus - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2025/2/5/an-introduction-to-virtual-consensus-in-delos monthly 0.5 2025-02-06 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/823d7a9b-ac8f-4170-88e9-f610223c8a65/VirtualConsensusInDelosFig1.PNG Blog - An Introduction to Virtual Consensus in Delos - Make it stand out From Virtual Consensus in Delos. Shows database servers, in their role as the state-machines, sitting above the Virtual Log abstraction. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ed94a4ec-e5ee-442f-acfe-9bca4ef0fb99/LogletChainSmall.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 1. The shared log, formed by a chain of loglets, https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/855f66b4-49a2-46b6-b349-c15e58dbbdeb/AddressSpaceSmall.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 2. Virtual address space maps onto loglets https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3e1bb339-5236-4cfb-8fad-81ec89718759/ReconfigurationSmall.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 3. Reconfiguration moves the system from one steady state configuration to another in response to failures, policy triggers or other factors such as load balancing. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9ec6ab78-24bd-47d6-af5f-15f5f792f2b4/ComponentsSmall.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 4. Under the Virtual Log abstraction, we have the Client, Metastore and Loglet Layer. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d94ab2ce-2f61-4fb5-a21b-8d7983f6de56/QuorumReplicationReconfigurationSmall.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 5. A reconfiguration with a quorum-replication loglet implementation. Now that Loglets 1 and 3 are under-replicated because of the loss of server 9, the virtual log will also have to perform reconfig-modify operations to replace server 9 in those loglets to restore the correct level of redundancy. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bbcce877-0207-440b-9158-4f18c864fbb8/PulsarManagedLedgerSmall.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 6. The Pulsar/BookKeeper architecture is akin to a specific implementation of Virtual Consensus. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/52db2883-afde-47cf-bfac-4dbf8565831d/DelosExample1Small.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 7. Delos ensures strict consistency across database replicas by applying commands from the virtual log. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/54d6272a-ffa4-4137-9ead-0f17854bcdf9/DelosExample2Example.png Blog - An Introduction to Virtual Consensus in Delos - Make it stand out Fig 8. The NativeLoglet uses a dedicated sequencer component within the loglet abstraction. Otherwise, we see similarities to Pulsar. One difference is that Delos maintain state with read replicas, whereas Pulsar brokers do not. https://jack-vanlightly.com/blog/2025/2/3/why-snowflake-wants-streaming monthly 0.5 2025-02-03 https://jack-vanlightly.com/blog/2025/1/16/ai-agents-in-2025 monthly 0.5 2025-01-16 https://jack-vanlightly.com/blog/2024/12/9/on-1-million-page-views monthly 0.5 2024-12-09 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cebaed2c-1307-4249-bf9a-ab7ea96d2b83/AllTimeStats.png Blog - On 1 million page views - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2024/12/6/to-be-atomic-or-non-atomic-that-is-the-question-fizzbee monthly 0.5 2024-12-06 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3078c213-d7cb-47db-be16-69b86752d777/ping_pong_small.png Blog - To be atomic or non-atomic, that is the question (Fizzbee) - Make it stand out Fig 1. The basic idea of this ping pong spec. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/604a8934-214e-4d18-bee7-34b925c4b4f6/atomic-vs-nonatomic-small.png Blog - To be atomic or non-atomic, that is the question (Fizzbee) - Make it stand out Fig 2. Atomic actions with explicit network vs non-atomic actions with implicit channels (invoking funcs on other roles). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/332f7ea6-d699-4a28-911e-843d44fbb256/error_trace.png Blog - To be atomic or non-atomic, that is the question (Fizzbee) - Make it stand out Fig 3. The error-states.html of the non-atomic spec with the silly mistake. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/72b82326-4aea-4816-8399-f1d4ba0cc8cd/error_trace_atomic.png Blog - To be atomic or non-atomic, that is the question (Fizzbee) - Make it stand out Fig 4. The error-states.html of the atomic spec with the silly mistake. https://jack-vanlightly.com/blog/2024/12/5/an-introduction-to-symmetry-in-tla monthly 0.5 2024-12-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5368e0a9-a9be-4ac1-9df1-8c0b3c110f8e/CtrsWithoutSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 1. Two counters, without symmetry. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3b7694d8-d3bc-4258-a046-b8a7ad4546bc/CtrsWithSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 2. Two counters with symmetry https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f3fed187-78b6-4c33-bf63-ee114fdfe0b1/NodesCountersWithoutSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 3. Nodes and counters, without symmetry. Download link. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/968710cc-027d-47d2-8827-ed2aceef652a/NodesCountersWithSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 4. Nodes and counters, where both sets are symmetrical. Download link. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fe4297ff-0653-4d60-8d17-8dec97be4b29/NodesCountersCounterSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 5. Nodes and counters, where only the counter set is symmetrical. Download link. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3f3c83d2-ea1e-4029-aa02-d9321450d108/NodesCountersNodeSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 6. Nodes and counters, where only the node set is symmetrical. Download link. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/784b11c7-abc0-4005-9a5b-640db9d1d610/PerNodeCounterWithoutSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 7. Per node counters, without symmetry. Download link. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ea17bb43-89e6-4f65-8834-9eb4aa30e7c8/PerNodeCounterWithSymmetry.png Blog - An introduction to symmetry in TLA+ - Make it stand out Fig 8. Per node counters, with both sets symmetrical. Download link. https://jack-vanlightly.com/blog/2024/11/26/dismantling-elt-the-case-for-graphs-not-silos monthly 0.5 2024-11-26 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b5b329db-b53b-44c5-b79c-ebb8c4baab21/ELT_small.png Blog - Dismantling ELT: The Case for Graphs, Not Silos - Make it stand out Fig 1. The ELT mindset: Extract from left side to deposit on right side. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4f527cab-420e-4c98-a188-d28e8835b237/Graph_small.png Blog - Dismantling ELT: The Case for Graphs, Not Silos - Make it stand out Fig 2. A graph of derived data sets https://jack-vanlightly.com/blog/2024/11/21/the-law-of-large-numbers-a-foundation-for-statistical-modeling-in-distributed-systems monthly 0.5 2024-11-21 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/48d5bdd4-1ba6-43da-a376-18a33617e477/swim_formula.png Blog - The Law of Large Numbers: A Foundation for Statistical Modeling in Distributed Systems - Make it stand out Fig 1. From SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d9e93e60-4c68-45b7-a89e-57950d9752aa/swim_stats.png Blog - The Law of Large Numbers: A Foundation for Statistical Modeling in Distributed Systems - Make it stand out Fig 2. Violin plots effectively combine summary statistics and detailed distribution shapes, making them ideal for visualizing data density, variability, and comparisons across groups. In this case we see that a higher dissemination limit (the number of times a node will share the same piece of information with peers) results in fewer rounds to reach convergence, with much lower variance compared to lower dissemination limits. https://jack-vanlightly.com/blog/2024/11/19/obtaining-statistical-properties-through-modeling-and-simulation monthly 0.5 2024-11-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2f3c062a-adf0-4dfa-89b3-5345882fe21e/swim_stats.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 1. A violin plot for the number of rounds of message exchanges (as a factor of dissemination limit) for a gossip protocol (SWIM) to reach convergence after a single node dies. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0e7c6f61-064e-4616-846f-e5f6b282ec82/cooperating_consumers_small.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 2. Two applications over three queues reach a balance of the number of active consumers. Each queue has an associated subscriber-queue that RabbitMQ uses to determine which consumer is the active-consumer. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d67a0359-c2b6-4bce-a0a5-2dac38f60b29/py_sim_active_q10_horizontal.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 3. X-axis: The number of applications. Y-axis: The number of rounds for the group of applications to reach a balance over 10 queues, after starting up. Left: the order of queue subscriptions is completely random. Right: Each application starts one at a time, a sequentially subscribes to the queues. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/45642db5-28e7-40d8-aec4-a81c8f505c99/py_sim_active_q20_horizontal.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 4. At 20 queues, the p99 for a randomized order of subscriptions was 37 with 20 applications, equating to around 18 minutes of queue releases until balance was achieved. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ee9befb4-e042-43d8-901c-e933ec213ee4/py_sim_active_q10_horizontal_lose_one.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 5. “Lose-one-app” scenario. The rounds to balance of a stable group of applications that lose one member. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/15b87384-8cc1-4161-9e47-65539b2c26b5/py_sim_active_q10_horizontal_lose_one_opt.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 6. “Lose-one-app“ scenario. Left: Original algorithm. Right: The non-active release optimization. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d740f1c0-f9d1-4902-b668-4f63638a27fe/py_sim_active_q10_horizontal_opt.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 7. “Start-up” scenario. Left: Original algorithm. Right: The non-active release optimization. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3896d3a5-a8f1-4091-b4b9-3207bdc1eb68/tla_sim_active_q10_horizontal_opt.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 8. The TLA+ version of fig 6. The results are almost identical, despite the vastly different implementations and runtimes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cb661c8f-3de6-4247-b0e5-0b7ad7416bee/tla_sim_active_q10_horizontal_lose_one_opt.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 9. The TLA+ version of fig 5. Again, they are almost identical. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/512de78d-0cf2-497d-b1bc-8bad65223b9e/tla_dist.png Blog - Obtaining statistical properties through modeling and simulation - Make it stand out Fig 10. The number of runs per AppCount-Algorithm combo. https://jack-vanlightly.com/blog/2024/11/13/incremental-jobs-and-data-quality-are-on-a-collision-course-part-2-the-way-forward monthly 0.5 2024-11-13 https://jack-vanlightly.com/blog/2024/11/13/incremental-jobs-and-data-quality-are-on-a-collision-course-part-1-the-problem monthly 0.5 2024-11-13 https://jack-vanlightly.com/blog/2024/11/5/on-writing-and-getting-from-zero-to-done monthly 0.5 2024-11-06 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7f1fc405-9597-4d22-b75a-1efbb9ba210d/bafkreidj2fjjc3ybqnc24n3nxtqb5gfyidapjpbojibssa5a5luxdad4oa.jpg Blog - On writing and getting from zero to done - Make it stand out Robin Moffatt’s twist on the owl meme. https://jack-vanlightly.com/blog/2024/11/2/learning-first-that-you-can-learn monthly 0.5 2024-11-02 https://jack-vanlightly.com/blog/2024/10/31/forget-the-table-format-war-its-open-vs-closed-that-counts monthly 0.5 2024-10-31 https://jack-vanlightly.com/blog/2024/10/28/the-ultimate-guide-to-table-format-internals monthly 0.5 2024-10-28 https://jack-vanlightly.com/blog/2024/10/24/the-teachers-nemesis monthly 0.5 2024-10-24 https://jack-vanlightly.com/blog/2024/10/21/the-curse-of-conway-and-the-data-space monthly 0.5 2024-10-21 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a6f59249-29ce-4580-8f78-1405858ca5bc/wind-blowing2.png Blog - The curse of Conway and the data space - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2024/9/26/table-format-interoperability-future-or-fantasy monthly 0.5 2024-09-26 https://jack-vanlightly.com/blog/2024/9/19/table-format-comparisons-change-queries-and-cdc monthly 0.5 2024-10-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c113d471-a86d-4325-a29e-b4dbe207fda5/DifferentResultsSmall.png Blog - Table format comparisons - Change queries and CDC - Make it stand out Fig 6. Five examples of change events produced from a set of three operations. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/84d44009-524a-46d4-b36a-c792d4bfb517/MaterializeCDCFilesSmall.png Blog - Table format comparisons - Change queries and CDC - Make it stand out Fig 7. Non-CDC incremental readers tail the log of deltas/snapshots to learn of data files to read. CDC readers also tail the log, but to discover CDC files to read. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7f8b724a-ba32-4b7d-a71a-644010faca1a/InferOnReadSmall.png Blog - Table format comparisons - Change queries and CDC - Make it stand out Fig 8. A CDC reader must compute the difference between adjacent snapshots to infer the row-level changes. https://jack-vanlightly.com/blog/2024/9/11/byoc-not-the-future-of-cloud-services-but-a-pillar-of-an-everywhere-platform monthly 0.5 2024-09-11 https://jack-vanlightly.com/blog/2024/8/23/constraints-breed-innovation-but-so-does-tenacity monthly 0.5 2024-08-23 https://jack-vanlightly.com/blog/2024/8/22/table-format-comparisons-streaming-ingest-of-row-level-operations monthly 0.5 2024-10-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e34a0172-f127-4324-83ac-a2c07a0822be/HudiFileSlicesSmall.png Blog - Table format comparisons - Streaming ingest of row-level operations - Make it stand out Fig 1. Depicts two file slices of a single file group in a MOR table. A compaction job compacts a file slice with three log files into a new base file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6f599b7a-9fd5-4fc2-9145-13242c7bfcd8/HudiRowLevelIngestSmall.png Blog - Table format comparisons - Streaming ingest of row-level operations - Make it stand out Fig 2. How batches of row-level operations translate to files in a file group. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4d3949cc-4366-4b93-a938-f20390252b5a/PaimonRowLevelIngestSmall.png Blog - Table format comparisons - Streaming ingest of row-level operations - Make it stand out Fig 3. Paimon natively stores data as row-level operations. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4653b74f-db8e-4476-b8e3-6a2a44aff110/DeltaIcebergRowLevelIngestSmall.png Blog - Table format comparisons - Streaming ingest of row-level operations - Make it stand out Fig 4. The data and delete files written with Iceberg/Delta Lake. https://jack-vanlightly.com/blog/2024/8/13/table-format-comparisons-append-only-tables-and-incremental-reads monthly 0.5 2024-08-13 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/09f8270c-fbf0-4614-bb82-ef0a0c7a2f17/FlinkTopologySmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 1. A simplified view of a Flink topology that incrementally reads from a table. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b4c19e41-f0bf-4381-a54f-d6771d7af7c5/IncrementalReadIcebergSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 2. A compute engine incrementally reads new data files by reading the next snapshot in the log of snapshots and searched for manifest entries with the ADDED status. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1d2202e0-0b41-4804-9f15-b6d0a5b85d45/IcebergBucketPartitionerSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 3. Partitioning data with the bucket(n, col) transform. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4853b15a-10ab-44e5-82e6-4abe5c162d52/IncrementalReadDeltaSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 4. A compute engine incrementally reads new data files by reading the log of deltas searching for AddFiles actions. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fc568f81-6ef9-4594-ab97-59a882a4f55a/IncrementalReadHudiSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 5. A compute engine incrementally reads new data log files by reading the timeline of a MOR table. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5cfcf325-e0df-4298-b4e3-4bbe5a691412/IncrementalReadPaimonSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 6. A compute engine incrementally reads new data files by reading the next snapshot in the log of snapshots and searching the delta manifest list for manifests with entries with the ADDED status. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fff78323-0bb4-455d-9a57-9ab568407d8a/PartitionsAndBucketsSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 7. Each partition is sharded by a number buckets. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/390dc664-08ee-497f-b761-57d7b54ec528/BucketShufflePaimonSmall.png Blog - Table format comparisons - Append-only tables and incremental reads - Make it stand out Fig 8. Append Queues. Rows are mapped to buckets by a bucket key (one of the table columns) and data is shuffled to writers by bucket. This produces a similar topology to Iceberg when its bucket partition transform is used, however, Paimon is able to separate partitioning from this bucket based sharding. https://jack-vanlightly.com/blog/2024/8/7/table-format-comparisons-how-do-the-table-formats-represent-the-canonical-set-of-files monthly 0.5 2024-08-13 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1c7d9b4a-04d0-434b-9ab8-de57ced1698d/DeltaLogSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 1. Depicts a Delta Log with a set of four “Add/Remove Files” actions. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8d77c17c-961f-4629-bfe0-04db0a7ff79c/DeltaLog2Small.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 2. A table scan would only need to read three metadata files of the log: 00005.checkpoint.parquet, 00006.json and 00007.json. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b1d53781-40aa-41b7-8366-3de40c64ba17/HudiFileGroupsSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 3. A table is divided into a number of file groups. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8ab4e0f9-7541-49f9-8447-e5474fc5b52c/HudiFileSlicesSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 4. Depicts two file slices of a MOR table. A compaction job compacts a file slice with three log files into a new base file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f4840d50-eebe-4eb1-9cb0-969ae4bc68f9/HudiTimelineSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 5. An Apache Hudi timeline of a COW table. Depicts a table with 5 file groups and a log of completed commit instants where each instant lists a set of added file slices. The requested and inflight instants are omitted here. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2786ec10-db32-4268-a211-fb61443b2e7b/IcebergSnapshotSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 6. Each table version is stored as a snapshot. A snapshot contains a single manifest-list file, which contains a list of entries that point to a set of manifest files. Each manifest file contains a list of entries that point to a set of data files that were added, deleted or existing. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/177a1808-2692-4c20-9ebe-62e258c4ea2e/MetadataFilesSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 7. An Iceberg client will start by consulting the catalog to learn of the current metadata file. It then loads that metadata file which contains all live snapshots. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/26177874-86cc-4376-9cdc-54f3328693af/PaimonSnapshotsSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 8. Snapshot-4 is the current snapshot. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c06d8d7a-ca4e-44b9-bbda-7449e3cead99/PaimonSnapshotTreeOfFilesSmall.png Blog - Table format comparisons - How do the table formats represent the canonical set of files? - Make it stand out Fig 9. Depicts a snapshot that was written by a compaction operation that deleted two existing data files and rewrote them as a single data file. https://jack-vanlightly.com/blog/2024/6/10/a-cost-analysis-of-replication-vs-s3-express-one-zone-in-transactional-data-systems monthly 0.5 2024-06-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/281dd189-b66c-4982-bd65-78a27b9c35f8/pg_and_sk.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 1. The fault-tolerant write-ahead-log in the Neon architecture. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7b3c59df-0a88-4c42-8d4a-a17df813fa7d/ReplicationWriteCacheSmall.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 2. Data lands on a leader node and is replicated to followers synchronously. Data is offloaded to long-term storage asynchronously (Kora/Kafka). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fcc09075-66b2-4dca-8d4a-5da749fda14b/S3WriteCacheSmall.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 3. Clients interact with zone-local proxies, which in turn write to S3 Express One Zone synchronously. No cross-AZ data transfer occurs, avoiding this large source of cloud costs. Data is compacted into larger objects and offloaded to long-term storage in S3 Standard asynchronously. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aaadfc12-1aa7-4d87-8705-8204bca71982/five-write-caches.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 4. Single-zone replication deployments have no cross-AZ traffic. However, multi-zone deployments have cross-AZ traffic for both producer and replication traffic. The proxy-object-store architecture avoids all cross-AZ by ensuring that clients interact with zone-local proxies. Multi-AZ is achieved by writing to multiple buckets where each bucket is in a different zone. There is no cost for writing to a bucket in a different AZ. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fa5a39b8-f43f-4bc6-9228-f2050e9c848d/ReplicationResources.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 5. Storage and networking breakdown of replication. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/27f9f538-2bd0-4301-84e8-a27d4bac463b/blog_final_1_raw_net_cost_and_discount.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 6. The relationship between throughput and data transfer cost, and the impact of discounts. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/92166307-3b13-4f49-aa93-ffcb9be9fd01/blog_final_2_buffering_objsize.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 7. The relationship between buffering time, put rate and object/request size. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e9d87fee-9eaf-4757-b309-06982f683b73/blog_final_3_s3_costs_kb.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 8. How request/object size affects Express One Zone request costs for a 1 GB/s workload. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ef9cc268-952a-4651-adb8-aa99ecd00087/blog_final_4_fixed_buffering.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 9. Variable throughput, with fixed proxy count, max buffering time of 10ms and max 512 KB requests. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/05bf0477-bb21-4a0d-b4bb-40e1b1aaef81/blog_final_5_autoscaling.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ce73363b-6cbc-451c-82ee-53bbebd9c41f/blog_final_6_fixed_buffering.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 10. Auto-scaling down to 3 proxies improves the situation at lower throughputs. By 150 MB/s we reach the theoretical best cost of $0.015 per GB for 3-zone S3 Express One Zone. Lowering the number of proxies to three all the way up to 1 GB/s would make no difference to the request costs (with max 10ms buffering). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/01acb3ed-44c0-4226-ab3b-1844ad8c624d/blog_final_7_fixed_buffering_low_tp.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 11. 1-20 MB/s throughput with 3 proxies. The cost-per-GB rises sharply as the throughput drops, due to the flat baseline cost based on the 10 ms buffering time. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f29fd884-3987-466f-bff8-8379e7a46eb9/blog_final_8_storage.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 12. Storage costs https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cad25c46-c225-4d71-ada0-e4dff586eacb/blog_final_9_saz_1_50.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/793d5f3d-cec7-46b4-b138-9f5c998c0504/blog_final_10_saz_500_1000.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 13. Single-AZ total cost. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aa43a71b-07a8-40cb-8be8-bc5dc370ad6f/blog_final_11_maz_1_50.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/eb1eea17-87c4-4cb7-8ad6-10cebc1081c8/blog_final_12_maz_500_1000.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 14. Multi-AZ total cost. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d712dbac-f6e5-4e45-9d18-9861b377e5fe/blog_final_13_var_buffering.png Blog - A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems - Make it stand out Fig 15. Impact of buffering time on costs for Express One Zone vs Standard. https://jack-vanlightly.com/blog/2024/5/7/learning-and-reviewing-system-internals-tactics-and-psychology monthly 0.5 2024-05-07 https://jack-vanlightly.com/blog/2024/5/2/hybrid-transactional-analytical-storage monthly 0.5 2024-05-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f7bb03e3-aeff-4d76-bf8e-aa165e004a74/headless-data-architecture.png Blog - Hybrid Transactional/Analytical Storage - Make it stand out Fig 1. The headless data architecture of bring-you-own-compute/query-engines over shared tables. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/20272113-4408-4be5-b624-7264a53da2d9/htas1.png Blog - Hybrid Transactional/Analytical Storage - Make it stand out Fig 2. Kora integrates stream and table storage into a single data set (the feature known as Tableflow). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/396d8c8c-e757-4f9a-b9f4-322fa93bcdad/htas2.png Blog - Hybrid Transactional/Analytical Storage - Make it stand out Fig 3. Regular topics can straddle the transactional and analytical divide, as well as the row-oriented and column-oriented divide. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c4d8fdd2-fbba-4e65-a132-a1c6fc6141b7/htas3.png Blog - Hybrid Transactional/Analytical Storage - Make it stand out Fig 4. Freight cluster topics provide a low-cost option for high volume topics. On the analytics side, this object storage stream can be consumed with streaming semantics (via the Kafka API), or as Iceberg tables for a more incremental, columnar approach. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0c1cd8a8-df14-415c-9834-1b0db122ae90/htas4.png Blog - Hybrid Transactional/Analytical Storage - Make it stand out Fig 5. Customers choose the API and storage format that suits them best. https://jack-vanlightly.com/blog/2024/3/26/the-sisyphean-struggle-and-the-new-era-of-data-infrastructure monthly 0.5 2024-05-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b41f6210-44a7-4777-99ba-ebdd81219566/SpinalTapMeme.jpg Blog - The Sisyphean struggle and the new era of data infrastructure - Make it stand out (Spinal Tap reference for those of you not born in the prior millenia) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a7c8f819-0338-417c-a1fd-16f5210e1a2c/Sisyphean.png Blog - The Sisyphean struggle and the new era of data infrastructure - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2024/3/19/tableflow-the-stream-table-kafka-iceberg-duality monthly 0.5 2024-05-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6e82d030-7164-48cf-8795-e8fecda3dd4c/JackTweet.png Blog - Tableflow: the stream/table, Kafka/Iceberg duality - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6eb25766-afb5-4bbc-b420-997598fd5301/snakes_cropped_small.png Blog - Tableflow: the stream/table, Kafka/Iceberg duality - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/36d88ba2-8527-4518-9170-535e3025b956/apps_over_storage.png Blog - Tableflow: the stream/table, Kafka/Iceberg duality - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fc6a8aaf-6172-45e0-bc6c-6c2fb9d99abb/KafkaApiMemeSmall.png Blog - Tableflow: the stream/table, Kafka/Iceberg duality - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2024/3/13/the-beauty-of-writing monthly 0.5 2024-03-26 https://jack-vanlightly.com/blog/2023/11/29/s3-express-one-zone-not-quite-what-i-hoped-for monthly 0.5 2024-05-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/138cf7fa-f051-43ba-92f9-cbfaee780e14/Choices1.png Blog - S3 Express One Zone, not quite what I hoped for - Make it stand out Fig 1. The data systems from chapters 1-5 of Architecture of Serverless Data Systems, and their choices for integrating cloud object storage. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/52e733fa-cf52-444e-958d-14ee44629b25/HolyGrail.png Blog - S3 Express One Zone, not quite what I hoped for - Make it stand out Fig 2. The holy grail is still out there. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/db1547ac-b2ba-4fbe-af7e-d6c7c787f624/ChoicesAfter.png Blog - S3 Express One Zone, not quite what I hoped for - Make it stand out Fig 3. System builders have a 4th choice for integrating cloud object storage. https://jack-vanlightly.com/blog/2023/11/14/the-architecture-of-serverless-data-systems monthly 0.5 2024-03-12 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/086539d6-ae55-4894-9d34-d1c964657761/AOSA_book_small.jpg Blog - The Architecture of Serverless Data Systems - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/04b7f0d0-328d-4206-ac98-4c4b4957ec10/BigQueryFig1_small.png Blog - The Architecture of Serverless Data Systems - Make it stand out Fig 1. from Dremel, A Decade of Interactive SQL Analysis at Web Scale. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d5556b84-f949-4a67-aa17-2e1fc457522c/dreier.png Blog - The Architecture of Serverless Data Systems - Make it stand out Fig 2. The advances in throughput across SSDs, memory, and networking. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/085f4cc5-fcce-41ae-8cbd-04c6b4dab707/fast_storage_slow_object_storage.png Blog - The Architecture of Serverless Data Systems - Make it stand out Fig 3. Choices for integrating (or not) cheap, durable cloud object storage. https://jack-vanlightly.com/blog/2023/10/10/the-importance-of-liveness-properties-part-2 monthly 0.5 2023-10-11 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3900fe94-2cfb-44e5-97c0-f99a31c78921/gossip_with_gen.png Blog - The importance of liveness properties (with TLA+ Part 2) - Make it stand out Fig 1. A gossip exchange where a node refutes its deadness. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/734f4fff-54e2-48d1-8e78-d471c24f5134/StateConstraintWarning.png Blog - The importance of liveness properties (with TLA+ Part 2) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3ab3b1a2-f9d2-4dd2-9e5b-262f0a1dc10b/liveness-vs-invalid-safety.png Blog - The importance of liveness properties (with TLA+ Part 2) - Make it stand out Fig 2. The invalid safety property detects that the darker green state at the bottom is reachable, but only the liveness property can detect the behaviors where that state is unreachable. https://jack-vanlightly.com/blog/2023/10/10/the-importance-of-liveness-properties-part1 monthly 0.5 2023-10-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/73e9036d-9ed6-42b5-8cae-e8359537115c/gossip.png Blog - The importance of liveness properties (with TLA+ Part 1) - Make it stand out Fig 1. Node 1 gossips its knowledge of the group to node 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3fde4c10-5407-40dd-aeb0-423c61bb8140/no-one-listens.PNG Blog - The importance of liveness properties (with TLA+ Part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2023/10/10/a-primer-on-formal-verification-and-tla monthly 0.5 2023-10-09 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a14217f8-5d1a-4f7a-9ba9-d1be76a5a4ca/states.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 1. Two different states of a clock specification. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5d486d67-f9b8-4dae-8ba8-471b3d3de8e5/TwoCounterStateSpaceGrey.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 2: State transitions for two counters, up to the value of 2. There are 9 unique states. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/967fc27f-1ae4-4784-9ab3-5303f4eaf776/TwoCounterStateSpacePathsGrey.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 3: Six different paths through the state space https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/25aa8699-5f75-4f69-886b-402cb92d7f9e/ClockStepSmall.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 4: A state transition for our clock specification. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/121cf106-f1de-4834-b2bc-a493e4ca09b7/ClockSequenceGrey.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 5: Clock graph is a chain as each state has only one next state https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9d8233e5-b1b0-4467-a205-4a61a2eb042b/JumpyClockSequenceGrey.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 6: Jumpy clock graph, same number of states but many more possible transitions. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/abf369b6-fa3a-42d7-8bad-8e24abd1054a/FocusOnProperties.png Blog - A primer on formal verification and TLA+ - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2769a997-4eff-45e9-af35-93adb79a162c/LeaderCandidateCompletenessPropertyTLA.png Blog - A primer on formal verification and TLA+ - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1976cdcd-8cdf-4c6d-bcef-30dfbaf23f48/property_graph_small.png Blog - A primer on formal verification and TLA+ - Make it stand out Fig 7. Kafka replication protocol safety properties. https://jack-vanlightly.com/blog/2023/10/2/the-advantages-of-queues-on-logs monthly 0.5 2023-10-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ee609a0a-e2ce-48cf-a48f-4d6e7c0fe199/Queue.png Blog - The advantages of queues on logs - Make it stand out Fig 1. Queue operations https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d2b8c9f3-e77f-4b60-a7dd-5e378655c5bb/QueueParallelConsumptionSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 2. Consumers compete to read items https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9c24ed4e-a4df-4577-bfe6-de2997588c99/QueueParallelConsumptionOrderingSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 3. Simple load distribution over multiple consumers loses processing order guarantees. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f8e2b374-780c-4220-8391-b6a8c45cbd94/LogSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 4. Log operations https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e41fce74-3e57-408a-aa79-c3791df0d8ce/LogGCSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 5. Log growth is controlled by garbage collecting the log tail. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a3791622-d9f1-4d34-b557-5db63efd3270/LogConsumersSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 6. Multiple independent consumers. Each consumer consumes every item. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1cca1eda-1789-49f9-b0b9-ed36c36d13aa/QueueMultipleLogicalConsumersSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 7. A single queue cannot support multiple logical consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ddbf79a4-0de7-4727-831e-3d97cc73eaae/MultipleQueuesForMultipleLogicalConsumersSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 8. Two queues for two logical consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/95639d2b-292e-4b17-b52a-696ee87f3b0c/OneLogTwoLogicalConsumersSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 9. One log supports multiple logical consumers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f0214d6e-6bb5-4614-b73b-ebbe37694f20/LogPartitioningSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 10. Partitioned logs allow for parallel consumption by a single logical consumer. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/92c7166b-3194-4cff-a111-7fa22e24e1de/QueueItemStatesSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 11. Each message has a state machine. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b6ad33f2-6f12-4605-8ca5-1c0b2c24fac4/QueueOperationsSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 12. A queue with different messages in different states. Next message 3 will be processed after message 4. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/be5bda34-59b4-4436-8033-396145f0c56b/LogReadProcessAdvanceCursorSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 13. Reads and cursor advancement split into separate operations to achieve at-least-once. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ca3d992b-297f-42d1-9379-773f9a8f5c61/QueueOnLog.png Blog - The advantages of queues on logs - Make it stand out Fig 14. A queue can just be metadata that points into a log. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/47777221-00ab-4fe2-a36f-af0512cae4ac/QueueOnLogPerSubSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 15. Multiple queues over the same log. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b4dbcd94-944a-4491-84f0-c927556e014e/MixedSemanticsSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 16. Producers append, consumers can choose log or queue semantics. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6b82510e-cc9b-4fc8-a67c-f125c67b28ff/NonBlockingKeyedDispatchSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 18. Non-blocking keyed dispatch. Messages of the same key get delivered to the same consumer. Requires a session-based protocol. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3558d0fb-e850-477e-b4c1-ff128216cf19/BlockingKeyedDispatchSmall.png Blog - The advantages of queues on logs - Make it stand out Fig 19. Blocking keyed dispatch. The queue only dispatches messages of a given key if there are no messages of that key outstanding. https://jack-vanlightly.com/blog/2023/9/25/on-the-future-of-cloud-services-and-byoc monthly 0.5 2024-09-11 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/04bd7049-a70b-4b7d-92b7-6a48c4c9fd07/encryption.png Blog - On the future of cloud services and BYOC - Make it stand out Encryption comes in many forms and can be used to limit access to data by third parties. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/95f0ca65-0ba3-4519-a486-ff1f7ce4ca2c/networking.png Blog - On the future of cloud services and BYOC - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9da83950-8e14-4665-8197-ba00cda46927/BYOC_TCO.png Blog - On the future of cloud services and BYOC - Make it stand out BYOC pricing is based on a subscription to the software, not the underlying infrastructure it needs nor does it include the budget needed for the additional overheads of private networking and security. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/23a552c2-f49b-426a-81f8-c73219875d80/BYOC_Billing.png Blog - On the future of cloud services and BYOC - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/20c548e0-a8d9-403f-a4cb-708119cceea4/ST-vs-MT.png Blog - On the future of cloud services and BYOC - Make it stand out Consider the efficiency at scale of thousands of single-tenant clusters each sized with enough margin for steady performance compared to a large-scale multi-tenant system based on resource pooling. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/66cd92c7-b956-4bee-82f9-b0c951cedc95/MultiTenancy.png Blog - On the future of cloud services and BYOC - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8f414cf3-cbc0-41d7-82f7-97493f9e79da/SingleStackFallacySmall.png Blog - On the future of cloud services and BYOC - Make it stand out The single-cloud-stack: bringing single-tenancy to any VPC. https://jack-vanlightly.com/blog/2023/8/17/kafka-kip-966-fixing-the-last-replica-standing-issue monthly 0.5 2023-09-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3d053a4b-7220-42ba-9c49-4b2908caf523/ISR_ELR_Completeness.png Blog - Kafka KIP-966 - Fixing the Last Replica Standing issue - Make it stand out Fig 1. The view of a replica from the perspective of the controller. https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda-performance-do-the-claims-add-up monthly 0.5 2023-10-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/11b84a42-d108-4c36-a2b8-4e7d2402ca6f/TLS_500MB_4vs50prod_labelled.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 1. Redpanda end-to-end latencies with 50 producers reach 24 seconds. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9722c34b-359c-4e2c-83da-b2a616caeacc/TLS_1000MB_AllWorkloadsProducethroughputMBs.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 2. Redpanda was unable to reach 1000 MB/s with TLS. With 50 producers it only managed 850 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b2a38a51-e9f1-45b4-9b45-d0665c86b4fb/rp-long-running-p50.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 3. Redpanda p50-p90 end-to-end latency jumped after 12 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1e6a945c-d233-49ed-9326-66cfce9bb0dc/rp-long-running-p95.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 4. Redpanda end-to-end tail latencies shot up to tens of seconds after 12 hours of sustained load. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e880d8f1-056e-49b9-8914-8f22803636e7/rp-long-running-starving-disk.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 5. One Redpanda broker starts reporting being starved for disk after 12 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1a445f95-1b18-4f29-b5ed-73298536bd0e/rp-long-running-rp1-disk-latency.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 6. The average write and read time per op jumped on one broker after 12 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b506e333-a3b7-47ce-a565-741805282d81/kafka-long-running-p99.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 7. Kafka p99 end-to-end latencies for a 24 hour period. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2675c118-1229-425f-a43f-34a1c208e3e6/kafka-long-running-p9999.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 8. Kafka p99.99 end-to-end latencies for a 24 hour period. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3c82237a-7203-4c5f-8ded-04ddd7d44804/rp-ret-limit-p95.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 9. The Redpanda stepwise increase in end-to-end latency once the brokers start deleting segment files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3fcaa171-f3bb-4e96-9a4e-3ecfb23d6174/rp-ret-limit-retest.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 10. Redpanda end-to-end latency results for the 1 GB/s benchmark, when recording latencies after the data retention limit had come into effect. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f162f37a-0b9c-4723-9d48-098bcdf0f66b/record-keys-500MBs-throughput.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 11. Kafka was able to reach the target 500 MB/s with 100 producers using record keys. Redpanda topped out at 330 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/29ab31c2-16b5-4dda-8c31-c0c01fafabb4/TLS_e2elat_200MB_400_800parts.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 12. Redpanda struggled with more producers which used record keys for message ordering. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f79fc39d-7a45-4227-93b7-ad7f9cbca1e6/1-2-GBs-throughput.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 13. Redpanda only reached 1400 MB/s with acks=1, whereas Kafka reached 1900 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2ff758e9-39da-4c9b-a8fc-27274db27671/kafka-drive-writes-1000-1900MB.png Blog - Kafka vs Redpanda Performance - Do the claims add up? - Make it stand out Fig 14. When Kafka topped out at 1.9 GB/s, it had actually reached the 2 GB/s physical limit of the NVMe drives on the i3en.6xlarge. https://jack-vanlightly.com/blog/2023/5/9/is-sequential-io-dead-in-the-era-of-the-nvme-drive monthly 0.5 2023-05-09 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a8a08f4c-a172-45e2-b466-1386d121876f/BK-WAL-LongTermStorage.png Blog - Is sequential IO dead in the era of the NVMe drive? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9e84fcac-a6a8-4eae-ac5e-dec171ddd77c/NAND-block.png Blog - Is sequential IO dead in the era of the NVMe drive? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c1d3fbfb-aada-4802-b0e0-51eff9831487/GC.png Blog - Is sequential IO dead in the era of the NVMe drive? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f7886d31-b1fa-4806-86a4-5375cc8a1451/SequentialRandomBlocks.png Blog - Is sequential IO dead in the era of the NVMe drive? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f3614047-d85a-4868-a584-9fd5c5854f35/SlowdownFactor.png Blog - Is sequential IO dead in the era of the NVMe drive? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ae15e30f-c75e-44d7-bcad-09b19f4f977c/RemappingPaper.png Blog - Is sequential IO dead in the era of the NVMe drive? - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-need-fsync-to-be-safe monthly 0.5 2025-12-09 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3d299bb6-f546-46eb-86a7-e865751047d0/RaftReplicatedStateMachine.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Fig 1. Raft is a state-machine replication (SMR) algorithm which consists of a cluster of state machines which sit on a replicated log. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d74fb010-cc20-45c9-8a74-716da5c2245b/RaftMajorityQuorums1.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Fig 2. Three possible quorums of two nodes: {n1,n2}, {n2,n3} and {n1,n3}. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/54a2d601-9559-44cb-a365-8b996a6e4a9c/RaftMajorityQuorums2.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Fig 3. There exist no two replication and election quorums that do not overlap. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6e02e519-841a-40df-af0b-37f3b4c59bb3/RaftExample1.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Raft example step1: all is good. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/249f45cf-5f1e-4472-b617-ba7fa521f1be/RaftExample2.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Raft example step2: broker 1 loses m2 and m3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1d73d4d9-e074-4e4c-90ca-7484298de97a/RaftExample3.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Raft example step3: Broker 1 is back with only m1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/45f3a107-59a5-4eee-bcc8-035d83ff2bfd/RaftExample4.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Raft example step 4: Oh no! An election and replication quorum that don’t overlap! https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5e0e8d77-dfe7-4b21-8de3-3d24bcf41d68/RaftExample5.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Raft example step 5: Broker 2 truncates its log to m1. The cluster just lost two acknowledged entries. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/835bbb6a-5adf-4d75-b142-b068bae07dfc/KafkaExample1.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Kafka example step 1: All is good though the ISR is down to two to make this similar to the “Raft without fsync” example. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/799cae20-9b0a-427d-872b-bd9132ec388b/KafkaExample2.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Kafka example 2: The server of broker 1 goes offline, taking the page cache with it and losing m2 and m3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/84708776-ac9f-4476-9346-18445d7bc268/KafkaExample3.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Kafka example step 3: Broker 1 comes back without the unflushed messages m2 and m3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8eaaaacd-970c-4ce4-b591-0d779b5f08f2/KafkaExample4.png Blog - Why Apache Kafka doesn't need fsync to be safe - Make it stand out Kafka example step 4: Broker 1 fetches from broker 2 until it is caught up and rejoins the ISR. https://jack-vanlightly.com/blog/2022/12/28/windows-shortcut-to-invert-colours-that-works monthly 0.5 2022-12-28 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ec54f993-1c87-46ca-805c-53440a4bffef/ColorFilter.PNG Blog - Windows shortcut to invert colours that works - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/blog/2022/6/9/applying-flexible-paxos-to-raft monthly 0.5 2022-06-14 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7ffd3c6e-a778-404e-b8a9-3bc54b1beb86/Majorities.PNG Blog - Applying Flexible Paxos to Raft - Make it stand out All majority quorums overlap https://jack-vanlightly.com/blog/2022/1/25/write-for-others-but-mostly-for-yourself monthly 0.5 2024-03-26 https://jack-vanlightly.com/blog/2021/12/9/tweaking-the-bookkeeper-protocol-unbounded-ledgers monthly 0.5 2021-12-12 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7946687a-3e61-43b6-8b89-aa6ad8faa173/fragments-colored.jpg Blog - Tweaking the BookKeeper protocol - Unbounded Ledgers - Make it stand out Fig 1. A log-of-ledgers vs an unbounded ledger in terms of fragments which are themselves logs of entries. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1fa636d6-0f6e-4c55-ab8e-ebfc4b1f114d/bk-term.jpg Blog - Tweaking the BookKeeper protocol - Unbounded Ledgers - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/338781d7-cb33-4f11-8f85-7b2179333d11/term-and-status.jpg Blog - Tweaking the BookKeeper protocol - Unbounded Ledgers - Make it stand out Fig 2. The ledger lifecycle now status and term. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/53307f2a-ec70-41ce-981e-f60c91df97e6/term-metadata.jpg Blog - Tweaking the BookKeeper protocol - Unbounded Ledgers - Make it stand out Fig 3. Metadata gets a new field for term. The ensembles field now needs to be rethought as it will likely grow too large. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d66a5739-631e-4c86-b3db-0c5ce7c0af69/term-fencing-off2.jpg Blog - Tweaking the BookKeeper protocol - Unbounded Ledgers - Make it stand out Fig 4. Fencing involves including the new term instead of the fencing flag. https://jack-vanlightly.com/blog/2021/12/7/tweaking-the-bookkeeper-protocol-guaranteeing-write-quorum monthly 0.5 2021-12-12 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c8e4aaca-4ff3-4b5d-b8bd-1729b5fb6422/wq-aq-laughing-colored.jpg Blog - Tweaking the BookKeeper protocol - Guaranteeing write quorum - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cb1b15e8-88d4-4396-a0d7-4351039b07e1/aq-is-minimum-guarantee.png Blog - Tweaking the BookKeeper protocol - Guaranteeing write quorum - Make it stand out Fig 1. A series of ensemble changes leaves multiple entries at AQ. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8b043282-3b11-4923-bd7f-7cda76109ddc/AQ-entries-coloured.png Blog - Tweaking the BookKeeper protocol - Guaranteeing write quorum - Make it stand out Fig 2. WQ=3, AQ=2. Top: entries 0 and 2 left at AQ after ensemble changes. Bottom: all entries left at AQ is also a legal state of a ledger with WQ=3, AQ=2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/654d7526-3b5c-4a6e-8286-ccb4e746a746/broken-wq-guarantee-change.png Blog - Tweaking the BookKeeper protocol - Guaranteeing write quorum - Make it stand out Fig 3. By moving acknowledged entries into the next ensemble, and changing the ensemble until its ensemble is completely disjoint from the bookies that host the committed entries, the committed entries become lost. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aceacadf-ae10-47dd-8379-9412c5705267/wq-has-guarantees.png Blog - Tweaking the BookKeeper protocol - Guaranteeing write quorum - Make it stand out Fig 4. A series of ensemble changes where entries starting at the first entry below AQ are moved into the next fragment. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/00d85c52-ce6a-4bb3-9527-3d01bdc1699b/bookies-can-be-pinned.png Blog - Tweaking the BookKeeper protocol - Guaranteeing write quorum - Make it stand out Fig 5. Ensemble changes that would have replaced bookies that hosted committed entries below WQ are postponed until they no longer host committed but not fully replicated entries. https://jack-vanlightly.com/blog/2021/10/9/learn-about-tla-and-the-formal-verification-of-apache-bookkeeper monthly 0.5 2021-11-23 https://jack-vanlightly.com/blog/2021/10/9/posts-in-wrote-on-the-rabbitmq-blog-in-2020 monthly 0.5 2021-10-27 https://jack-vanlightly.com/blog/2021/10/9/kafka-and-rabbitmq-blog-posts-i-wrote-elsewhere-in-2019 monthly 0.5 2021-10-09 https://jack-vanlightly.com/blog/2020/5/26/with-great-observation-comes-great-insight monthly 0.5 2020-05-26 https://jack-vanlightly.com/blog/2020/5/14/why-im-not-writing-much-on-my-blog-these-days monthly 0.5 2020-05-14 https://jack-vanlightly.com/blog/2019/9/4/a-look-at-multi-topic-subscriptions-with-apache-pulsar monthly 0.5 2019-09-14 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1567607197625-5KWHRWNS1MMIJ4K1B3EK/MultiTopicSubRoundRobin.jpg Blog - A Look at Multi-Topic Subscriptions with Apache Pulsar Fig 1: The messages in each topic when sending in a round-robin fashion https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1567686526981-7W5BTI83RME91D19ISN7/BestEffortCrossTopic.jpg Blog - A Look at Multi-Topic Subscriptions with Apache Pulsar Fig 2: Best-effort timestamp strategy https://jack-vanlightly.com/blog/2019/2/9/building-a-simple-distributed-system-logging monthly 0.5 2019-03-11 https://jack-vanlightly.com/blog/2019/2/1/building-a-simple-distributed-system-the-implementation monthly 0.5 2019-02-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1549101641083-82M5CV8BAPHVHGJBX6SD/RebalanserImplementation1.png Blog - Building A "Simple" Distributed System - The Implementation Fig 1. The state machine. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1549100469840-D0CF9ZDLMMOE92VIWWN3/RebalanserImplementation2.png Blog - Building A "Simple" Distributed System - The Implementation Fig 2. Leader state https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1549100513326-ZYRRKG68BQSPES0CMSH9/RebalanserImplementation3.png Blog - Building A "Simple" Distributed System - The Implementation Fig 3. Follower State https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1549098839408-CLG0P7J6GLNJVJ3T1ZZ5/RebalanserImplementation4.png Blog - Building A "Simple" Distributed System - The Implementation Fig 4. Close propagates cancellations through the code. All ZooKeeper operations perform retries which are cancellable. https://jack-vanlightly.com/blog/2019/1/27/building-a-simple-distributed-system-formal-verification monthly 0.5 2019-02-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548669558564-WAT7OUUM2XMLTW99JRB4/ResourceBarrierSpec1.png Blog - Building A "Simple" Distributed System - Formal Verification https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548679668653-UEW7XAQY1LVZ9IK6H9CP/ModelCheckerResults.PNG Blog - Building A "Simple" Distributed System - Formal Verification https://jack-vanlightly.com/blog/2019/1/27/building-a-simple-distributed-system-the-protocol monthly 0.5 2019-02-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548583387338-U9EHN9VITAJBCJ9946ES/LeaderResourceBarrier1.png Blog - Building A "Simple" Distributed System - The Protocol Fig 1. The registry stores all meta-data about a Rebalanser group and acts as a communication medium between nodes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548583739929-LLZ2AG1L70Z91W7YQP7P/LeaderResourceBarrier2.png Blog - Building A "Simple" Distributed System - The Protocol Fig 2. The interactions between nodes and the registry https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548588412747-RFC4Q1HGMY7AWLLSJLDZ/LeaderResourceBarrier3.png Blog - Building A "Simple" Distributed System - The Protocol Fig 3. A flow diagram that describes the sequence of actions. Failure can occur at any moment and is described in a separate diagram. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548584240667-ZONUQDKRI36Q0DDYQPAC/LeaderResourceBarrier4.png Blog - Building A "Simple" Distributed System - The Protocol Fig 4. Nodes stop resource access and revert to no role when detecting that the registry is unreachable. https://jack-vanlightly.com/blog/2019/1/25/building-a-simple-distributed-system-the-what monthly 0.5 2019-02-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548492461713-4RXEHVCEFHD5427H2JXS/ConsumerGroupV1.png Blog - Building a "Simple" Distributed System - The What Fig 1. The Rebalanser group starts off with two applications and five resources. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548492563656-SWTZJKKGFKD4QR8JYXSS/ConsumerGroupV2.png Blog - Building a "Simple" Distributed System - The What Fig 2. A new application is added to the group. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548493089355-W026HJ5N3D0P06NBZGLK/ConsumerGroupV3.png Blog - Building a "Simple" Distributed System - The What Fig 3. Two new resources are added. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548493266827-561195GUNZT7XCCEJXDM/ConsumerGroupV4.png Blog - Building a "Simple" Distributed System - The What Fig 4. App 2 fails or is shutdown. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548493531459-A7UOZ6M1SS542795U1CX/image-asset.png Blog - Building a "Simple" Distributed System - The What Fig 5. A Rebalanser group of two applications and one resource. https://jack-vanlightly.com/blog/2018/11/20/quorum-queues-making-rabbitmq-more-competitive monthly 0.5 2019-04-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1542721655701-C0DVJ796RWY0XVQLWX7N/image-asset.png Blog - Quorum Queues - Making RabbitMQ More Competitive in Reliable Messaging https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1542724839246-D4LN8JC5NCHK0APGWS3R/MajorityWrite.png Blog - Quorum Queues - Making RabbitMQ More Competitive in Reliable Messaging Fig 2. The leader acknowledges the write to the client once it has a quorum. In this case it only needs two followers to confirm they have the entry. https://jack-vanlightly.com/blog/2018/11/14/why-i-am-not-a-fan-of-the-rabbitmq-sharding-plugin monthly 0.5 2018-11-25 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1542228937555-O942QOSD3LVV6DWOJOFW/PartitionedQueue.png Blog - Why I Am Not a Fan of the RabbitMQ Sharding Plugin https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1542274012268-P3KR2AF33NF92UEMFH3R/ShardingPlugin.PNG Blog - Why I Am Not a Fan of the RabbitMQ Sharding Plugin Four consumers connect to the same broker, leaving queues on other brokers unconsumed. https://jack-vanlightly.com/blog/2018/10/25/testing-producer-deduplication-in-apache-kafka-and-apache-pulsar monthly 0.5 2019-04-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1541186524690-LE7TRVBH7UDFSXULTQC2/image-asset.png Blog - Testing Producer Deduplication in Apache Kafka and Apache Pulsar https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-messages-on-an-apache-pulsar-cluster monthly 0.5 2018-11-25 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1540076882585-S3JCBOEZETWGEU4EAUFP/PulsarBlockade.png Blog - How to (not) Lose Messages on an Apache Pulsar Cluster https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works monthly 0.5 2021-12-15 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538489997612-OYXFLJVUBYP4TOQDPHW9/LayersOfAbstraction.png Blog - Understanding How Apache Pulsar Works Fig 1. Layers of abstraction https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538490975658-WSDKSTF2F07QFPMCPJ5H/TopicsSubscriptions.png Blog - Understanding How Apache Pulsar Works Fig 2. Topics and Subscriptions https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538572211110-2C6EPHJTLLAOTVMVHXM1/PulsarBkZkCluster.png Blog - Understanding How Apache Pulsar Works Fig 3. Apache Pulsar, BookKeeper and ZooKeeper working together https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538497275047-DGDCYE3LX945H79I1MOI/LogLedgerFragmentEntry.png Blog - Understanding How Apache Pulsar Works Fig 4. Entries at the bottom https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538497570494-EIP81VKC3W4VZQBWCWWH/SubTopicLedgerFragmentBookie.png Blog - Understanding How Apache Pulsar Works Fig 5. Apache Pulsar, Apache BookKeeper and Apache Zookeeper working together https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538556992096-2Y1LIUC0GI3XKGGM064I/FragmentEnsemble.png Blog - Understanding How Apache Pulsar Works Fig 6. A fragment of 8 entries stored across an ensemble of 3 with each entry written to 3 bookies. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538557218910-2K8O0CLWOD9N7ZXX45JS/FragmentWithStriping.png Blog - Understanding How Apache Pulsar Works Fig 7. Striping https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538559874547-UYUESUUV7Q33RK5BJXDB/PulsarBrokerWrites.png Blog - Understanding How Apache Pulsar Works Fig 8. A single broker serves all reads and writes of a given topic. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538560169571-MKFXQQ8FJ6MMJH4V59EO/PulsarBrokerReads.png Blog - Understanding How Apache Pulsar Works Fig 9. Reads only need go to a single Bookie https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538601042753-J8IXQ06HUPV1R7PEAPFE/image-asset.png Blog - Understanding How Apache Pulsar Works Fig 10. A Bookie with the default (with Apache Pulsar) DbLedgerStorage architecture. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538583413563-QVJ2MWQQK1B10NIP45NZ/RoundUp.png Blog - Understanding How Apache Pulsar Works Fig 11. Round-up of concepts https://jack-vanlightly.com/blog/2018/9/18/how-to-lose-messages-on-a-kafka-cluster-part-2 monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2018/9/14/how-to-lose-messages-on-a-kafka-cluster-part1 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537225068940-RQ6TFLO9E9W9A03SROHR/Title.png Blog - How to Lose Messages on a Kafka Cluster - Part 1 Chaos testing Apache Kafka with Blockade, Docker, Python and Bash https://jack-vanlightly.com/blog/2018/9/10/how-to-lose-messages-on-a-rabbitmq-cluster monthly 0.5 2018-11-01 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536795400197-IB0QDDQ5H3NAACNBHEKO/Title.png Blog - How to Lose Messages on a RabbitMQ Cluster RabbitMQ chaos testing with Blockade, Docker, Python and some Bash https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537442112810-AUK1B4U93RDGLSM00H2E/RabbitQueueWithUnsyncedMirrors.PNG Blog - How to Lose Messages on a RabbitMQ Cluster https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537442359789-IWQMAOF7U91DIQE2Z6RH/RabbitQueueOffline.PNG Blog - How to Lose Messages on a RabbitMQ Cluster https://jack-vanlightly.com/blog/2018/9/2/rabbitmq-vs-kafka-part-6-fault-tolerance-and-high-availability-with-kafka monthly 0.5 2019-04-01 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535898428232-NY6TSM87WZLBVW0CV2LW/KafkaPartitions.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 1. Four partitions distributed across three brokers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535898716029-CEBI3RHJP2386WYLHWHT/KafkaPartitionLeaderFollower.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535902300576-U5MZB0GH8KXYS19BPC5H/KafkaPartitionFailOver1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 2. Broker 3 dies and the partition 2 follower on broker 2 is elected the new partition leader. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535902482198-C367XNPEOGORJ7QLB0ES/KafkaPartitionFailOver2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 3. One broker left. All leaders on a single broker with zero redundancy. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535902580550-ZDLLNEXBHVXCTYZVWD2B/KafkaPartitionFailOver3.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 4. Leaders remain on broker 2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535902652106-CZ65WGLYWHAU59YYD90U/KafkaPartitionFailOver4.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 5. Unbalanced leaders after recovery of broker 1 and 3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535905064179-WX2U70292ITPY8V47XWQ/KafkaPartitionFailOver5.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 6. Replica leaders rebalanced https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807783088-BOHVBHGO75YIY2GMPSNL/Correction_KafkaFailOverAcksOne1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 7. ISR has three replicas. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807810911-89F8CDEBF3IX3P1E71I6/Correction_KafkaFailOverAcksOne2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 8. Fail-over loses messages. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807838221-OO8P0ZGJVE8I8IVQGT7W/Correction_KafkaFailOverAcksOne3.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 9. Message sending resumes after brief interruption https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807864379-3QZBEJT0QCBMUDTY2AQ2/Correction_KafkaFailOverAcksOne4.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 10. Broker 3 follower removed from ISR. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807887912-D49GIOVQAN2M706CQMA3/Correction_KafkaFailOverAcksOne5.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 11. Broker 1 dies. Fail-over loses a large number of messages. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807910232-CF1AFM37XB70QUYT20LW/Correction_KafkaFailOverAcksOne6.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 12. After brief interuption, message again being sent to partition 0. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807931289-W0O6ZVG7EFWWUPR8QX5C/Correction_KafkaFailOverAcksAll1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 13. ISR has three replicas. One replica is slow, adding latency to writes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807951574-2H6VO0PFCL35DMIEQSZG/Correction_KafkaFailOverAcksAll2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 14. All replicas persist the message and an ack is sent. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807974483-TMOQB8ONCK42K0F2JBJ2/Correction_KafkaFailOverAcksAll3.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 15. Replica on Broker 3 is removed from the ISR. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537807994395-NVLM0HX014R0E15CH20E/Correction_KafkaFailOverAcksAll4.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 16. Broker 2 fails. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537808016286-RNO9XVX4WY83L38Z1TWY/Correction_KafkaFailOverAcksAll5.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 17. The replica on broker 1 takes over leadership without message loss. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537808036235-T1E4XP5VWPU40Y07OTOF/Correction_KafkaFailOverAcksAll6.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 18. Broker 1 dies and the unclean fail-over produces messive data loss. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537808057964-RPRQLU0VIMD9MF3W60I4/Correction_KafkaFailOverAcksAllMinInsyncReplicas1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 19. ISR of two. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537808081007-DMM2KTV27P29L43VK27T/Correction_KafkaFailOverAcksAllMinInsyncReplicas2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 20. ISR of one lower than min.insync.replicas https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536060724892-ZGUF2CB2ZXGDWI42EA49/KafkaConsensus.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 21. Kafka consensus https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130438623-1E1H88H6YW4PJKYOTUP3/KafkaNetworkPartitionScenarioOne1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 22. Scenario 1: ISR consists of three replicas. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130477196-6852E0EZKISYA0IANXIS/KafkaNetworkPartitionScenarioOne2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 23. Scenario 1: Broker removed from the ISR after passing replica.lag.time.max.ms without a fetch request. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130351467-MTXL3CRJ1JDW1H8118FK/KafkaNetworkPartitionScenarioTwo1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 24. Scenario 2: Leader and two followers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130516569-VZAF0PBZCHFMY9JL5U81/KafkaNetworkPartitionScenarioTwo2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 24. Scenario 2: ISR shrunk to only the leader https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130627764-Z21CSSHA1316FTSNNAC0/KafkaNetworkPartitionScenarioThree1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 25. Scenario 3: Follower continues to send fetch requests to the leader https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130719271-9NMWEUCWLPMJB6K07W3M/KafkaNetworkPartitionScenarioFour1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 26. Scenario 4: Leader and two followers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130765821-YTLDE52FJLXYNUZNN7ER/KafkaNetworkPartitionScenarioFour2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 27. Scenario 4: Leader isolated from Zookeeper. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536130929776-X1ZR5ZGMHP46TK8OG2NU/KafkaNetworkPartitionScenarioFour3.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 28. Scenario 4. The leader on Broker 1 becomes a follower after the network partition is resolved. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536131324036-5JC879PSEXAYMB1LCFL1/KafkaNetworkPartitionScenarioFive1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 29. Scenario 5: The isolated follower is removed from the ISR. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536131398080-J8XY7L2WHOF4Q4GEPJ2V/KafkaNetworkPartitionScenarioSix1.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 30. Scenario 6: A leader and two followers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536131456504-YHIFNS2OD7G4ZYNHRM5N/KafkaNetworkPartitionScenarioSix2.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 31. Scenario 6: The leader becomes isolated from other Kafka nodes and Zookeeper https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536131526384-U72I7SCEK2T0GFP5NIGQ/KafkaNetworkPartitionScenarioSix3.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 32. Scenario 6: Two leaders https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536131632972-YJ3GEJMGRI2S7R3DF5WM/KafkaNetworkPartitionScenarioSix4.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 33. Scenario 6. Producers switch over to the new leader https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1536131714329-RV2UPS6OP1DX1UYDSZUD/KafkaNetworkPartitionScenarioSix5.PNG Blog - RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka Fig 34. Scenario 6: The original leader becomes a follower after the network partition is resolved. https://jack-vanlightly.com/blog/2018/8/31/rabbitmq-vs-kafka-part-5-fault-tolerance-and-high-availability-with-rabbitmq monthly 0.5 2019-11-28 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535701450819-Q4Q6CUMKOZFUCLHNZ53I/DurableQueuePersistentMessage.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 1. Durability matrix https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1574964430941-NNQAW7XTYIR8BIALYSXQ/QQ.png Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering A leader and two followers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1574964138663-GVDBZ3JA35Z1VCIMOI92/QuorumQueue.png Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Three quorum queues distributed across a cluster https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1574964354259-3NXCD2W53AX9MRSCXVD6/QuorumQueueFailOver.png Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Broker 3 goes down and queue C gets a new leader. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535702422979-8WDITA57FBC6VARTLAYY/QueueMirroring.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 2. A mirrored queue. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713303609-P07S8QCBPAF2H2KWVCWX/RabbitMqQueueFailover1.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 2. Multiple mirrored queues and their policies. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713329636-LH4FSHBG5IJAFMAX6INU/RabbitMqQueueFailover2.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 4. Broker 3 dies which causes a fail-over for Queue C. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713345954-EK2D0ISECJA6PA5GQYMW/RabbitMqQueueFailover3.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713609396-VNS0XTJMKEE73A8KR4LL/RabbitMqQueueFailover4.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 5. Broker 1 comes back online. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713639893-NDX1PI6KIFZJR03PJK4T/RabbitMqQueueFailover5.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 6. Broker 3 comes back online. All masters on one node! https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713665557-H4F64ALLKZCFGWHR7WNZ/RabbitMqSynchronization1.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 7. Two queues with different synchronization modes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713684742-LR0U6I83RP1XC49VKU7N/RabbitMqSynchronization2.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 8. Broker 3 is lost. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713699772-J7DQ0DCPJVZYIQNKT2PE/RabbitMqSynchronization3.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 9. The new Queue A mirror gets all the existing messages and new Queue B mirror does not. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1537802023740-97FKRBI7C8UM1OA7BO29/RabbitMqSynchronization4.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 10. Queue A fails-over to Broker 1 without message loss. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713733777-UZ5G8J26PHQLDWOOSE8T/RabbitMqSynchronization5.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 11. Queue A fails-over to Broker 3 without message loss. Queue B fails-over to Broker 3 losing 10 messages. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713751965-L7X4XQOPSSDLWIEW2D8S/RabbitMqSynchronization6.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 12. Queue B remains unavailable after the loss of Broker 1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713768650-42REFXABNXGINAUCMUMC/RabbitMqSynchronizationCost1.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713785408-P8Z6S3WNB0GG5U3X48Y2/RabbitMqSynchronizationCost2.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 13. Broker 3 dies, leaving a master and one mirror per queue. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535713803996-T72SRCZ80EINMRO6ZCOA/RabbitMqSynchronizationCost3.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 14. Queue A remains unavailable during synchronization. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535716951787-WGD2LW2MGZSVTMPG915I/RabbitMqSplitBrain.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 15. A master and two mirrors, each on a separate node. Then a network partition occurs and one mirror is separated. The separated node, seeing that the other two nodes have gone away promotes its mirrors to master. Now we have two masters and both can be written to and read from. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535718360575-48ZNX9ERROBT50GUCAKT/RabbitMqPartitionIgnore1.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 16. Three publishers are connected to three brokers. Internally the cluster routes all requests to the master queue on Broker 2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535719075541-COUHOLTU9IVOI64GH06L/RabbitMqPartitionIgnore2.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 17. Split-brain. Writes go to two masters and the two copies diverge. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535733189963-YOPGDSMEBFD0NJCLUDBF/RabbitMqPartitionIgnore4.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 18. Administrator shutdown Broker 3 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535733239752-80OB6ON4ROZTZWOUWZY8/RabbitMqPartitionIgnore5.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 19. Administrator starts-up Broker 3 and it rejoins the cluster, losing any messages that remained on that Broker. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535733685801-RP0MARSDUIWBSMCDE2TP/RabbitMqPartitionPauseMinority1.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 20. Three publishers are connected to three brokers. Internally the cluster routes all requests to the master queue on Broker 2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535733865879-3DHZKJB6F41UAFLU140P/RabbitMqPartitionPauseMinority2.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 21. Broker 3 pauses itself, disconnected any clients and refusing connection requests. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535733938880-ZSAY28TMVIVLFEUKCJ3E/RabbitMqPartitionPauseMinority3.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 22. Master on Broker 3 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535734039988-3J0XDUB8XOPL1JWUYHPM/RabbitMqPartitionPauseMinority4.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 23. Fail-over to Broker 2 while Broker 3 is unavailable. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1535734103339-19TXHQGFISBIDUC83FPA/RabbitMqPartitionPauseMinority5.PNG Blog - RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering Fig 24. The cluster is back to normal again. https://jack-vanlightly.com/blog/2018/8/14/aws-security-securing-your-use-of-the-aws-cli-and-automation-tools monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1534217529334-7YLDFL8R7OR7C3OD3KUH/CreateRole1.PNG Blog - AWS Security - Securing Your Use of the AWS CLI and Automation Tools https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1534218001803-748LHGT7PT6EKS0G818F/CreatePolicy.PNG Blog - AWS Security - Securing Your Use of the AWS CLI and Automation Tools https://jack-vanlightly.com/blog/2018/7/25/rabbitmq-work-queues-avoiding-data-inconsistency-with-rebalanser monthly 0.5 2018-12-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532494980716-6NZV9Z8TOPGVC9FDWBOO/RabbitMqCompetingConsumers.png Blog - RabbitMQ Work Queues: Avoiding Data Inconsistency with Rebalanser Fig 1. Work queue with multiple competing consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532496124555-DHDN47RKD4AZHVZ4KAEM/WorkQueueWithBuckets.png Blog - RabbitMQ Work Queues: Avoiding Data Inconsistency with Rebalanser Fig 2. Bucketing of Customer Id to route to a fixed number of queues. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532496309957-ZXPO3QD3I365J2IMHXUJ/image-asset.png Blog - RabbitMQ Work Queues: Avoiding Data Inconsistency with Rebalanser Fig 3. Using a Consistent Hash Exchange to partition a single logical queue into multiple physical queues. https://jack-vanlightly.com/blog/2018/7/22/creating-consumer-groups-in-rabbitmq-with-rebalanser-part-1 monthly 0.5 2018-12-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532295252810-LGPD3V0VNQWLAHDJCAXG/QueuePartitioning.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 1. Messages partitioned across five queues, with one consumer per queue. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532295444820-X5GB9WPXHYGD1Z95HY18/KafkaPubSubSimple.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 2. One topic, three partitions with three consumers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532295555112-ZX8GZYYC769QJFCY322Q/KafkaMorePartitionsThanConsumers.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 3. Five partitions with three consumers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532295620574-U299PTNPKDJ2FQROVUJZ/KafkaSpareConsumer.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 4. One consumer stays idle as one partition can only be read by one consumer of the same consumer group. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532295772562-A60WNPJ0HX4ROETQMDCP/KafkaRebalancing.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 5. Rebalancing triggers after two new consumers come online. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532295877340-V08C7VRC02TR94J303GM/KafkaPubSub.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 6. Two consumer groups consuming from the same topic. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532296366170-HK1OM69RXZ3NN82MJ2N7/RabbitMqCompetingConsumers.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 7. Competing consumers of a single queue. Message ordering weakened, limited to scalability of a single queue. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1532296738596-EDXXN64X2ECZKL07GKPD/Rebalanser.png Blog - Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1 Fig 8. Rebalanser manages queue assignment and rebalancing, using SQL Server as a backend. https://jack-vanlightly.com/blog/docker-net-core-and-redshift-drivers monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2018/5/21/event-driven-architectures-queue-vs-log-case-study monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526891469008-2IBX93J0ZD3313MEZCEP/QueuesVsLogsExampleArchitecture.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 1. Logical events architecture https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526892207363-6FR9GJKCEAHQ8VMV6OVH/QueuesAndTopics.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 2. Exchange to queue bindings https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526894917059-X01F076A63156MPVIN9V/QueuesVsLog_RabbitMQTopology.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 3. RabbitMQ topology with fanout exchange per event, single queue per application https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526900209908-D83XUI5Y2KPLM6NRTO6D/QueuesVsLogs_QueueBottleneck.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 4. Fulfillment consumer binds its queue directly to event exchanges https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526896592351-MRMW3AZWEOW7IHYA9F5J/QueuesVsLogsScaledOutQueue.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 5. Fulfillment consumer binds a Consistent Hashing Exchange to the event exchanges it wants to consume, scales out its queue to 5 partitioned queues and has a single consumer instance per queue. Messages have an OrderId header which is used for routing to the partitioned queues. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526899093284-AJ5S3LUUYDS8MZKBOBC3/QueuesVsLogsSingleLogTopic.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 6. A single topic for all order related events. Consumers read all messages and ignore the events they are not interested in. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1526898740650-IOWYAN8Z5SNAZLKGBEQR/QueuesVsLogsMultipleTopics.png Blog - Event-Driven Architectures - Queue vs Log - A Case Study Fig 7. Using multiple topics wuith smaller groupings of related events https://jack-vanlightly.com/blog/2018/5/20/event-driven-architectures-the-queue-vs-the-log monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2018/4/28/sql-server-cdc-to-redshift-pipeline monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524905992004-SEJVRYYG61UWHQR7UQ63/CdcProcess.png Blog - SQL Server CDC to Redshift Pipeline Fig 1. CDC process https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524902842477-MD5TEE93HDYUPGXVM9DZ/CdcTable.PNG Blog - SQL Server CDC to Redshift Pipeline Fig 2. CDC table for the Person table. Shows 4 inserts. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524902284344-2SLACE7QS7TM0H08TADK/TranLog.png Blog - SQL Server CDC to Redshift Pipeline Fig 3. Changes of all tables maintained in the transaction log (a linearized sequence of transactions) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524902351533-JWAEFZIL2AN3B1XY6DIW/CdcTableQueries.png Blog - SQL Server CDC to Redshift Pipeline Fig 4. Using LSNs, we can query all tables in a sub window and get a consistent snapshot of changes across multiple tables. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524903132239-1N52AAZR8LHNVSIBPZ38/lsnTimeMappingTable.PNG Blog - SQL Server CDC to Redshift Pipeline Fig 5. Start and and times for each LSN. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1525081581374-Y22PYO25VQ47RKO0QYD0/AllVsNetChanges.PNG Blog - SQL Server CDC to Redshift Pipeline Fig 6. All vs Net changes https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1525897743605-1X1D7EWQAYFCDKR3KESE/GlobalTranPipeline.png Blog - SQL Server CDC to Redshift Pipeline Fig 7. Processing pipeline to handle transactions of any size across multiple tables. https://jack-vanlightly.com/blog/2018/4/19/processing-pipelines-series-reactive-extensions-rxnet monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524212010759-A6V3O2LV28C6NM4FOSUC/RxPipeline2.png Blog - Processing Pipelines Series - Reactive Extensions (Rx.NET) Fig 1. The pipeline with Reactive Extensions https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524215778812-XOOTY33TDHF0PYMFY2CB/RxPipelineWithoutHotObservable.png Blog - Processing Pipelines Series - Reactive Extensions (Rx.NET) Fig 2. Without the hot observable we have four separate pipelines https://jack-vanlightly.com/blog/2018/4/19/processing-pipelines-series-tpl-dataflow-alternate-scenario monthly 0.5 2018-10-18 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524127168858-CSUYK6INZXJLBV8EXJ7U/TplDataflowWithBackPressure.png Blog - Processing Pipelines Series - TPL Dataflow - Alternate Scenario Fig 1. The pipeline with BroadcastBlocks https://jack-vanlightly.com/blog/2018/4/18/processing-pipelines-series-tpl-dataflow monthly 0.5 2018-10-18 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524039131956-CJIPYJRFDIEN4ACOBJ3S/TplDataFlowBlocksAndBuffers1.png Blog - Processing Pipelines Series - TPL Dataflow Fig 1. Each block has a buffer and a Func delegate to perform work https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524039428077-Q3AD4JE1WS4T5N868V9B/TplDataFlowBlocksAndBuffers2.png Blog - Processing Pipelines Series - TPL Dataflow Fig 2. Incoming and outgoing buffers with a shared capacity limit. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1524055202780-85E2B9Z0BZHX2XODUP3D/TplDataflowPipelineBlocks.png Blog - Processing Pipelines Series - TPL Dataflow Fig 3 - Our pipeline with TPL Dataflow blocks https://jack-vanlightly.com/blog/2018/4/17/processing-pipelines-series-introduction monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1523995048886-N9P4ECWCU18O3IMRGKLC/FacilityProcessingTopology2.png Blog - Processing Pipelines Series - Concepts Fig 1 - Processing pipeline https://jack-vanlightly.com/blog/2018/1/25/agiletm-vs-real-agility-the-constant-rush monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2018/1/16/agiletm-vs-real-agility-truly-self-organizing-teams monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1516135705695-K7HXGCKRQTNWVSSA4PVJ/TrueAgilityBehaviours.png Blog - AgileTM vs Real Agility - Truly Self-Organizing Teams Fig 1. True agility https://jack-vanlightly.com/blog/2018/1/15/agiletm-vs-real-agility-the-fight-is-on monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1516049273915-H8DDOAULKDOEULAM5GEJ/TrueAgilityBehaviours.png Blog - AgileTM vs Real Agility - The Fight Is On Fig 1. True agility https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1516050144819-NYTUT39M1B80IEV6H2OG/FakeAgilityBehaviours.png Blog - AgileTM vs Real Agility - The Fight Is On Fig 2. A process that enables micro-mamagement and control is not agile https://jack-vanlightly.com/blog/2018/1/4/dont-believe-the-hype monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2017/12/15/rabbitmq-vs-kafka-part-4-message-delivery-semantics-and-guarantees monthly 0.5 2019-11-28 https://jack-vanlightly.com/blog/2017/12/8/rabbitmq-vs-kafka-part-3-kafka-messaging-patterns monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512813188413-EEI12VI1FMQLJ4XMRQTJ/KafkaPubSub.png Blog - RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns Fig 1. Multiple produces send messages to a partitioned log, consumed by multiple consumer groups https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512897366315-OKH2CS1I6ENC7TGSA4H0/ArchBefore.png Blog - RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns Fig 2. Multiple types of data integration https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512897785986-EILMLD9PZYJ0KWXBW8A8/ArchAfter.png Blog - RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns Fig 3. Kafka centric data integration as an event sourcing platform https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512921474821-WD11WUJ0EOG4Q8MKMFES/MasterDerivedData.png Blog - RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns Fig 4. Master and derived data https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512932894838-T0GWL1EKMG6QWYR0F66M/KafkaMessageLifecycle.png Blog - RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns https://jack-vanlightly.com/blog/2017/12/5/rabbitmq-vs-kafka-part-2-rabbitmq-messaging-patterns-and-topologies monthly 0.5 2019-11-28 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512577450141-P0ASIRW196KG5VWLLF7F/RabbitMQRoutingFanout.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig1. Fanout exchange broadcasts to three queues (independent consumers) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512577465906-ZA2WBZYXHEO46PRLI57Z/RabbitMQRoutingDirect.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 2. Direct exchanges routes by exact match Routing Key to Binding Key https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512578069749-2BMGO8WZA8CDI759CYP1/RabbitMQRoutingDefault.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 3. Default exchange has an implicit binding to each queue https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1513007854559-PW8CD408JIC9BM63O4QI/RabbitMQConsistentHashing.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 4. Messages are distributed by partitioned hash space https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512591297076-5W6LPCLZ6DK7AE04ENH7/RabbitMQMultiplePubAndMultipleSetsOfSubs.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 5 - Simple pub sub pattern with Fanout exchange https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512593788755-1UM0FS99FDKNROMC239I/RabbitMQRoutingLayeredTopic.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 6. Optimise routing overhead through layered exchanges https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512594335385-WP1D0K7HJJJYM9FIS875/RabbitMQRoutingLayeredFanout.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 7. Fanout broadcasts to more specialised exchanges https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512638978653-ZIY87O38TXYWC7A9RMJB/ReadFromMailbox.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 8. Applications reading directly from a mailbox https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512639085260-32VDBF7F4GDYLWBJG9QN/FromMailBoxToDb.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 9. Applications reading from a database https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512639305342-9CSI91Y3J68QG2KOAPX5/MailBoxToRabbitMQ.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 10. Applications receive emails they want over RabbitMQ https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512670176041-NHYG36TEXCX518TS8QFP/MessageExchanges.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 11. Public message exchanges, private consumer exchanges https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512671313188-UVJRLHT3P1I3K96CMOFE/image-asset.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 12. Point-to-point message via the Default exchange https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1513009187941-14AGZ0T0C2R9B2KC5EWB/RabbitMQNoProcessingOrderGuarantee.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 13. Multiple competing consumers consume a single queue and lose processing order guarantees https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1513009249764-244052JH5QUVAXXFUCYU/RabbitMQWithProcessingOrder.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 14. Messages distributed by hashing function, each queue consumed by one consumer https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512674526587-QJG1ZUTKMSLUZ1558JSD/RabbitMQHierarchical.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 15. Hierarchical routing https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512676943953-ABTSPKQH8A2ID4USQ1P1/RabbitMQPriorityQueue.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 16. Priority Queue https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512677126951-NEP2XI09KXQA0RTT24NK/RabbitMQPriorityViaTopic.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 17. Routing by priority https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512722000122-8ACVLZSDXXI93QKYFS4G/RabbitMQAlternate.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 18. If/Else routing with alternate exchange https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512723776930-7MUSWG9RZN7GZ67XGGXV/RabbitMQNotRouting.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 19. Consumer App 2 gets all booking messages except mytravel.com ones https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512725959933-XRRSFP3XBKSWZW4K2ZWA/RabbitMQDelayedRetriesCascade.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 20. Cascading delay exchanges and queues https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512730541862-EKOK4D8OMCZ0U5VRVP6R/RabbitMQRetriesEthemeral.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 21. Ephemeral exchanges and queues for delayed routing https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512731190129-PC06BFBS8B62L1VV3NHC/RabbitMQRetriesEthemeralPrivateExchange.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 22. Routing to private consumer exchange instead of Default exchange https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512747586678-5EJULIGE7B0GE9BKYHEQ/RabbitMQPubSubNoDelay.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 23. Three publishers send to a topic exchange https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512747785265-U86N8QFV7SY58ZMF52QN/RabbitMQPrivateDelay.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 24. One publisher adds a delay with its own exchange and queue https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512751601938-JXXUDQD4N6NKOF0M3CCW/RabbitMQReplyToFixed.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 25. Single reply-to queue https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512751646951-MR59CKFZUEHHPBV3S265/RabbitMQReplyToPerHost.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 26. Reply-To queue per application host https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512751707102-OIKQ2GQVZOISWMHB5JBS/RabbitMQReplyToEphemeral.png Blog - RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns Fig 27. Ephemeral reply-to queue per message https://jack-vanlightly.com/blog/2017/12/4/rabbitmq-vs-kafka-part-1-messaging-topologies monthly 0.5 2019-11-28 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486187418-PJNVMZWARY4NN8ECB73K/RabbitMQOnePubOneSub.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 1 - Single publisher and single consumer https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486256489-LWM9E4T45G241RLZJ9YW/RabbitMQMultiplePubAndSub.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 2 - Multiple publishers, multiple independent consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486306273-10AWR7GDIOHZRR0PHWGM/RabbitMQMultiplePubMultipleCompetingSub.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 3 - Multiple publishers, one queue with multiple competing consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486354898-326EHTHC96YAH9B0EHFT/RabbitMQMultiplePubAndMultipleSetsOfSubs.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 4 - Multiple publishers, multiple queues with competing consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512493440649-1IFKXHZRJGGAP0ABHJLQ/RabbitMQTopicExample.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 5. Topic exchange example https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486445386-HSXWB3M8CYDJCHZIO2HX/KafkaOneProdOnePartOneCon.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 6 One producer, one partition, one consumer https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486510360-82ZY18RU450XAMEQ1UHS/KafkaOnePartTwoCon.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 7 One producer, one partition, two independent consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512486559653-ZJV6NHWOWPFMJ4CNM2L2/KafkaMultiplePartitions.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 8 Three partitions and two sets of three consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512419320698-8GSC7PPLRPPSMOZQDQ1K/KafkaOnePartTwoCon.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 9. Consumers with different offsets https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512419470263-QLKF6V6XXCG4C8CDFB5L/KafkaPubSubSimple.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 10. One producer, three partitions and one consumer group with three consumers https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512422787367-MLO39JASGFTCZZQYYSLV/KafkaMorePartitionsThanConsumers.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 11. Sone consumers read from more than one partition https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512423003570-XQ9OAF76YFLL7PAOZSWS/KafkaSpareConsumer.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 12. One idle consumer https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1512423601894-59UOAGCSPNHS0E2FM8SG/KafkaRebalancing.png Blog - RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging Fig 13. Addition of new consumers requires rebalancing https://jack-vanlightly.com/blog/2017/12/3/rabbitmq-vs-kafka-series-introduction monthly 0.5 2018-09-24 https://jack-vanlightly.com/blog/2017/9/17/you-dont-need-scrum-you-need-agility monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2017/6/16/kathryn-mcconkey-the-best-osteopath-in-barcelona monthly 0.5 2018-02-13 https://jack-vanlightly.com/blog/2017/6/11/improving-reliability-and-incident-response-via-a-message-lifecycle monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497216374828-P6A2AYSW2HGRLCDOX3US/image-asset.png Blog - How to Make Your Messaging System Reliable and Keep Your Support Engineers Happy https://jack-vanlightly.com/blog/2017/6/10/dsl-parser-sample-code monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497086756042-6BNOX3HKZJXN6INVK00I/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497086804433-U6K9I2V47A02SXBFAXAX/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497086857604-EONU7HG917RJH2Q85SRC/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497086884209-VLI98GMIG7K4EMRPP8H1/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497086998038-1AA9X6UGETTQPFBJNTAP/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497087049400-1D3XQBXYG5EU20LKBTOR/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497087105846-60Q7J39PG9XZBOR3M3EJ/image-asset.png Blog - DSL Parser - Sample Code https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1497087157463-1GL9CQCSBHQ8IJ4J3P67/image-asset.png Blog - DSL Parser - Sample Code https://jack-vanlightly.com/blog/2017/3/24/rabbitmq-delayed-retry-approaches-that-work monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2017/3/20/reliability-custom-retries-nservicebus-with-rabbitmq-part-5 monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2017/3/19/reliability-default-retries-nservicebus-with-rabbitmq-part-5 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489930211972-1MKW7GG1XMABRAWFKTL9/image-asset.png Blog - Reliability - Default Retries - NServiceBus with RabbitMq Part 5 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489933421466-25AS6FAM71OXFDP4TOCK/NServiceBusDelayInfrastructure.png Blog - Reliability - Default Retries - NServiceBus with RabbitMq Part 5 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489958267747-PKC90XQYCSVKT536SLPN/image-asset.png Blog - Reliability - Default Retries - NServiceBus with RabbitMq Part 5 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489958616556-VCGEQCPP185EGYFOJOEZ/image-asset.png Blog - Reliability - Default Retries - NServiceBus with RabbitMq Part 5 https://jack-vanlightly.com/blog/2017/3/18/custom-topology-nservicebus-with-rabbitmq-part-4 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489819649361-TD3DQZ78WK88D9PP8O0S/image-asset.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489819838767-F68R7ZAK5KOVD2FO0AA1/image-asset.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489820838034-T30XJTJPTF6CRTXA0PJW/image-asset.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489821983739-5ONUDFF56LWUQWCBF7TE/image-asset.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489822088579-JPBNDEPO2PYRK1Z2QSP4/image-asset.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489822670623-DJ0WAKDAMW44IRR4XN1C/RetailDemoGoldInsiders.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489825927955-JPR6J8MDEGUZMJBMKD50/image-asset.png Blog - Custom Topology - NServiceBus with RabbitMq Part 4 https://jack-vanlightly.com/blog/2017/3/17/custom-direct-topology-nservicebus-with-rabbitmq-part-3 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489747823288-5ZWGBEZUP5RATXAFEXFR/image-asset.png Blog - Custom Direct Topology - NServiceBus with RabbitMq Part 3 https://jack-vanlightly.com/blog/2017/3/16/default-topologies-nservicebus-with-rabbitmq-part-2 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489667431528-YK8OKERR485WDTSJV0VV/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489667549397-ZR50NTVKWZU4SZIHZC7Q/RetailDemoExtra.png Blog - Default Topologies - NServiceBus with RabbitMq Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489739345741-CMDR83ZJZ75XS08PAOL4/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489740214668-17D3BVXC3I6MA4TNOVSB/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 2 https://jack-vanlightly.com/blog/2017/3/16/default-topologies-nservicebus-with-rabbitmq-part-1 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489658574279-UNP5OO65YXCZ7O8ZVOHL/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489740144298-Z448BYIITX8T7C2WZ1D3/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489740175214-VS1D2395K8WY9GE6GOBG/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489731433863-2LTNMLCEL8UMV0BQQYEQ/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489735530150-KI3W9TZ8F66GTRLWM1UY/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489735706946-18HR1QQRQMFE5BEV76MN/image-asset.png Blog - Default Topologies - NServiceBus with RabbitMq Part 1 https://jack-vanlightly.com/blog/2017/3/12/how-to-deal-with-unroutable-messages-rabbitmq-publishing-part-3 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489326421674-WM398TOVXPA0Y7QGEQBZ/image-asset.png Blog - How to Deal with Unroutable Messages - RabbitMq Publishing Part 3 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489326582965-29N4W50DJZKRGJ2IYBA1/image-asset.png Blog - How to Deal with Unroutable Messages - RabbitMq Publishing Part 3 https://jack-vanlightly.com/blog/2017/3/11/sending-messages-in-bulk-and-tracking-delivery-status-rabbitmq-publishing-part-2 monthly 0.5 2019-03-09 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489235758475-G5GUIWQPG4R782BUL9B6/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489235891868-DCQPGVOEISOORGM5XU4B/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489236363309-5H9K2NOIFBXOE2A4LCJT/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489236387581-34DKN8YU7MQLIR6E5HFK/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489236447607-4XGNLIDY55WHSA188M3E/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489306490243-PCMWJAVTEDWQGRAAYXST/RetriesExhausted.PNG Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489262830649-KXJ8H2P1W88EA9I3ANH3/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489263332265-2ZJZS7AO0WKMOYE67WUW/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489264420648-Q672H27EF7K6H7TC8QDI/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 One batch, 1415 milliseconds https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489264460006-GMSCMA950T8DQNJ7EBSU/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 100 batches of 100 messages, 2210 milliseconds https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1489264502796-YG0BQ3RNE9NL78FZX6LQ/image-asset.png Blog - Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2 10000 batches (one message per batch), 54378 milliseconds https://jack-vanlightly.com/blog/2017/3/10/rabbitmq-the-different-failures-on-basicpublish monthly 0.5 2019-12-08 https://jack-vanlightly.com/blog/2017/2/21/10-years-of-eye-pain monthly 0.5 2021-10-09 https://jack-vanlightly.com/blog/2016/11/15/new-sqlconnection-the-requested-performance-counter-is-not-a-custom-counter monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/11/12/building-synkronizr-a-sql-server-data-synchronizer-tool-part-1 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1478988839192-LR7J2QVOIZLHPJ8EU5FR/image-asset.png Blog - Building Synkronizr - A SQL Server Data Synchronizer Tool - Part 1 https://jack-vanlightly.com/blog/2016/11/9/generating-sql-from-a-data-structure monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1478725880219-GOK7E39K7JI9TNUZP80U/image-asset.png Blog - Generating SQL from a Data Structure https://jack-vanlightly.com/blog/2016/11/8/how-to-kill-a-keep-alive-with-a-weak-reference-c monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/11/7/how-row-locking-makes-taskling-concurrency-controls-possible monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/11/6/announcing-tasklingnet-a-c-batch-job-api monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/10/30/too-busy-to-create-your-own-visualizations-just-leverage-the-neo4j-console monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477861377719-RNFCBXNXC5RM5H709S1B/image-asset.png Blog - Too Busy to Create Your Own Visualizations? Just Leverage the Neo4J Console https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477861617506-QEVHNDFALRE8VX7ESYA2/image-asset.png Blog - Too Busy to Create Your Own Visualizations? Just Leverage the Neo4J Console https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477861707583-KQSOHGBSTR8BWNE4UZRN/image-asset.png Blog - Too Busy to Create Your Own Visualizations? Just Leverage the Neo4J Console https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477861730900-9OCUX9R2DX7DVSHU0Q6I/image-asset.png Blog - Too Busy to Create Your Own Visualizations? Just Leverage the Neo4J Console https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477861887950-OV65HK4POMSW3V94JUQY/image-asset.png Blog - Too Busy to Create Your Own Visualizations? Just Leverage the Neo4J Console https://jack-vanlightly.com/blog/2016/10/27/why-you-should-understand-databases monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/10/24/exploring-the-use-of-hash-trees-for-data-synchronization-part-1 monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477339776419-63GC1Q29KR7359GYFEI6/image-asset.png Blog - Exploring the use of Hash Trees for Data Synchronization - Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477338792266-DM5YR65AKVZBDHTDSP9C/image-asset.png Blog - Exploring the use of Hash Trees for Data Synchronization - Part 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477338885647-8U9QK2T7KQHUUJZX2Z5M/image-asset.png Blog - Exploring the use of Hash Trees for Data Synchronization - Part 1 https://jack-vanlightly.com/blog/2016/7/2/5-tricks-for-people-who-are-hyper-sensitive-to-computer-screens monthly 0.5 2018-10-25 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1467496609001-X8DFHF08EUCJ4ZH365TE/image-asset.png Blog - 5 Tricks For People Who Are Hyper Sensitive to Computer Screens https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1467495822459-J8Y80591K4RA8KR3Q6CK/image-asset.png Blog - 5 Tricks For People Who Are Hyper Sensitive to Computer Screens See the shielding at the base and sides https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1467495934744-ZDIQNHID80HQQZX2H1J7/image-asset.png Blog - 5 Tricks For People Who Are Hyper Sensitive to Computer Screens https://jack-vanlightly.com/blog/2016/2/25/optimizing-regex-performance-with-regexoptionsrighttoleft monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/2/24/a-more-efficient-regex-tokenizer monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1456348892112-TVOC28B6TDYLSO4GRJGT/image-asset.png Blog - A More Efficient Regex Tokenizer https://jack-vanlightly.com/blog/2016/2/11/implementing-a-dsl-parser monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/2/4/understanding-grammars monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/2/3/creating-a-simple-tokenizer-lexer-in-c monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/2/3/how-to-create-a-query-language-dsl monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1454526650259-0IA2R7BOVRBKF8O0150W/image-asset.png Blog - How to Create a Query Language DSL with C# https://jack-vanlightly.com/blog/2016/1/26/detective-optimizers monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/19/the-freedom-quadrant monthly 0.5 2018-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1453205437222-G7Q8B3V5202C269B2UG6/image-asset.png Blog - The Freedom Quadrant https://jack-vanlightly.com/blog/2016/1/6/logging-query-language-lql-a-dsl-for-log-analytics monthly 0.5 2017-09-25 https://jack-vanlightly.com/blog/2016/1/4/bitching-will-erode-your-team monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/4/we monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/4/breaking-your-own-code-and-designs monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/4/make-mini-technical-leads monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/4/code-analysis-rules-versus-training-and-coaching monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/3/code-analysis-rules-and-the-click-through-culture monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/3/being-open-sharing-your-knowledge monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/2016/1/3/stop-time-wasters-and-protect-your-knowledge-sharing-culture monthly 0.5 2018-09-23 https://jack-vanlightly.com/blog/category/Non-Tech monthly 0.5 https://jack-vanlightly.com/blog/category/Strategy+and+commentary monthly 0.5 https://jack-vanlightly.com/blog/category/Serverless+Data+Systems monthly 0.5 https://jack-vanlightly.com/blog/category/Messaging+Systems monthly 0.5 https://jack-vanlightly.com/blog/category/Programming monthly 0.5 https://jack-vanlightly.com/blog/category/People+and+Practice monthly 0.5 https://jack-vanlightly.com/blog/category/AI monthly 0.5 https://jack-vanlightly.com/blog/category/Data monthly 0.5 https://jack-vanlightly.com/blog/category/Security monthly 0.5 https://jack-vanlightly.com/blog/category/Distributed+Systems monthly 0.5 https://jack-vanlightly.com/blog/category/Essays monthly 0.5 https://jack-vanlightly.com/blog/category/Formal+verification monthly 0.5 https://jack-vanlightly.com/blog/tag/Application+Support monthly 0.5 https://jack-vanlightly.com/blog/tag/Apache+ZooKeeper monthly 0.5 https://jack-vanlightly.com/blog/tag/Security monthly 0.5 https://jack-vanlightly.com/blog/tag/Architecture+Visualization monthly 0.5 https://jack-vanlightly.com/blog/tag/Synkronizr monthly 0.5 https://jack-vanlightly.com/blog/tag/Apache+Pulsar monthly 0.5 https://jack-vanlightly.com/blog/tag/STS monthly 0.5 https://jack-vanlightly.com/blog/tag/Processing+Pipelines monthly 0.5 https://jack-vanlightly.com/blog/tag/Event-Driven+Architectures monthly 0.5 https://jack-vanlightly.com/blog/tag/Osteopathy monthly 0.5 https://jack-vanlightly.com/blog/tag/Testing monthly 0.5 https://jack-vanlightly.com/blog/tag/High+Availability monthly 0.5 https://jack-vanlightly.com/blog/tag/Micro-Batch monthly 0.5 https://jack-vanlightly.com/blog/tag/TLA%2B monthly 0.5 https://jack-vanlightly.com/blog/tag/AgileTM monthly 0.5 https://jack-vanlightly.com/blog/tag/Durability monthly 0.5 https://jack-vanlightly.com/blog/tag/Team+Leadership monthly 0.5 https://jack-vanlightly.com/blog/tag/Redshift monthly 0.5 https://jack-vanlightly.com/blog/tag/Rebalanser monthly 0.5 https://jack-vanlightly.com/blog/tag/Log+Analysis monthly 0.5 https://jack-vanlightly.com/blog/tag/IAM monthly 0.5 https://jack-vanlightly.com/blog/tag/Apache+BookKeeper monthly 0.5 https://jack-vanlightly.com/blog/tag/Regular+Expressions monthly 0.5 https://jack-vanlightly.com/blog/tag/C%23 monthly 0.5 https://jack-vanlightly.com/blog/tag/Coaching monthly 0.5 https://jack-vanlightly.com/blog/tag/Visual+Impairment monthly 0.5 https://jack-vanlightly.com/blog/tag/Kafka monthly 0.5 https://jack-vanlightly.com/blog/tag/Code+Quality monthly 0.5 https://jack-vanlightly.com/blog/tag/NServiceBus monthly 0.5 https://jack-vanlightly.com/blog/tag/Formal+Verification monthly 0.5 https://jack-vanlightly.com/blog/tag/Databases monthly 0.5 https://jack-vanlightly.com/blog/tag/DSLs monthly 0.5 https://jack-vanlightly.com/blog/tag/Agile monthly 0.5 https://jack-vanlightly.com/blog/tag/Scrum monthly 0.5 https://jack-vanlightly.com/blog/tag/AWS monthly 0.5 https://jack-vanlightly.com/blog/tag/Eye+Pain monthly 0.5 https://jack-vanlightly.com/blog/tag/Integration monthly 0.5 https://jack-vanlightly.com/blog/tag/Health monthly 0.5 https://jack-vanlightly.com/blog/tag/Taskling monthly 0.5 https://jack-vanlightly.com/blog/tag/SQL+Server monthly 0.5 https://jack-vanlightly.com/blog/tag/TPL+Dataflow monthly 0.5 https://jack-vanlightly.com/blog/tag/Rx.NET monthly 0.5 https://jack-vanlightly.com/blog/tag/Batch+Jobs monthly 0.5 https://jack-vanlightly.com/blog/tag/RabbitMQ monthly 0.5 https://jack-vanlightly.com/blog/tag/Neo4J monthly 0.5 https://jack-vanlightly.com/blog/tag/Chaos+Testing monthly 0.5 https://jack-vanlightly.com/blog/tag/Writing monthly 0.5 https://jack-vanlightly.com/blog/tag/Messaging monthly 0.5 https://jack-vanlightly.com/blog/tag/Apache+Kafka monthly 0.5 https://jack-vanlightly.com/blog/tag/Raft monthly 0.5 https://jack-vanlightly.com/blog/tag/Redpanda monthly 0.5 https://jack-vanlightly.com/blog/tag/Fault+Tolerance monthly 0.5 https://jack-vanlightly.com/sketches daily 0.75 2024-10-28 https://jack-vanlightly.com/sketches/2024/10/28/on-merging-apache-iceberg-and-delta-lake-to-make-a-new-standard monthly 0.5 2024-10-28 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f1b5b5bd-9344-44df-b02c-284fb8429c4b/standards_3x.png Sketches - On merging Apache Iceberg and Delta Lake to make a new standard... - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2024/2/7/three-stages-of-engineering-mindset monthly 0.5 2024-02-07 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e152488d-e5d3-4469-927d-c4b1a5f78532/ThreeStagesEngineeringMindset.png Sketches - Three stages of engineering mindset - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2023/8/23/benchmarketing-games monthly 0.5 2023-08-24 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8453572e-a126-4d32-9b52-b43bc4e98e66/BenchmarketingGames.jpg Sketches - Benchmarketing Games - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2023/8/3/bring-your-own-cloud-security monthly 0.5 2023-08-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/71f5e3be-3c6e-4890-b0a2-3d5bb37825d4/BYOC_Security.jpg Sketches - Bring Your Own Cloud! Security - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2023/8/1/bring-your-own-cloud-billing monthly 0.5 2023-08-01 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9867262c-26a6-4f62-985e-c8f3663c73e1/BYOC_Billing.jpg Sketches - Bring Your Own Cloud! Billing - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2023/7/31/bring-your-own-cloud-data-confidentiality monthly 0.5 2023-07-31 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b41b012e-db3c-4156-bf65-c22487db82f3/BYOC_DataConfidentiality.jpg Sketches - Bring Your Own Cloud! Data Confidentiality - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2023/6/27/scaling-and-economics-of-distributed-data-systems monthly 0.5 2023-06-27 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/97e538ec-2e73-48da-a3b4-9a420688da98/MultiTenancy.png Sketches - Scaling and economics of distributed data systems - Make it stand out Inspired by the excellent blog post Surprising Scalability of Multitenancy by Marc Brooker. https://jack-vanlightly.com/sketches/2023/1/31/selling-tco monthly 0.5 2023-01-31 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d3834344-61a6-4869-ad3c-6e5d3580d7b4/SellingTCO.png Sketches - Selling TCO - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2023/1/30/estimating-the-cloud-infrastructure-bill monthly 0.5 2023-01-30 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1bd7577a-ac7d-4be8-952f-12f16b0dafb3/AreWeMissingSomething.png Sketches - Estimating the cloud infrastructure bill - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/29/brain-fail-part-3 monthly 0.5 2022-11-29 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3016587d-12ae-4556-878f-820a6d9db6bd/BrainFailPart3.png Sketches - Brain fail - part 3 - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/26/brain-fail-part-2 monthly 0.5 2022-11-26 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/65aedb01-bdb7-44b9-aeef-efdcfbab71a2/BrainFailPart2.png Sketches - Brain fail - part 2 - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/26/brain-fail-part-1 monthly 0.5 2022-11-26 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bc7de575-1e14-40ce-a442-f8e2308cca73/BrainFailPart1.png Sketches - Brain fail - part 1 - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/17/data-governance monthly 0.5 2022-11-17 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ea9ccfef-7fe5-4273-8a11-e191f7dbfeec/Governance.png Sketches - Data governance - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/16/a-tribute-to-apache-zookeeper monthly 0.5 2022-11-16 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9203b712-d632-4c47-bffb-0441c563d171/ZooKeeperTribute.png Sketches - A tribute to Apache ZooKeeper - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/14/everyone-has-connectors-these-days monthly 0.5 2022-11-14 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4908a400-5c2c-4b61-9c84-637e3e4a6d14/ELTCorpNewConnector.png Sketches - Everyone has connectors these days - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/3/the-total-cost-of-ownership-marketing-guide monthly 0.5 2022-11-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d71127d5-22d9-44cc-a257-5da8dbf1d07f/TCO.png Sketches - The Total Cost of Ownership marketing guide - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/11/3/the-benchmarketing-101-guide monthly 0.5 2022-11-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a96b2f4a-8347-498f-aac8-274d632b5c98/Benchmarketing101.png Sketches - The benchmarketing 101 guide - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/7/2/theyll-figure-it-out-one-way-or-another monthly 0.5 2022-07-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aec5d327-1b8d-4c05-ab46-25f68706b23b/TlaJepsenFinal.png Sketches - They'll figure it out, one way or another - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2022/1/20/tla-specification-cycle monthly 0.5 2022-01-20 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e4c471cf-1cbb-46d5-a3d7-a6ddf28fe81b/tlaplus-cycle.png Sketches - TLA+ specification cycle - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2021/6/17/distributed-systems-are-hard monthly 0.5 2024-10-31 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6d4098ff-69e9-4e8b-8967-59e44f6b900b/DIstSysHardAll.jpg Sketches - Distributed Systems Are Hard - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/sketches/2019/10/7/kafka-topics-are-shared-resources monthly 0.5 2019-10-07 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1570457600723-U9HHGRMEJJHPCJMH6XND/SharedTopicsVsIndependentQueue.png Sketches - Kafka topics are shared resources https://jack-vanlightly.com/sketches/2019/3/14/being-an-enterprise-developer monthly 0.5 2024-10-31 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/555fb64e-877e-4468-b0ad-de60383b1f84/Being-an-enterprise-developer.png Sketches - Being an enterprise developer https://jack-vanlightly.com/sketches/2018/12/17/the-value-of-integration-tests monthly 0.5 2018-12-17 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1545082806868-OWN04KQI06A8AGDQ2MEW/MarsMission.png Sketches - The Value of Integration Tests https://jack-vanlightly.com/sketches/2018/10/7/rabbitmq-mirror-rejoin-procedure-sketch monthly 0.5 2018-10-07 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538914961655-TQQJJSB79OMLH0N209OQ/RabbitMqDiscard.png Sketches - RabbitMQ - Mirror Rejoin Procedure (Sketch) https://jack-vanlightly.com/sketches/2018/10/2/kafka-vs-pulsar-rebalancing-sketch monthly 0.5 2018-10-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1538494255595-8M03XPVR7YCI2Y6V1U6G/KafkaPulsarScaling.png Sketches - Kafka vs Pulsar - Rebalancing (Sketch) https://jack-vanlightly.com/sketches/category/Sketch monthly 0.5 https://jack-vanlightly.com/sketches/tag/RabbitMQ monthly 0.5 https://jack-vanlightly.com/sketches/tag/Apache+Pulsar monthly 0.5 https://jack-vanlightly.com/sketches/tag/TLA%2B monthly 0.5 https://jack-vanlightly.com/sketches/tag/Apache+Kafka monthly 0.5 https://jack-vanlightly.com/sketches/tag/Distributed+Systems monthly 0.5 https://jack-vanlightly.com/analyses daily 0.75 2024-12-22 https://jack-vanlightly.com/analyses/2024/12/22/verifying-kafka-transactions-diary-entry-5-addpartitionstotxn-in-tla monthly 0.5 2024-12-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1d52c1d8-80f7-4156-b11c-7ec2dd5aa788/add_happy_path_small.png Analyses - Verifying Kafka transactions - Diary entry 5 - AddPartitionsToTxn in TLA+ - Make it stand out Fig 1. The happy path for a producer obtaining a PID and adding a first set of partitions to a transaction. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/36a05994-a613-4736-8e5c-d570ae8eef8f/fence_and_abort_path_small.png Analyses - Verifying Kafka transactions - Diary entry 5 - AddPartitionsToTxn in TLA+ - Make it stand out Fig 2. The path followed when another producer obtains a PID using the same Transaction Id as the first producer (from the happy path). https://jack-vanlightly.com/analyses/2024/12/5/verifying-kafka-transactions-diary-entry-4-writing-an-initial-fizzbee-spec monthly 0.5 2024-12-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/061628b1-358d-49b2-b46d-28d1558178a5/bee-left-to-right.png Analyses - Verifying Kafka transactions - Diary entry 4 - Writing an initial Fizzbee spec - Make it stand out Fizzbee.io https://jack-vanlightly.com/analyses/2024/12/4/verifying-kafka-transactions-diary-entry-3-getting-confidence-in-the-tla-spec monthly 0.5 2024-12-04 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/949ad6c6-da00-47ca-aaeb-5d5e0ad996dd/ConcurrentTransactions.png Analyses - Verifying Kafka transactions - Diary entry 3 - Getting confidence in the TLA+ spec - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/594463f0-a8ce-4371-b4d6-22e6dfc57d6b/NotCoordinator.png Analyses - Verifying Kafka transactions - Diary entry 3 - Getting confidence in the TLA+ spec - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/analyses/2024/12/3/verifying-kafka-transactions-diary-entry-2-writing-an-initial-tla-spec monthly 0.5 2024-12-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/05471bb3-b9d9-433c-a577-a3f0e13c746a/state_variables_small.png Analyses - Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec - Make it stand out Fig 1. The variables client, tc_txn_metadata, tc_txn_transition, tc_part_metadata, txn_log, txn_log_hwm. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5173fc78-4b3c-40d9-9eab-30aee0212f36/atomic_actions_small.png Analyses - Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec - Make it stand out Fig 2. The four atomic actions and how they map to the sequence diagram of the last diary entry. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ece7a00e-888c-41a4-8c35-a8c5be18cddb/eventually_gets_pid_small.png Analyses - Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec - Make it stand out Fig 3. A history where a client must retry its InitPidRequest, ulitmately resulting in both clients obtaining a PID, though with one client with a now fenced epoch. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e2729be4-3d5b-40a3-a6b8-2eb08199e6ca/states_space_without_processed_msgs.png Analyses - Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec - Make it stand out Fig 4. A simplified view of a state space without the history of previously processed messages being used to compute fingerprints. Max length of 5 and 10 unique states. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6d267df9-95b9-4183-b29f-0e8e31d92416/states_space_with_processed_msgs_small.png Analyses - Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec - Make it stand out Fig 5. A simplified view of a state space with the history of previously processed messages included in state fingerprints. Max length of 11 and 33 unique states. https://jack-vanlightly.com/analyses/2024/12/2/verifying-kafka-transactions-diary-entry-1-a-first-step monthly 0.5 2024-12-02 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0fdb8929-f4d9-4a6f-8484-ec9eb6912a4d/MainActors.png Analyses - Verifying Kafka transactions - Diary entry 1 - A first step - Make it stand out Fig 1. A diagram from the design doc that describes principal actors with numbered references to the message exchanges. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b5053909-96fc-4608-8ccb-f7607e8d9ac6/InitPidRequestResponseSmall.png Analyses - Verifying Kafka transactions - Diary entry 1 - A first step - Make it stand out Fig 2. The basic happy path and unhappy path of obtaining a PID. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/733f5e3c-9738-4fa8-b1ec-0a8e22be272c/ClientStateMachineSmall.png Analyses - Verifying Kafka transactions - Diary entry 1 - A first step - Make it stand out Fig 3. A client modeled as a finite state machine. The circles represent the states and the arrows are the transitions. https://jack-vanlightly.com/analyses/2024/10/9/change-query-support-in-apache-paimon monthly 0.5 2024-10-09 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/814c051a-16fc-4237-943b-c8c4231f4ad3/LogOfSnapshotsSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 1. Depicts a log of snapshots, highlighting the data files added by each snapshot (that are listed in each delta manifest list). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d5b01c1d-23b4-48ab-8b0e-94c9a306d5cf/BucketsSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 2. Partitions create data locality by grouping data by the partition keys. Buckets provide for parallelism of compute within a partition. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6a0f71d9-4a86-4f06-bda5-3ca9004859e4/CompactionSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 3. Paimon writers (such as a sink task in a Flink topology) write to level 0. Data is compacted and moved to higher levels by compaction jobs. The maximum level is configurable. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cc22db7c-07fb-4bdf-a305-167263acae2a/SortedRunsSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 4. Shows two sorted-runs of level 0 where the ‘name’ column is the primary key. There is key overlap between the two sorted-runs, but not between files within any single sorted-run. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/65036d65-940e-45b8-b5ab-c18ccfe9a1d5/OperationsToDataFilesSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 5. A set of operations are written as data files (the above would be written as three sorted-runs). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a4795d5b-2450-428a-8396-52f2a60e929f/ScanAndMergeSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 6. The scanned rows are merged to generate a result. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f567c999-7512-4c4c-a2b6-b34117d1db49/CompactionDetailedSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 7. A compaction scans the files of the input levels, merges the rows and writes them as one or more data files of a single sorted-run in the output level. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9f3ab6b3-78c5-4486-8f0c-27be2a07115a/LogOfSnapshotsANdCompactionSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 8. Shows the snapshots of three commits to level 0, followed by a compaction that logically deletes the first three data files and rewrites them as data file #4 in a higher level. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2e7cd485-069a-424b-af0d-d2bcf43bca9f/CompactionBeforeAndAfterSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 9. The data files before and after the full compaction. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/795fe0cf-6cc7-4e85-8458-b8ff59ed421e/CompactionBeforeAndAfterWithCDCSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 10. The data and CDC files before and after the compaction. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cd630d3f-c339-49b5-960d-f5c8c0066207/LookupsSmall.png Analyses - Change query support in Apache Paimon (0.8) - Make it stand out Fig 11. Lookups to higher levels are made to find the prior row state. https://jack-vanlightly.com/analyses/2024/9/27/change-query-support-in-apache-hudi monthly 0.5 2024-09-30 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0b2e92b9-d969-49f6-9580-f5d16cdf9f62/ThreeCommitsSmall.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 1. The three commits used throughout this series, and the expected table scans executed at different timestamps. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/dda37919-c22a-4c37-b13f-a3888996809a/ThreeCommitsFileGroupSmall.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 2. Depicts the same three operations on a file group, either with COW or MOR. The MOR table also depicts a compaction producing a new file slice consisting of a base file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5148d029-f91d-4d0f-9ad8-d576089aec47/IncrementallyReadTimelineSmall.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 3. Depicts a timeline with three file groups in the data layer. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/84d224d8-55d9-4b87-ae6a-549c85044397/CommitTsColSmall.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 4. Depicts the row_commit_ts metadata column in a COW table file group. It also works the same way in MOR tables. Soft deletes in Hudi are upserts with the columns nulled out. A hard-delete is a delete operation that removes the row. Both types of delete are treated the same way by reads. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3ccd30ae-20d9-4de1-9dc0-9252ef3a31c6/cdc_data_before_after_small.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 5. A CDC reader reads a mix of base files and CDC files to return CDC query results. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b7d7acc7-1b2d-4d01-aa58-04326cd50ae4/cdc_data_before_small.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 6. A CDC reader reads a mix of base files and CDC files to return CDC query results. It must source the after image of the row in commit ts=2 from the file slice of the current commit (at that time, being ts=2). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f3841b68-fd08-417c-9d3b-fc9e8435ce58/cdc_data_op_key_small.png Analyses - Change query support in Apache Hudi (0.15) - Make it stand out Fig 7. With OP_KEY_ONLY, the reader must load the file slice at the current and previous commit for the before and after images. https://jack-vanlightly.com/analyses/2024/9/24/change-query-support-in-delta-lake monthly 0.5 2024-09-27 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cd60317c-2fc6-4737-a95c-cfcf6b9d20c1/DeltaLogSmall.png Analyses - Change query support in Delta Lake (3.2.0) - Make it stand out Fig 1. Compute engines can walk forwards through the delta log entries by sorting them lexicographically. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6d6745ad-29e2-4305-ba19-bd29a02d79f8/DeltaCOWSmall.png Analyses - Change query support in Delta Lake (3.2.0) - Make it stand out Fig 2. Example of the three operations using copy-on-write. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/55747cdf-2fee-4ff5-8d25-2a0a1fe06cb9/DeltaMORSmall.png Analyses - Change query support in Delta Lake (3.2.0) - Make it stand out Fig 3. The same three operations, but with deletion vectors enabled (making this a merge-on-read table). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3e54f640-03c3-41dc-bbbf-275dbf28c977/MORSparkSSExplainedSmall.png Analyses - Change query support in Delta Lake (3.2.0) - Make it stand out Fig 4. Explaining the results of the last Spark SS query, https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0b68a087-b6ee-45cb-a018-159cc30abe53/DeltaCdcCowSmall.png Analyses - Change query support in Delta Lake (3.2.0) - Make it stand out Fig 5. Depicts a simplified view of a Delta Lake table that materializes changes as CDC files, with deletion vectors disabled (copy-on-write). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2cf0b7a3-b6cd-44cf-a327-4b90e593e4b1/DeltaCdcMORSmall.png Analyses - Change query support in Delta Lake (3.2.0) - Make it stand out Fig 6. Depicts a simplified view of a Delta Lake table that materializes changes as CDC files, with deletion vectors enabled (merge-on-read). https://jack-vanlightly.com/analyses/2024/9/23/change-query-support-in-apache-iceberg-v2 monthly 0.5 2024-09-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/06eadf3f-792d-417f-b069-60ce8da18e3d/COW-cdc-small.png Analyses - Change query support in Apache Iceberg v2 - Make it stand out Fig 1. The deleted and added data files of snapshot S are compared to yield a CDC result. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/041aeb23-45ad-4acd-8eb4-c21d3d823344/SnowflakeQueryPlanning.PNG Analyses - Change query support in Apache Iceberg v2 - Make it stand out Fig 2. Taken from the Snowflake paper: What’s the Difference? Incremental Processing with Change Queries in Snowflake. Snowflake tables use copy-on-write (COW). Shows a Snowflake min-delta query plan that results in the rows returned from scanning multiple added data files (known as partitions in Snowflake) and multiple (logically) deleted data files being unioned into a diffing phase. A similar approach is required for Iceberg COW tables. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4fd3a0f2-f392-4ea7-823e-2f90487627a3/MERGE-small.png Analyses - Change query support in Apache Iceberg v2 - Make it stand out Fig 3. A merge statement synchronizes a source and target table. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/493d5c26-c2bd-4288-8472-9cdbc0004814/MOR-cdc-small.png Analyses - Change query support in Apache Iceberg v2 - Make it stand out Fig 4. The added data file and delete file of snapshot S, as well as the referenced data file of the two delete file entries, are read to yield a CDC result. https://jack-vanlightly.com/analyses/2024/8/6/apache-icebergs-consistency-model-part-3 monthly 0.5 2024-08-07 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/04539edf-c7af-4278-b1e7-ca85431f01a8/LinearizedHistorySmall.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 3 - Make it stand out Fig 1. How a series of operations is registered as a set of column value histories. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7dd0b711-ae76-4295-87ca-eb304480d5b2/CounterexampleMetadataSmall.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 3 - Make it stand out Fig 2. A simplified view of the files of the table at the end of the counterexample. https://jack-vanlightly.com/analyses/2024/8/5/apache-icebergs-consistency-model-part-2 monthly 0.5 2024-08-07 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7250117d-42a0-44ad-811c-4b7cc3429480/IcebergWriteStepsSmall.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 1. The steps of an Iceberg write. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/74b2cb44-58fd-4f9a-bb73-1d1d8d203b91/Interleaving1Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 2. Writer 1 and 2 start their operations at the same time, based on the same snapshot. Writer 1 quickly commits snapshot-2. When writer 2 reaches the Refresh Metadata stage, it loads snapshot-2. This is a new snapshot that has been committed since its scan snapshot. It performs conflict detection against that snapshot, which passes in this case, and then writes its metadata, based on snapshot-2. The commit is successful as it is a CAS of metadata-2 for metadata-3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c1015b55-a23b-4e76-a9c6-900ebef79c1f/Interleaving2Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 3. Two writers concurrently go through the write steps, but writer 1 commits first though after writer 2 refreshes its metadata. Writer 2 fails to commit, so goes back and refreshes its metadata again, reruns the conflict check, writes the metadata based on snapshot-2, and commits successfully. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f1a9e072-86aa-4a4d-b17f-3b6cea95092a/DataConflictSmall.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 4. In the top, two concurrent operations read and modify disjoint sets of data files and have no data conflict. In the bottom, the files that are read and modified by two concurrent operations overlap and thus have conflicting data changes. What counts as overlap depends on the isolation level desired. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aaa549e8-906e-4ac0-be8b-85616aa371c6/Interleaving3Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 5. In this case, the data conflict is not detected in the first conflict check as writer 1 had not committed yet. The second time around, writer 2’s data conflict check detects the conflict, and it aborts. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8ee4ec5d-3595-41ef-9bd8-c8ce9f0d8869/DataConflictFilterSmall.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 6. A data conflict filter causes some manifest entries to be skipped when checking for conflicts. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/23e56716-5a6d-40b8-a554-3e474d94c97b/OverwriteFilesExampleConflict1Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 7. Operation A just committed, creating snapshot-2, which deleted data-1. Operation B, an OverwriteFiles, also has logically deleted data-1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c08dbb30-c7bd-44d9-9fb0-f8d08d0313da/OverwriteFilesExampleConflict2Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 8. A simplified representation of snapshots and the data files they would include if Operation B were allowed to commit. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/48fbf085-faab-474b-9b05-2424f2c0a033/FailMissingDeletePaths1Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 9. This operation logically deletes data-2. The manifest entry status is changed to DELETED, as its data file path is a member of the deleted-set. The validation check ensures that all paths in the deleted-set have a corresponding manifest entry with status DELETED. In this case it does, so the check passes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/743955cb-2529-4558-b638-095d63d955d2/FailMissingDeletePaths2Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 10. A previous operation deleted data-2 and it gets filtered out in the logical manifest. The check does not find a manifest entry for data-2 and the check fails. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/153b24df-7a29-46e4-b2b7-596571a21b7d/RowDeltaExampleConflict1Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 11. Operation B adds a delete file that references data-1 which was just deleted by Operation B. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3ef6dba9-6c2f-4c8d-95d2-5d0f9cc12cea/RowDeltaExampleConflict2Small.png Analyses - Understanding Apache Iceberg’s Consistency Model Part 2 - Make it stand out Fig 12. Snapshot-3 would return two rows for “sarah” with different values, and a delete file that points to a data file that is not part of the table’s dataset. https://jack-vanlightly.com/analyses/2024/7/30/understanding-apache-icebergs-consistency-model-part1 monthly 0.5 2024-08-05 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e29f02a5-d407-4622-b7f6-cfc2f43c5629/CatalogMetadataAndDataFilesSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 1. A writer writes metadata that references data files and commits the metadata’s location to a catalog. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8804b4c7-ebc5-4415-8193-75b52a76d64d/SnapshotSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 2. A snapshot forms a tree of files that make up the table at a point in time. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9d008d23-2fd5-4cd3-b5f5-fba19a82103d/MetadataFilesSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 3. Each write operation creates a new metadata file that contains the new snapshot and all prior snapshots. Snapshots eventually must get expired to prevent the snapshot log growing to large. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1360ef32-a4f0-43a0-847c-727445aa6fee/MetadataCommitSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 4. Two writers performing a concurrent operation against the table attempt to perform the atomic CAS using the same original metadata location. The second commit is rejected. Note that metadata files are appended with a UUID to avoid collisions. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c1579123-31e5-4373-83c1-78d32bf22d07/ThreeInsertsSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 5. Three insert operations result in three live snapshots (each representing a different table version). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/930e38a4-e17c-4a7a-a0ab-59ac7cdef4b0/InsertInsertDeleteSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 6. Snapshot 3 only references manifest-2 and manifest-3, tracking the delete of data-1 in manifest-3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/630eb51a-f1b0-4d86-b8f1-1745488ac1c6/ManifestLifecycleExampleSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 7. The lifecycle of a manifest file created in snapshot-2, read in snapshot-3 (to a logical version), has an entry deleted and finally gets written as a new manifest file in snapshot-4. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7f0945e8-9ef8-41ea-9253-3d4e7ff4979e/AddedDeletedExistingMultipleSnapshotsSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 8. Manifest files across 5 snapshots. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/052b3ea1-7d14-47bc-9168-9c13c0d1f447/CopyOnWriteSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 9. data-1 gets completely rewritten as data-2 due to a single row being updated. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c4646688-8904-4a10-965c-c2d0c1aa08a4/CowMetadataSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 10. The copy-on-write operation results in a new snapshot where data-1 has been deleted and data-2 has been added. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ec9c81d1-0399-43e2-882e-143addd9e71f/MergeOnReadPositionDeletesSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 11. data-1 remains in place in the new table version, with the new value written to data-2 and the original row invalidated by a delete file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9cc1fede-f5b0-48cf-9dcc-e71f5bad4cc6/MergeOnReadPosDeletesMetadataSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 12. The merge-on-read operation results in a new snapshot where data-1, data-2 and delete-1 are all members of the new table version. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/847f579d-ef02-4cf5-8d81-a85ba13411ca/MergeOnReadEqualityDeletesSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 13. The same set of files as position deletes, only that the delete file entry is an equality delete. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8a4c177c-fd5c-4761-a9fc-b26e8bf47b18/MergeOnReadEqualityDeletesMetadataSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 14. The same metadata as with position deletes except that this diagram includes the sequence numbers and the delete file contains an equality delete. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/50252a19-6385-4967-b0db-70bbbbe2168f/CompactionSmallpng.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 15. Compaction rewrites data-1, data-2 and delete-1 as data-3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/37abf2eb-ab52-46dd-b7d2-0e35888336fe/ClusteringSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 16. Data is unordered across files on the left, but sorted across files on the right leading to better query performance that can prune more files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f1e840a4-d027-497b-9e84-0748410e0312/PartitionEvolutionSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 17. One table, two sets of data files with different partition specs. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/76bd69c6-0fdc-414e-ada8-25cce30f0cec/WritePathSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 18. Simplified model of an Iceberg write. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ccc1f717-4825-47a7-9cab-f5826c0fff18/IcebergModulesSmall.png Analyses - Understanding Apache Iceberg's Consistency Model Part 1 - Make it stand out Fig 19. Some of the modules of the Apache Iceberg project. Note that some compute engine Iceberg adaptor modules are hosted in the compute engine’s repo, such as Trino’s Iceberg plugin. https://jack-vanlightly.com/analyses/2024/7/3/understanding-apache-paimon-consistency-model-part-3 monthly 0.5 2024-07-03 https://jack-vanlightly.com/analyses/2024/7/3/understanding-apache-paimon-consistency-model-part-2 monthly 0.5 2024-07-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/303eb978-5719-44b0-9d82-12616a3ab0ad/FlinkTopologySmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 2 - Make it stand out Fig 11. Simplified Flink topology for a Paimon sink. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2dc876b6-36db-49f0-b12c-a24e6ccc7078/Topology1And2Small.png Analyses - Understanding Apache Paimon's Consistency Model Part 2 - Make it stand out Fig 12. Alternative writer and committer topologies. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b72e199b-74ab-473c-9e60-9eb9b5a18805/WriteAndCommitSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 2 - Make it stand out Fig 13. The steps performed by writer-committers. In a Flink topology, steps 0-1 would be run by writers and the rest run by the single committer task. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b993436b-c1ab-4855-8805-a172c0f460d1/LostUpdatesSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 2 - Make it stand out Fig 14. At the top, two transactions start at the same time, and attempt to modify the row {‘Jack’, ‘Yellow’, ‘Hiking’}. The first transaction changes FavColor to ‘Blue’ and commits successfully. The second transaction only wants to change FavHobby to ‘Cycling’. If transaction 2 is allowed to commit it overwrites the commit of the first, resulting in [‘Jack’, ‘Yellow’, ‘Cycling’] which is a Lost Update anomaly. On the bottom, two clients each want to add one point to Jack. They each begin a transaction, then obtain the current number of points (1), then write an update row with points=2. If the second transaction is allowed to go through, then again, we have an anomaly violating snapshot isolation. https://jack-vanlightly.com/analyses/2024/7/3/understanding-apache-paimon-consistency-model-part-1 monthly 0.5 2024-07-03 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c686cef2-84b4-4b43-91a9-5890128ab9e5/MetadataTreeSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 1. A tree of metadata and data files forming one point in time (one version) of a Paimon table. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a48f6529-0dee-4142-bf47-4b24eaa805d2/ManifestListBaseAndDeltaSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 2. The base manifest list files. In reality, manifest files also get compacted to reduce the number of metadata files, which is not reflected here. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3ac4d131-0bef-41e3-88be-783749576a42/PartitionsAndBucketsSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 3. A table of two partitions (by date) and four buckets per partition. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9bca9470-81a2-40fd-bc5a-697413f670b9/FileStructure.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 4. File organization (from https://paimon.apache.org/docs/master/concepts/specification) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8cf1e4b1-da5c-4489-b44f-25d4bbecafea/Architecture-of-an-LSM-tree-as-implemented-by-RocksDB-taken-from-RocksDB-wiki-11.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 5. An LSM tree architecture, taken from https://www.researchgate.net/figure/Architecture-of-an-LSM-tree-as-implemented-by-RocksDB-taken-from-RocksDB-wiki-11_fig2_358081916 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aea3fff0-e3f1-4f05-b8b7-385f89281cab/CompactionSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 6. Time-aligned compaction. Higher levels are further back in time. Time-based expiry can be very efficient as it can mean just deleting entire files as they fall off the end of the retention period. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/93ac5bee-4a92-46f2-8d21-26bdb523434e/RowsAndFilesSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 7. A sequence of insert, update and delete operations translated into rows across data files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/291dfece-2c10-40bc-985d-94836b4579f6/DVFIleSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 8. The deletion vector file of the bucket has 5 deletion vectors invalidating 5 rows in 2 data files. Level 0 files are never invalidated. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8e9467cf-c723-4c3c-98a2-42f4f9535975/CompactionDVSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 9. Before compaction, two valid rows exist for “Jack”, one in level 0 and one in level 3. After the compaction, the row in level 3 has been invalidated by a deletion vector entry in the bucket deletion vector file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/55c86862-0bdd-465b-a104-0318bd3a4f0d/CompactionDetailedSmall.png Analyses - Understanding Apache Paimon's Consistency Model Part 1 - Make it stand out Fig 10. Deletion vectors are added during compactions when a row in a higher level (than that of the levels being compacted) is invalidated by a row in the compacted levels. Deletion vectors are removed when they point to a file that gets logically deleted. DV files themselves only get deleted when they ,and the other data files of their version, get expired. https://jack-vanlightly.com/analyses/2024/4/29/understanding-delta-lakes-consistency-model monthly 0.5 2024-04-29 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/aed400d1-6eb0-4510-8c1b-2a9110dd1f0b/DeltaLogAndDataFiles.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4232ebb7-6cb0-4f61-a60c-54b7ee53d559/colors_example1.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/06ed81b5-1d04-4990-8b05-c481d4f23f8f/colors_example2.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/715d01a1-84f2-4545-9575-7775b3cc7eae/write_path_1.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/68b6f8ae-36fe-40e7-a713-772117c4feac/cow_example_1.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7ac04a5d-4187-45a5-9696-8a93e6a8ccd9/mor_example_1.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e5619184-8610-4447-ac6f-4394302bed36/no_occ1.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9d3e5a84-85c3-4b4a-b82b-ea3fae255cbf/no_occ.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d881c358-4a63-4200-8f61-7f43f51fdf5e/put_if_absent_occ.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3bf89945-ce85-4463-b758-2b2e412c3dbc/partitions_conflict.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d5c0de2c-beeb-4a3d-b763-b44525dabb10/coordination_occ.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/acce2f50-abae-47e6-85e3-0582f48904bc/read_path_small.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/73683c93-08c9-45b7-a73e-6956dbb8d12c/next_state_formula.png Analyses - Understanding Delta Lake's consistency model - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/analyses/2024/4/25/understanding-apache-hudi-consistency-model-part-3 monthly 0.5 2024-04-24 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/341d28f1-a016-4402-b3cf-7f5d7df50d85/hudi_multi_writer_guarantees.png Analyses - Understanding Apache Hudi's Consistency Model Part 3 - Make it stand out https://hudi.apache.org/docs/concurrency_control https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/93f4dad9-29bb-43be-a9e8-84cafcddadfb/trace1.png Analyses - Understanding Apache Hudi's Consistency Model Part 3 - Make it stand out Fig 1. The issue is that concurrent operations of the different primary keys were mapped to the same file group, and both writers read the timeline at the same time, not finding any existing file slice. This led to the second operation not merging the contents of the first operation, resulting in a lost write of primary key k1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/170a0296-a699-4af1-9650-10bb4b8de12b/trace2.png Analyses - Understanding Apache Hudi's Consistency Model Part 3 - Make it stand out Fig 2. The concurrency control check of w2 scanned the timeline and found the completed instant of w1, which touched the same file group as the operation of w2. Writer w2 had no merge target for its upsert, so used a timestamp of 0 for the check. The completed instant of w1 had a higher timestamp than 0, so a conflict was detected. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/15d771b8-9ef4-460e-9cc3-00b1a76e7fd7/trace3.png Analyses - Understanding Apache Hudi's Consistency Model Part 3 - Make it stand out Fig 3. If PK conflict detection had been used, w2 would have seen that a mapping now existed for key k1, which conflicted with its own assignment, and it would have failed the check and aborted. Because it didn’t do that, it overwrote the mapping of w1 and orphaned the row in file group 1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3e4b06bb-cf57-4037-be04-097028f002e5/trace4.png Analyses - Understanding Apache Hudi's Consistency Model Part 3 - Make it stand out Fig 4. Both writers chose timestamp ts=1. While the OCC check prevented the second operation from completing, it did not prevent the file slice of the first op being overwritten with the file slice of the second op (as the file names were exactly the same). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/dc8df207-2703-4d96-bd88-d8fc1d01d4df/trace5.png Analyses - Understanding Apache Hudi's Consistency Model Part 3 - Make it stand out Fig 5. Writer w2 aborts on its put-if-absent of the instant 1.commit.requested. https://jack-vanlightly.com/analyses/2024/4/24/understanding-apache-hudi-consistency-model-part-2 monthly 0.5 2024-04-24 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8752a20a-2163-4fa1-a027-01f02201ccf1/collision_trace1.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 1. A completed instant in the timeline gets overwritten, orphaning a committed file slice. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ae0dc33a-df9f-4394-a6be-692c5bca8e26/collision_trace2.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 2. A file slice gets overwritten, losing committed rows. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fbfead1b-a685-4877-8cf7-20c80531e4d6/birthday_probem.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/40fec586-3077-48e0-9ffd-aab12d341653/collisions1.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 1. We see that with 1 minute intervals (+/- 3 seconds jitter), we see the same S-curve as the birthday paradox. The probability reaches above 50% in a 24-hour period with 8 writers and asymptotes towards 100%. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/38139ad8-20e8-4055-9bda-368a87650c42/collisions2.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 2. Across the 1000 simulations, the average reaches 1 collision at 9-10 writers. The max tops out at 14 collisions at 20 writers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e8a63b77-ba2d-4425-bb97-d5592ffae152/collisions3.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 3. Increasing the write interval to 5 minutes (+/- 15 seconds) lowered the probability significantly. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cd6a3535-8721-49ee-9672-14a8ba826cbc/collisions4.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 4. We see that the probability of a collision over a 24 hour period drops off quickly as the write interval increases. The curve is not perfect due to this being a simulation with a 1000 runs. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ad5370d2-f11d-4903-93f7-c9a1311ef700/collisions5.png Analyses - Understanding Apache Hudi's Consistency Model Part 2 - Make it stand out Fig 5. Increasing the duration to 7 days naturally increases the probability of a collision. https://jack-vanlightly.com/analyses/2024/4/24/understanding-apache-hudi-consistency-model-part-1 monthly 0.5 2024-10-10 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/345f7064-19aa-4bf5-8065-54fe1baceb75/HudiBasicsSmall.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 1. A writer writes metadata about data files to the timeline (a write-ahead-log) https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1029fa95-08b9-4404-9f77-a13679dbaac4/timeline_order.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 2. Timeline ordering is by timestamp and not insertion order. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3dfaa1b6-7398-4036-b373-553963711068/OperationsAndExpectedScanResultsSmall.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 3. Three commits that insert, update and delete rows, alongwith the expected table scan results at different timestamps. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/142ce66c-c7df-468c-bcc9-d0b3752de706/FileGroupCOWSmall.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 4. Each commit results in a new base file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/03dab8b9-f45e-4fc4-8ba9-8aed61a38d37/FileGroupMORSmall.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 5. Depicts the same three commits, plus a compaction. The three commits result in one base file and two log files on top. A subsequent compaction rewrites the file slice as a new base file. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/594c13d6-293f-4a52-8676-b9c1b3e160fe/timeline_and_fgs1_small.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 6. Timeline completed instants point to immutable data files. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4169dc49-7546-48e0-baa0-6c16f7fc3953/timeline_and_fgs2_small.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 7. The operation at ts=150 fails before writing a completed instant, therefore its file slice remains unreadable. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d038796d-0167-4915-b4b1-8edb6453c29f/write_model_occ.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 8. The write path of the simplified model, with optimistic concurrency control. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/83862a44-97f9-4b20-ab46-6da7e20f9eb0/write_model_pcc.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 9. OCC check and table lock is replaced with per-file-group locks. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9c77ee2d-c634-4ab0-bea7-2e7a96738cbf/tlaplus_next.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 10. The next state formula of the TLA+ specification https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c5fbdabd-573a-495c-aca6-5079102e0be7/fg_conflicts.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 11. Disjoint file group commits have no conflicts. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/57127afc-6791-4ff0-9d0c-a235f0b1caa0/occ1.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 11. Either w1 or w2 could now acquire the table lock and successfully complete the operation. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/34398627-d51c-4def-b077-d5706b291f18/occ2.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 12. The operation at ts=100 cannot commit now as its OCC check would fail. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1aa7316f-35a4-4e61-bff1-33690a7ad231/read_model.png Analyses - Understanding Apache Hudi's Consistency Model Part 1 - Make it stand out Fig 13. The read path of this simplified model. https://jack-vanlightly.com/analyses/2024/3/12/scaling-models-and-multi-tenant-data-systems-asds-chapter-6 monthly 0.5 2024-03-12 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/63e967f5-de68-4eb5-b7ac-11f7406eb54a/single-tenant-pod-scaling.png Analyses - Scaling models and multi-tenant data systems - ASDS Chapter 6 - Make it stand out Fig 1. Scaling single-tenant bin-packed pods to match tenant load changes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d0b9434d-5bbc-4d6f-a80b-0a3c34ccf79c/serverless-white-paper-illustrations-06-1.png Analyses - Scaling models and multi-tenant data systems - ASDS Chapter 6 - Make it stand out Fig 2. CockroachDB auto-scaling with a pre-warmed pod pool. Tile 1: when scaling in, pods are first drained to avoid disruption. Tile 2: scaling out adds a new pod for the tenant (by stmaping a pod from the pre-warmed pool). Tile 3: a tenant is scaled to zero after a period without requests. Tile 4: an inactive tenant is resumed by stamping a pod in the pre-warmed pool. The CockroachDB proxy layer can also move pods between hosts without terminating TCP connections. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/499afeba-dd44-4a11-b50d-e2c713be5a3d/rate_limits.png Analyses - Scaling models and multi-tenant data systems - ASDS Chapter 6 - Make it stand out Fig 3. There is no tenant auto-scaling in this picture, only consumption based pricing with rate limits. https://jack-vanlightly.com/analyses/2024/1/23/serverless-clickhouse-cloud-asds-chapter-5-part-2 monthly 0.5 2024-01-24 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8a921564-1fa6-4e95-a2e6-df721329136d/SharedMergeTree.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 2) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c1cc28b6-3fdb-4abf-8937-6e7b4d70d1b2/SharedMergeTreeSequence.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 2) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/11e09824-7710-4130-bedd-b330ce26872e/ServerlessArchitecture.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 2) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/analyses/2024/1/23/serverless-clickhouse-cloud-asds-chapter-5-part-1 monthly 0.5 2024-01-24 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b6228348-aeae-4c4e-8b86-72177cd195b4/PartMerging.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d610c440-fc3e-4b41-83e2-d6f4d2cedbf4/PartFiles.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/32d33c75-e08b-4595-8304-10f5b8775681/SelfContainedParts.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/94554e9a-217f-46ee-ab4f-7e27dbc1c361/IndexMarksGranules.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/89975f95-f7e6-42c1-a810-86ae9afd6372/Partitioning.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a3798f1f-9d16-4d00-aee0-485d58e733ea/MergingExamples.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/dd52190e-cdab-4ee5-be96-f3f159b95294/PartPruning.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/17bc301b-d477-4a22-98a5-1ed6246b3ba1/MergeTreeEngine.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/18e00771-03ce-4c8e-90bf-b6e1a22fb6bd/DistributedTable.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/abf981fd-893b-4a58-a253-b02934d5eecc/MergeTreeEngine.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0af405b7-98b4-4751-b3fc-a7be06e22add/ReplicatedMergeTreeSequence.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6ec189e6-7096-428f-9a5a-149d7491d9fc/ReplicatedMergeTreeEngineZeroCopy.png Analyses - Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/analyses/2023/11/21/serverless-cockroachdb-asds-chapter-4-part-3 monthly 0.5 2023-11-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7ecbbc31-7544-4c7a-bebc-fa7b9d9e6b16/partitioning.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 3) - Make it stand out Fig 1. Sequential keys require a different partitioning scheme. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9c9d9a81-7694-4585-81b3-0f37559c18e7/split_point.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 3) - Make it stand out Fig 2. Does there exist a split point such that it results in a net gain in performance? https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/7ac83cdd-652b-449f-8caa-88fe15291c1d/joint_consensus.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 3) - Make it stand out Fig 3. CRDB implements Raft reconfiguration to move replicas. https://jack-vanlightly.com/analyses/2023/11/21/serverless-cockroachdb-asds-chapter-4-part-2 monthly 0.5 2023-11-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3f67aeb2-3f8f-4067-be01-de1f7b172271/ac_slot_token.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 1. Admission queues can be based on slots or tokens. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/696d53e9-d7d5-4e1e-b179-1ae357a05fc4/ac_read_write.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 2. Writes must pass through a token-based queue followed by a slot-based queue. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d7063db4-e006-4e05-a44a-892ef0f719c6/ac_heaps.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 3. An admission queue is implemented as a set of heaps. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2808e91a-0a9c-483e-9649-8e903db69f17/lsm_writes.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 4. A write is a mix of synchronous and asynchronous work, https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0594b32e-48cd-4562-989c-50f3e602eb4b/lsm_reads.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 5. The read of k=6 must check multiple SSTables before it finds the current value of B. The k-7 read gets lucky and finds the latest value in the in-memory memtable. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5f1a3bb3-f3a1-42ce-9c2d-212f714bbbe7/lsm_read_amp.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 6. Read amplification rises and falls as a function of foreground writes to background work. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/87cdcf40-8e74-432e-9357-88f1427f9e57/token_buckets.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 7. The token bucket algorithm. Bursts above the replenishment rate drain the bucket, and periods below the replenishment rate cause the bucket to fill. When the bucket becomes empty, throttling kicks in. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/564e9fb0-4661-4c65-85e8-5a6629566543/token_buckets_lsm.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 2) - Make it stand out Fig 8. A right-sized token bucket, with the right replenishment rate, can effectively rate limit writes to an LSM based storage engine to balance writes with read efficiency. https://jack-vanlightly.com/analyses/2023/11/21/serverless-cockroachdb-asds-chapter-4-part-1 monthly 0.5 2023-11-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0bed4169-cf0e-4881-bec2-23542db49532/nodes_and_range.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 1. A single row insert query arrives at a gateway node, which sends a corresponding KV write operation to the relevant range leaseholder who in turn replicates it to its followers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2171de9b-2248-48aa-ab35-4f195590f815/write_path.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 2. The path of a write to a given key range. The distribution layer routes the operation to the second node, which acts as the range leaseholder. The leaseholder directs the write to the normally co-located Raft leader, which in turn replicates the operation to its followers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/eabf99b9-ccbe-4a18-9dd5-410e3685c0bd/read_path.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 3. A read operation is distributed across nodes that host the relevant range leaseholders. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/64e19490-2a21-4416-aa15-8ca1d272a1f5/range_splits_merges.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 4. Ranges split and merge according to data size and load. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/32df191a-5efe-40d2-91af-5b69c131e42a/pebble.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 5. Each range replica writes table data and Raft data to a local Pebble instance. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d8725e1c-d84f-450a-a427-75e4a3d8efc8/crdb_object_storage.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 6. Choices for integrating (or not) cheap, durable cloud object storage. CRDB chose to store all data on storage nodes, avoiding the latency penalty. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8c3063a8-0a0b-4224-bb48-9446d61d6169/separated_compute_storage_layers.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Fig 7. Separating the CRDB node into compute and storage. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a355f386-0a5b-4234-b2d3-4b3cad252d9a/virtual_clusters.png Analyses - Serverless CockroachDB - ASDS Chapter 4 (part 1) - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://jack-vanlightly.com/analyses/2023/11/15/neon-serverless-postgresql-asds-chapter-3 monthly 0.5 2023-11-16 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ce0a7c95-90f7-4388-9495-8665b9535e93/neon_architecture.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 1. Neon separates the Postgres monolith into a compute and storage layer. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/610b01d6-2430-4c96-84c9-22b9d638a24f/neon_postgress.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 2. On the left, we see the traditional Postgres monolith, which reads and writes to local files. On the right, we see Neon’s disaggregated Postgres, where reads and writes go against the remote storage services. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/69be1fa9-9c47-48c1-9cb8-b7feaf25b2b8/pg_and_sk.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 3. Writes (WAL records) go to Safekeeper nodes and data page reads go to Pageservers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bfa76006-1f42-48e3-b852-48009ef67b9c/safekeepers.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 4. A Postgres database writes WAL records to three Safekeepers (using quorum writes). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ef799720-c7e8-486f-8c7a-e45961e6989b/multi_paxos_roles.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 5. Paxos roles spread over different components. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e21873f4-a7c7-40e7-b6e6-c3a2a1aa421a/reads.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 6. Multiple layers of caching sit between the query processor and cloud object storage. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/45c555f9-aaf1-4ba8-b35d-63b6a3cf49c2/read_replicas.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 7. Safekeepers stream WAL records to read replicas. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/18d3e52e-9872-4358-b22c-86e9a57e21a2/readahead.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 8. Readahead reads can reduce latency by prefetching data from Pageservers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b47b5c89-c451-4ba8-bf8b-4c2e45b06de8/object_storage.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 9. Neon chooses the fast storage in front of higher latency object storage approach. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c2ba2f8e-9f02-4b14-839f-cebd3027682b/vms_k8s_scaling.png Analyses - Neon - Serverless PostgreSQL - ASDS Chapter 3 - Make it stand out Fig 10. The components involved in auto-scaling Postgres databases. https://jack-vanlightly.com/analyses/2023/11/14/kora-serverless-kafka-asds-chapter-2 monthly 0.5 2023-11-14 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b03ee061-9d60-4833-9f8c-853bd173b04a/fast_storage_slow_object_storage.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 1. Choices for integrating (or not) cheap, durable cloud object storage. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/516c7396-5a04-4436-9210-b3f3fc49a52e/fault_tolerant_cache.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 2. Kora brokers act as a fault-tolerant cache in front of cloud object storage as the primary data store for streams. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/cafd7a4a-bc13-4c31-9e8a-757e1343f13e/high_level_architecture.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 3. The separation of components into disaggregated proxy, compute and storage layers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/348df104-f901-4900-8b8b-4327cb2b61a4/lkc_isolation.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 4. Request interception to annotate resources with LKC ids. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3eab1b22-7e26-4c61-86f4-57ff2a95eab4/kafka_api_small.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 5. The Kafka API includes partitions (which can be abstracted by client libraries). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/388dac1a-dbd7-4983-84cb-f399decde67a/cells.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 6. Cells are independent of each other. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/52620cb8-c440-4780-ab92-7b5fa2e68272/metadata.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 7. KRaft controllers and Kora brokers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e770836a-92a9-4790-944b-6bf87c36b30f/shared_quota_service.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 8. Dynamic quota system for distributed rate limiting. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fce2c679-3ff7-499a-acf1-f5f7a7218d07/tenants_and_cells.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 9. LKCs spread over multiple cells such that any given partition remains in a single cell. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/20d1fb90-b25f-4b71-9fb6-69cde4159894/partition_reassignment.png Analyses - Kora - Serverless Kafka - ASDS Chapter 2 - Make it stand out Fig 10. Online reconfiguration of replicas, such as for replacing a degraded or dead broker; or for moving replicas across brokers. https://jack-vanlightly.com/analyses/2023/11/14/amazon-dynamodb-asds-chapter-1 monthly 0.5 2023-11-14 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e8b242f5-e719-4604-b9d8-25765db2e3b4/table_partition.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 1. Each table partition is formed of a Multi-Paxos group. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/da65f204-ff81-4eb8-9bab-4e057fd11468/partition_distribution.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 2. The table partition replicas are distributed across a set of storage nodes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/73d8b70a-98e2-4958-85ba-0b4c63c56c43/high_level_architecture.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 3. DynamoDB’s disaggregated components. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0288cfe8-0c39-4887-a8fc-0f6b1ffcc761/storage_nodes.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 4. Two storage nodes and two different replicas. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8bf14df6-488e-4250-b185-223a8e5efa37/md_lookups.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 5. How DynamoDB’s metadata system is designed with predictable performance over pure efficiency. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e627ff58-b934-4e5f-ad17-33d57eb17555/token_bucket.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 6. The Token Bucket algorithm. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6ac9cb48-caeb-4d8a-af4f-aa97b3995cdf/token_bucket_throttling.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 7. Throttling occurs when the bucket empties. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1c4e17b0-79a5-4838-be9d-2d12bae82c58/rr_gac_buckets.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 8. How request routers and GAC nodes interact to apply distributed rate-limiting via token buckets. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/42a2ef11-6cd1-469f-a842-b43ff139647d/split_point.png Analyses - Amazon DynamoDB - ASDS Chapter 1 - Make it stand out Fig 9. Partition splits use key distribution to determine the split point. https://jack-vanlightly.com/analyses/2023/5/15/kafka-vs-redpanda-performance-part-6-draining-backlogs monthly 0.5 2023-05-15 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4ac3387f-6e1d-4f83-b5c3-2dd24aa8142f/rp-backlog-1GB-run1-labelled.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 1. The backlog is not draining after 4 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8ec52916-41ed-45d0-ac41-704d234289e9/backlog-rp-1GB-disk.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 2. When Redpanda reads from disk, the catch-up or drain rate is more rapid. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/9112311a-ed86-4801-838f-43de9942758b/rp-backlog-1GB-run2-labelled.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 3. Redpanda reaches an equilibrium where the backlog becomes stable but never drains. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/92e8b98a-6716-44e5-9697-00c5d2eda528/backlog-rp-1GB-run2-disk.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 4. The early read to disk phase subsides, and so too does the catch-up rate. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0bd64c60-1293-4722-ad0f-5200401afa9e/backlog-kafka-1GB.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 5. Kafka drains the backlog in 3 hours. We see consumer lag build up, then starts to drop almost as quick as it grew. Kafka also saw a slow down in catch-up towards the end here. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d32f7ed7-f6fa-4be6-b04d-8517dc6e009d/rp-backlog-1GB-fanout2-labelled.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 6. Redpanda didn’t get close to being able to drain the backlog with two consumer groups and the constant 1 GB/s producer load. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f55b6ed7-3e78-4d60-83a2-ce19debb9236/backlog-kafka-1GB-fanout2.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 7. Kafka drains the backlog in 130 minutes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/88e94ef9-b914-48c5-b577-f81b7b857170/rp-backlog-800MB-labelled.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 8. At 800 MB/s, Redpanda initially does well, but then stalls and the backlog starts to grow again. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ddd30fd2-63f6-47b9-b7e8-d838fd66aecb/backlog-kafka-800MB.png Analyses - Kafka vs Redpanda Performance - Part 6 - Draining backlogs - Make it stand out Fig 9. With constant 800 MB/s producer load, Kafka drains the backlog in 74 minutes. https://jack-vanlightly.com/analyses/2023/5/15/kafka-vs-redpanda-performance-part-5-reaching-the-limits-of-the-nvme-drive monthly 0.5 2023-05-15 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ae39384d-94e4-417f-9136-b3fa32d5c55f/1-2-GBs-throughput.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 1. With acks=1, Kafka reaches 1900 MB/s but Redpanda tops out at 1400 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2ff758e9-39da-4c9b-a8fc-27274db27671/kafka-drive-writes-1000-1900MB.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 2. Kafka reaches the absolute NVMe drive limit at the 1900 MB/s test and continues to max it out when trying to reach 2000 MB/s throughput. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1407b258-13d3-4c3f-b51e-90be98e6eb40/redpanda-drive-throughput-1000-2000MB.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 3. Redpanda could not reach the NVMe drive limit. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fa58c89b-ebf4-456a-ba01-152bf88c660c/acks1_1000-1900MB_AllWorkloadsEndtoEndLatencyPercentilesUpTo999.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 4. Kafka end-to-end latencies, up to p99.9, at different throughputs, from 1000 MB/s to 1900 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6f20ae6d-f19b-4f28-be2f-bd53ec1d9c9f/acks1_1000-1900MB_AllWorkloadsEndtoEndLatencyPercentilesall.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 5. Kafka end-to-end latencies, up to p100, at different throughputs, from 1000 MB/s to 1900 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1d83baf0-e74f-4e78-84c7-4b616211a139/kafka-vs-redpanda-e2elat-1000-1200MB.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 5. Redpanda achieved lower end-to-end latency than Kafka on the lower throughput tests of 1000 and 1200 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ea853d1f-19f8-4b6f-820f-f0c637034a92/redpanda-ret-limit-1000-1200MB.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 6. The usual Redpanda step increase in end-to-end latency once the brokers start actively deleting segment files. Each test was two hours with a one hour retention limit. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/681d96e8-cc4d-4eaf-abfc-dad6b7090ae0/redpanda-none-vs-ret-limit.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 7. End-to-end latencies are markedly higher when we measure latency once data retention limits are enforced. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1922b2ad-2915-49be-b0b9-05e046b4f562/acks1-AK_RP-RetLimt_AllWorkloadsEndtoEndLatencyPercentilesall.png Analyses - Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive - Make it stand out Fig 8. Once data retention is included, the Redpanda advantage was much smaller. https://jack-vanlightly.com/analyses/2023/5/15/kafka-vs-redpanda-performance-part-4-impact-of-record-keys monthly 0.5 2023-05-23 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b7b533e1-0fd0-4363-92a3-6334c16a66f7/record-keys-1000MB%3As-throughput.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 1. Neither Kafka nor Redpanda reach 1 GB/s but Kafka reaches 200 MB/s more. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3c413d30-0487-4557-a180-aca0a712eaf2/record-keys-500MBs-throughput.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 2. Kafka manages the target 500 MB/s with 100 producers. Redpanda tops out at 330 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/08e9e84a-9ec5-4939-ae36-a1597ff48795/record-keys-100-partitions-throughput.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 3. Reducing the partition count to 100 helped Redpanda a lot and it was able to reach 500 MB/s with 100 producers. Kafka also benefited, reaching 500 MB/s with just 40 producers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b797985d-56ff-4e17-a6f7-9695695e727d/TLS_e2elat_200MB_400_800parts.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 4. Redpanda continues to fare poorly compared to Kafka even on 10 partition topics, using record keys. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/51a1a73d-820b-4acd-ace1-7212c9a25826/TLS-200MB-400-800-p99-e2e-lat.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 5. The Redpanda 400 partition test with two latency spikes. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/87f0d708-4101-4263-838c-08934d3190dd/TLS_e2elat_50MB_400-800parts.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 6. The trend continues. Kafka saw a latency spike in its 40 topic (400 partition) test which drive up tail latency, bringing it just under Redpanda. The 80 topic test saw much better results for Kafka. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b3d63536-2c07-43ef-9046-8588fa3cec49/TLS-50MB-400-800-parts-AllWorkloadsEndtoendLatencyP9999.png Analyses - Kafka vs Redpanda Performance - Part 4 - Impact of record keys - Make it stand out Fig 7. p99.99 end-to-end latency over time. Kafka showed the lowest, most stable p99.99 latencies, with a single latency spike in the Kafka 400 partition test. https://jack-vanlightly.com/analyses/2023/5/15/kafka-vs-redpanda-performance-part-3-hitting-the-retention-limit monthly 0.5 2023-05-15 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b401c1ad-3fed-4b4d-8da2-a95c51cad934/rp-ret-limit-p50.png Analyses - Kafka vs Redpanda Performance - Part 3 - Hitting the retention limit - Make it stand out Fig 1. The Redpanda stepwise increase is less pronounced in p50-p90, but visible. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ca34843e-007f-4dc7-8e4f-c3588d8efa37/rp-ret-limit-p95.png Analyses - Kafka vs Redpanda Performance - Part 3 - Hitting the retention limit - Make it stand out Fig 2. The stepwise increase becomes significant from p95. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/eb473963-844e-4066-a6e4-dee659467f9c/rp-ret-limit-p9999.png Analyses - Kafka vs Redpanda Performance - Part 3 - Hitting the retention limit - Make it stand out Fig 3. The stepwise increase remains proportionally around the same at the high tail latencies. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/08edff81-e4d7-4e64-ae3d-9f229f2c9e95/rp-ret-limit-20-50.png Analyses - Kafka vs Redpanda Performance - Part 3 - Hitting the retention limit - Make it stand out Fig 4. On the left we see the end of the 20% test. The middle is the 50% test, but the retention size has not been reached yet. The right-hand size is where the retention limit kicks in for the 50% retention size limit. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/89f1b09e-89ee-497d-9af7-05d7281b3874/rp-ret-limit-retest.png Analyses - Kafka vs Redpanda Performance - Part 3 - Hitting the retention limit - Make it stand out Fig 5. The impact of data retention on end-to-end latencies for the 1 GB/s Redpanda benchmark (without TLS in this case). https://jack-vanlightly.com/analyses/2023/5/15/kafka-vs-redpanda-performance-part-2-long-running-tests monthly 0.5 2023-05-15 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/486bcc2c-c51d-4e70-ad19-191744f1023c/rp-long-running-p50.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 1. Redpanda p50-p90 end-to-end latencies jump after 12 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/506ee70c-1288-41a7-9cde-b9bb91174bf4/rp-long-running-p95.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 2. The Redpanda tail latencies shoot up massively after 12 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c3ab1047-3305-4212-b893-24a2218dacb6/rp-long-running-rpc-latency.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 3. Redpanda reports increasing inter-broker latencies midway through the 24 hour test. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c0b1e095-4f70-4017-8775-4b0171ba533b/rp-long-running-io-queue-metrics.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 4. Indicators of saturation. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f5ea31a9-6d51-4672-808d-a73c762a4694/rp-long-running-starving-disk.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 5. Redpanda-1 starts reporting being “starved for disk”. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5691e760-8539-43f1-8132-f92ce6e55ffa/rp-long-running-rp1-disk-latency.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 6. Redpanda-1 NVMe drive read and write latency increases markedly. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b5c4e458-7674-44a7-b240-72e058cd194c/rp-long-running-rp0-disk-latency.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 7. Redpanda-0 also seeing increased NVMe drive latency. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b6ec4860-c330-4e8b-b2fb-be8035a4ddad/rp-long-running-rp2-disk-latency.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 8. Redpanda-2 shows healthy NVMe drive latency. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/60635bd6-d2b3-4742-8b11-a558b76fb64f/rp-long-running-rp0-disk-latency-36hr.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 9. Redpanda-0 NVMe drive latency over 36 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/33b8b29e-ef9a-4472-9862-fc41a393d622/rp-long-running-rp1-disk-latency-36hr.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 10. Redpanda-1 NVMe drive latency over 36 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f2007109-3359-48a1-bf92-08f2f77839ba/rp-long-running-rp2-disk-latency-36hr.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 11. Redpanda-2 NVMe drive latency over 36 hours. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/fd6ea9c0-6db2-43e0-8ace-0615d7dc3b6f/kafka-long-running-p50.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 12. Kafka p50 end-to-end latencies for a 24 hour period. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/43382444-7611-4cf8-bbb1-478c51b5d2ea/kafka-long-running-p99.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 13. Kafka p99 end-to-end latencies for a 24 hour period. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f738869f-4b10-43d4-be25-1a3f4c72d993/kafka-long-running-p9999.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out Fig 14. Kafka p99.99 end-to-end latencies for a 24 hour period. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1de04861-ab95-46da-8113-eee55aa4397d/ec2-op-docs.png Analyses - Kafka vs Redpanda Performance - Part 2 - Long running tests - Make it stand out https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html https://jack-vanlightly.com/analyses/2023/5/15/kafka-vs-redpanda-performance-part-1-4-vs-50-producers monthly 0.5 2023-05-15 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/80989c5d-d322-45c5-9697-2792e409ac71/500MB_4vs50prod_labelled.png Analyses - Kafka vs Redpanda Performance - Part 1 - 4 vs 50 producers - Make it stand out Fig 1. Redpanda end-to-end latency with 50 producers reaches 700 ms, far higher than its results with 4 producers and higher than Kafka. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/8c8bd255-2526-4f98-b787-54e6aa9f3084/TLS_500MB_4vs50prod_labelled.png Analyses - Kafka vs Redpanda Performance - Part 1 - 4 vs 50 producers - Make it stand out Fig 2. With TLS, Redpanda performance with 50 producers degraded significantly, reaching 24 second end-to-end latency. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ed20a060-1051-4a92-878c-6d8c4d7b0ddf/1000MB_4vs50prod_labelled.png Analyses - Kafka vs Redpanda Performance - Part 1 - 4 vs 50 producers - Make it stand out Fig 3. Without TLS, Redpanda could reach 1 GB/s with 4 and 50 producers, but its end-to-end latency with 50 producers also reached 24 seconds. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3e64a174-b02f-4242-b5c7-c74d3b505d93/TLS_1000MB_AllWorkloadsProducethroughputMBs.png Analyses - Kafka vs Redpanda Performance - Part 1 - 4 vs 50 producers - Make it stand out Fig 4. With TLS, Redpanda could only manage 850 MB/s with 50 producers, where as Kafka comfortably managed the target 1000 MB/s. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/189a58c9-5fee-4c89-9c88-8a6b9016a346/50prod-rp-metrics1.png Analyses - Kafka vs Redpanda Performance - Part 1 - 4 vs 50 producers - Make it stand out Fig 5. Left: 4 producers (500-1000 MB/s steps). Right: 50 producers (500-1000 MB/s steps). Without TLS. With 3x 24 vCPU instances, the maximum CPU time would be 72K ms. https://jack-vanlightly.com/analyses/2023/01/22/paper-vr-revisited-checkpoint-based-replica-recovery-part-6 monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c88ec2ec-f3e1-4db7-a8d7-6f40687e8161/CheckpointRecoveryActions.png Analyses - Paper: VR Revisited - Checkpoint-Based Replica Recovery (part 6) - Make it stand out Fig 1. Checkpoint recovery actions https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b4520c37-8b85-4d4b-b4f1-767af4dbac93/GarbageCollectedPrefix.png Analyses - Paper: VR Revisited - Checkpoint-Based Replica Recovery (part 6) - Make it stand out Fig 2. The log of replica 1 is entirely within the garbage collected log prefix of replica 2. Replica 1 cannot obtain operation 3 and execute it. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/640be2cf-3eff-4d78-9335-d7e6cb6ab20e/CheckpointTransfer.png Analyses - Paper: VR Revisited - Checkpoint-Based Replica Recovery (part 6) - Make it stand out Fig 3. The general approach to synchronizing using checkpoints, instead of replicating log entries alone. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/554811b9-8455-4f7b-a954-4c608acdac7a/OtherMsgsWithCheckpoints.png Analyses - Paper: VR Revisited - Checkpoint-Based Replica Recovery (part 6) - Make it stand out Fig 4. Other messages may also now contain a checkpoint. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a9393797-4aba-4b25-8032-5c0ae08baa99/CorrectedActions.png Analyses - Paper: VR Revisited - Checkpoint-Based Replica Recovery (part 6) - Make it stand out Fig 4. Recovery actions modified to handle checkpoint synchronization when a required portion of a log is missing due to log garbage collection. https://jack-vanlightly.com/analyses/2023/01/17/paper-vr-revisited-log-based-replica-recovery-part-5 monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6cba2a5a-cd65-4f21-844c-781d78c059b4/BasicRecoveryActions.png Analyses - Paper: VR Revisited - Log-Based Replica Recovery (part 5) - Make it stand out Fig 1. Four new actions for basic recovery. Described in more detail below. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b6089efb-c2f0-425c-9875-8ed768de2043/ReplicaDivergence.png Analyses - Paper: VR Revisited - Log-Based Replica Recovery (part 5) - Make it stand out Fig 2. Examples of how the logs of a recovering replica and the primary can be out-of-sync - with compound view-number/op-number log entry ids. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0450c610-18ff-42c7-8cbd-e1aacacbd078/Divergent3.png Analyses - Paper: VR Revisited - Log-Based Replica Recovery (part 5) - Make it stand out Fig 3. A history that leads to case F. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/26484e6e-371b-4f38-badd-b88e1a1b7d60/LogSuffixRecovery.png Analyses - Paper: VR Revisited - Log-Based Replica Recovery (part 5) - Make it stand out Fig 4. Log-suffix recovery actions. https://jack-vanlightly.com/analyses/2023/01/01/paper-vr-revisited-application-state-and-commit-number-monotonicity-part-4 monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/34423a5e-f0be-4245-8ce3-89dd4cc69a9e/SMR.png Analyses - Paper: VR Revisited - Application state and commit-number monotonicity (part 4) - Make it stand out Fig 1. State-machine replication (SMR) where each node is comprised two modules: log replication and state machine. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/78a7c561-c22f-4445-85da-b88a89fad6d0/AppState.png Analyses - Paper: VR Revisited - Application state and commit-number monotonicity (part 4) - Make it stand out Fig 2. Application state modelled as a log https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/b3857fce-ce56-4375-831f-3bdb73d6a2d4/LowerCommitNumber.png Analyses - Paper: VR Revisited - Application state and commit-number monotonicity (part 4) - Make it stand out Fig 3. A replica receives a StartView with a lower commit number than its own. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3dcbd5ce-2c0e-47c0-a910-469e9ae6aa4a/AppStateDivergence.png Analyses - Paper: VR Revisited - Application state and commit-number monotonicity (part 4) - Make it stand out Fig 4. r1 ends up applying operation (a) two times to the state machine module because the commit-number was not monotonic. https://jack-vanlightly.com/analyses/2022/12/28/paper-vr-revisited-state-transfer-part-3 monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/223e68f7-a45c-44d0-b6d0-1de4b8c58eae/StateTransferActions.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 1. State transfer actions for the eager view update approach. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/629dd5e9-3160-4975-992a-0d648177edf2/Part3Counterexample1.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 2. Eager view update counterexample 1 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3f8d641d-57ee-4aaa-bc5b-b4203be69b09/Part3Counterexample2.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 3. Eager view update counterexample 2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/583fbd99-e770-4b31-9185-085eeb9db7e5/StateTransferActionsV2.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 4. Amend the SendGetState action to not truncate the log, and the ReceiveNewState action to append/overwrite instead of just append. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1e49e198-3b6f-4a52-950d-e772435c2377/Part3Counterexample3.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 5. Counterexample 3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/eb5f06fe-af96-4d9e-96ed-fd179f3c2350/LazyCounterexample1.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 6. Lazy view update counterexample 1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/db34438a-00c8-45f0-b186-94bf2d04ddb4/LazyCounterexample2.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 7. Lazy view update counterexample 2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2ed91219-6e17-4370-90a0-2820233cb877/LazyCounterexample3.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 8. Lazy view update counterexample 3. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/0b02b8cb-4b5c-45bd-bdeb-9dbcc4171c39/FinalStateTransferActions.png Analyses - Paper: VR Revisited - State Transfer (part 3) - Make it stand out Fig 9. Final state transfer actions. https://jack-vanlightly.com/analyses/2022/12/12/22/paper-vr-revisited-view-change-answers-part2 monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/900d36a9-ace8-46e5-92a6-46d714f8b515/variables.png Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 1. The variables. The variables are either per replica, or global https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/778e6990-48bb-420f-a281-9de515b96158/invariants.png Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 2. Two safety properties (invariants). The /\ symbol means AND. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/e8d84374-f42c-4d6a-9f13-74643d45dc50/liveness.png Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 3. Liveness properties that specify good things that should eventually happen. The \/ means OR. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/20858484-9741-4713-9bdb-9b7ba35500e2/NormalOperationActions.PNG Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 3. The normal operations are split into 4 actions (Commit not included yet). https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/ce22d652-0590-4d77-92c3-b416022ebaf5/TwoCounters.png Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 4. Two counters have 4 states where both counters <= 1. There are two ways from reaching the state where both counters = 1. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/3af86dab-9af9-4876-852c-a68e54f60549/AssumeModeViewChangeActions.PNG Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 4. The view change actions of assume-mode. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/a0b9df55-3c97-47bc-86f7-5925990092f5/IncrementModeViewChangeActions.PNG Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 5. The view change actions of increment-mode. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/42d50505-5b2a-47a6-aede-e41f5c7f6c79/increment-mode-counterexample.png Analyses - Paper: VR Revisited - View changes - Answers (part 2) - Make it stand out Fig 6. Counterexample leading to data loss. https://jack-vanlightly.com/analyses/2022/12/20/paper-vr-revisited-view-change-questions-part1 monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/f63fef1b-09f9-49b7-833f-b826b7920caf/SVC_flood.png Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 1. StartViewMessage (SVC) flood after r1 triggers a view change. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/df15a93f-4860-4eec-9c1e-c7e655c5dfc3/SVC_DVC_SV.png Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 2. A complete view change with 5 replicas. All functioning replicas on receiving two StartViewChange messages with matching view numbers send a DoViewChange message to r4, the primary of view 4. Once r4 receives three DoViewChange messages it broadcasts a StartView message. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/6bd59500-74fc-4266-b2e9-5f3f6650cefa/IncrementModeOneAhead.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 3. Increment-only approach with Left and Right liveness issue examples. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/17ae34ae-76d5-4dce-91be-86f7f75ad247/IncrementModeOneBehind.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 4. Two view changes with a stale view number. The left example can complete despite r3 being ignored, the right example shows a view change getting stuck. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/4eadb21a-86b6-4e3f-86d2-4727c53c96bc/AssumeModeOneAhead.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 5. Assume-mode where timer expiry increments the view number but receiving an SVC or DVC message with a higher view causes the replica to assume the view of the message. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/44257f5c-924c-4331-93ae-5733d1a23e0b/AssumeModeOneBehind.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 6. A stale replica is able to assume the new view immediately and therefore avoids the liveness issues associated with stale view numbers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/61eccd3a-de52-407e-ac95-230979c532c3/ResendSVCs.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 7. Resend SVC message strategy. r3 sends an SVC but receives no response so it waits for the timer to expire and resends the same view change for view 2 again. Repeats until it receives a response. But does it receive no response due to some network issue or because its view number is too low and it is being ignored? https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/866bfc91-a634-47d6-85a1-3b261d14ff6a/IncrementModeStaleReplicaAndResendSVC.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 8. One stale replica holds back a view change because it is forced to increment its way through the view numbers until it catches up. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/bbffedd2-8cf3-404e-bb5a-52bb731d1c1a/ResendSVCs.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 8. Start new view change strategy. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/c0717978-a419-4172-a22d-bb9a517c6f4b/IncrementModeDisruptiveReplica.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 9. Combining increment-mode with the start new view change strategy for stuck view changes can create a disruptive non-functioning replica. This replica keeps starting new elections but cannot join the views as it ignores all messages from its peers with lower view numbers. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/d9eb6101-de74-4aee-9648-c3efa2f1d24d/NoSVResponse.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 10. r1 never receives the SV from r2 and ignores subsequent Prepare messages, in the end it starts a new view change after not hearing from r2. https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/677587c5-7741-4722-a886-3c7f674626f4/IncrementModeDVCsWithHigherViewNumber.PNG Analyses - Paper: VR Revisited - View changes - Questions (part 1) - Make it stand out Fig 11. An increment-mode history where a replica accepts f+1 DVC messages from a different view to its own. https://jack-vanlightly.com/analyses/2022/12/20/vr-revisited-an-analysis-with-tlaplus monthly 0.5 2023-01-22 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/2d9755fc-7268-4e81-838b-5547cf460635/SMR.png Analyses - Paper: VR Revisited - An analysis with TLA+ - Make it stand out Fig 1. State-machine replication https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/5e7a0495-8f7c-4c2d-81c6-6ac39ee50931/Overlapping-quorums.png Analyses - Paper: VR Revisited - An analysis with TLA+ - Make it stand out Fig 2. No matter the cluster size, all majority quorums overlap. In other words, there does not exist two majority quorums that do not overlap. https://jack-vanlightly.com/analyses/category/Table+formats monthly 0.5 https://jack-vanlightly.com/analyses/category/Distributed+systems monthly 0.5 https://jack-vanlightly.com/analyses/category/Performance monthly 0.5 https://jack-vanlightly.com/analyses/category/Serverless+data+systems monthly 0.5 https://jack-vanlightly.com/analyses/tag/Viewstamped+Replication monthly 0.5 https://jack-vanlightly.com/analyses/tag/Apache+Kafka monthly 0.5 https://jack-vanlightly.com/analyses/tag/Redpanda monthly 0.5 https://jack-vanlightly.com/home daily 0.75 2024-10-31 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1477845324246-NA9AVLGB4FKLI5M5RI1V/shutterstock_458475352.jpg About Me https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1548698483309-NHFMDG5QS4HXJS1NIQ2G/Observatory.PNG About Me https://jack-vanlightly.com/paper-planes daily 0.75 2017-08-06 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1502035747277-KA72CN7XSVREJ98XVATV/shutterstock_458475352.jpg Paper Planes https://jack-vanlightly.com/blog-archive daily 0.75 2025-12-19 https://images.squarespace-cdn.com/content/v1/56894e581c1210fead06f878/1506275677541-MWJUZ1598LLT212VI1PK/shutterstock_458475352.jpg Blog Archive https://jack-vanlightly.com/analysis-archive daily 0.75 2023-08-16