Then there's the Jepsen tests for humbling you into the right frame of mind for building these systems. Learn how to break things so you can build better things.
Redpanda bring out benchmark after benchmark claiming performance superiority over Apache Kafka. I decided to run my own tests to see if any of it was true.
Queue semantics are coming to Apache Kafka (KIP-932) and in fact there are many advantages to building queues on top of logs rather than opting for a more queue-native design.
I'm digging into Apache Iceberg internals for the final table format consistency model blog post.
Part of my process of understanding a project from its code is making a (throwaway) map of the important classes and functions. This is especially important in the early hours of
BYOC is something I’ve been thinking about recently so I decided to write down the thoughts I have on it and where I think cloud services are going in general.
As promised, I have written a complete Kafka replication protocol
description (with KIP-966 changes applied) which is inspired by the precise
but accessible style and language of the Raft paper.
Chapter 6 of The Architecture of Serverless Data Systems is out. This chapter focuses on commonalities in how these systems scale according to tenant load. Despite the varied workloads, patterns emerge that we can learn from.
The final part of my Apache Iceberg consistency model series is out, covering the formal verification work.
It's also the end of the table format consistency model series, but not the end of my writing about the table formats. More still to come...
The first post in my “Understanding Apache Iceberg’s Consistency Model” is out. This post covers the internals of the read and write path, details on metadata manipulation, concurrency control, and so on. If you ever wanted a more in-depth post about Iceberg internals, this is
Understanding Apache Iceberg's Consistency Model Part 2 - details of concurrency control and data conflict checks to allow Apache Iceberg to handle multiple concurrent writers correctly.
I have open sourced my TLA+ specification for Apache BookKeeper here:
It detected a data loss scenario that could occur during the ledger recovery process. Checkout the details here:
I've written a primer on formal verification and TLA+ so I can refer to that whenever I write posts about specific aspects of TLA+. I have a two-parter on liveness that is ready to go after this.
How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out.
I'm working on a set of blog posts that compare the internals of Apache Iceberg, Delta Lake, Apache Hudi and Apache Paimon. No benchmarking, no judgments etc, just a comparison of internal mechanics.
Liveness properties are often overlooked in TLA+ but they are actually surprisingly important. This is part 1 of a two part post on liveness properties by example.
Chapter 5 of the Architecture of Serverless Data Systems: Serverless ClickHouse Cloud.
Part 1 looks at OSS ClickHouse.
Part 2 looks at the serverless architecture of ClickHouse Cloud.
Kora: A Cloud-Native Event Streaming Platform For Kafka paper won Best Industry Paper at the VLDB conference.
There's so much work going on behind the scenes on the Kora engine that I think we'll need another paper this time next year 😀
Object storage direct-write Kafka topics are coming to Confluent.
In data system design, cost is a kind of unstoppable force that rewrites architectures every decade or so, and that is happening right now for high-volume streams.
We've published a new KIP that improves the durability of Apache Kafka with its asynchronous storage engine configured. See the high level description in my blog post.
I'd never really looked into OLAP database internals before, but my research into ClickHouse for chapter 5 of the Architecture of Serverless Data Systems is fascinating. I also get the feeling that the most insights will come when I look at my second OLAP system (probably Pinot)
I have a nice Apache Kafka vs Apache Pulsar post in the works regarding time-travel and multi-topic subscriptions. In case you're wondering, Pulsar got it right and Kafka leaves a lot to be desired.
My previous work looked at each table format in isolation. Now it's time to compare them. How do Apache Iceberg, Delta Lake, Apache Hudi and Apache Paimon store the canonical set of files that make up a table? What are the similarities and differences?
The 2nd post in my table format comparison series is out. Append-only tables and incremental reads are one of the pillars of streaming in the table-format space. This post looks at how each of the four table formats supports this workload.
Finally completed part 1 of putting
@apache_pulsar
's durability claims to the test. Spoiler alert, I couldn't make it lose data, or deliver any messages out-of-order. Still, I think it makes for interesting reading for those interested in how it works.
With the announcement of S3-native-streams (Freight clusters), here is a commentary on Confluent strategy regarding object storage, streaming and an open data architecture.
The thing is, systems have traditionally been designed around paying for capacity of network and disks, not the number of IO requests. That fundamentally changes with S3, and right now, I don't think that cost model is compatible with S3 being a universal storage layer.
The RabbitMQ that people know is changing:
- bit by bit we're replacing the old consensus protocols with Raft
- we have new queue types
- better upgrade features
- better observability
- Kubernetes support and more in the works.
Model checking KIP-848 (Kafka's next consumer group balancing protocol), 18 hours in on 32 CPU threads, I've used 800GB of my 2TB NVMe. I'd say its 50-50 whether my workstation can handle this one, might have to go distributed mode.
Using I can clearly explain how Apache Pulsar supports a replication factor of 2 safely, providing continued read/write availability with the loss of one bookie. Systems like Apache Kafka simply cannot do this, minimum for safety and availability is 3.
Queues for Kafka (KIP-932), my biggest ask for Apache Kafka since joining Confluent. Really excited about this one. It's not just queue semantics, but queues which can do replay.
With conditional writes, S3 takes one more step towards the universal storage layer. But can it go the next level? Is it possible to make a (profitable) storage service that hits all three needs of latency, cost, and durability?
GCP is continuing to innovate its cloud storage, with hierarchical namespace buckets which have a directory structure like an actual filesystem. Can rename, list directories etc. Very cool.
I've created a model checking optimized TLA+ spec of Raft plus a couple of variants such as exploring when to fsync and also Flexible Raft (flexible quorums of Flexible Paxos).
When I first joined Confluent in Feb 2022, looking for a place to make an impact, it was this proposal that excited me most. I had come from Splunk, running Apache Pulsar as a service internally. We wanted to scale up Pulsar to take on Splunk's big ingestion workloads but ran the
In Feb 2022,
@ghaz
and I wrote an internal proposal at Confluent arguing for a "cost-saving design (e.g., writing to S3 directly) that can eliminate cross-AZ traffic costs for high-rate elephant workloads". I was hoping to apply my research on Corfu and
I think I'm done with chapter 5 - serverless ClickHouse. If there are any engineers with deep understanding of CH internals, I'm open to reviews before I publish. Half is dedicated to open-source CH, so you don't need to work for CH Cloud to be a helpful reviewer :)
I'm writing a complete Kafka replication protocol description with a style which lies between the formal style of a paper and the informal style of a blog post.
I'm curious if people think it would be best deployed as a single post or spread over multiple posts?
The Architecture of Serverless Data Systems will resume in January with chapter 5. Right now I have to dedicate myself to formally verifying KIP-848, the new Kafka consumer rebalancing protocol before the holidays. Also need to move all my Kafka TLA+ into the Apache Kafka repo.
One thing I'm seeing from my series on serverless data systems, is that an extremely diverse set of systems, APIs and workloads can make use of S3 as the primary storage layer. S3 is not a threat to these systems but an opportunity, if it can be integrated well.
@criccomini
There are the APIs (Kafka API, Pulsar API, Flink API) and then there are the distributed systems behind those APIs. S3 can become the default storage layer for everything, without deprecating the APIs (although I think some APIs will need to evolve).
A new table-format comparison post is out. How do the table formats support the streaming ingestion of row-level operations (insert/update/delete), such as ingesting CDC streams?
Chapter 1 - Amazon DynamoDB. I made this chapter 1 because it really epitomizes the large-scale multi-tenant data architecture I've been talking about recently and it has been evolving for 11 years already!
I think the new S3 Express One Zone looks interesting for an S3 WAL (given the pricing model). Typically the WAL needs faster but smaller disks, then the actual data disks need to be bigger and cheaper. This maps well to Express for WAL and standard for long term data.
Such a familiar story, I was always an average student but did enough to get by, it was only after I read a book on SQL Server locking internals that I found something that got me hooked - a healthy obsession as Phil calls it. For me that has been how distributed/data systems
I wrote an essay on my mistakes trying to convince people to do something, on doing what you want to do, and on obsession.
Ended with a personal note on developing healthy discipline, and having fun. :)
Seriously, I was ready to throw my PC out the window two days ago due to TLA+. Today is my moment of triumph as my spec detected a defect in my
protocol. I've never been so ecstatic to find a defect in my own design
that escaped all randomized testing of my implementation.
I love these reading lists. Of special interest to me this time round is the Model Checking Guided Testing for Distributed Systems paper which maps TLA+ specifications to implementations and generates tests from the state space. /1
Here's how Pulsar avoids split-brain when ZK loses visibility of the owner broker of a topic and a 2nd broker takes on leadership (meaning for a short moment we have 2 brokers thinking they are the owner of the topic).
An interesting paper on building a virtual disk on top of S3-like storage. The main downer is that it's still only prefix consistency (given high latency of S3 and the need to do batching to avoid small objects and keep the request rate in check for economics).
Next week I start a new role at Splunk working on Apache BookKeeper primarily. The Apache Pulsar/BookKeeper stack is amazing and I am really excited to be contributing to it full time from next week.
The semantics section of this Snowflake paper nicely describes some of the core concepts of incrementally generating/consuming change streams. It's helped me write more cohesively about CDC support in the table formats (which is WIP).
My next table format comparison post is on native support for CDC. So far, I've been digging around the code of Delta and Hudi. As always, it's fascinating to see their similarities and differences.
The VSR state transfer data loss invariant violation detected by TLC. I posited one that required 4 view changes when I read the paper, but TLC surprised me with this one with 3 view changes.
The BookKeeper replication protocol is so nuanced I'm still having aha moments. BK devs need to tread very carefully. I should write a blog post about all the surprising and non-intuitive things about it.
With the support of
@streamlio
I have done a write-up on how
@apache_pulsar
works and how it uses
@asfbookkeeper
to achieve its durability and latency guarantees. Quite a remarkable system I have to say.
I've finished my first real TLA+ spec and I have to say that it feels like a real achievement. It took 18 days of evenings and weekends with lots of mind bending and mental sweat but it was all worth it.
Something I love about TLA+ is the liveness checking. I can assert that state A leads to state B (A ~> B). Then the model checker will find histories where the protocol is unable to reach state B from state A.
🎉 We’re excited to share our intent to acquire
@Immerokcom
!
Together, we’ll build a cloud-native service for
@apacheflink
that delivers the same simplicity, security, & scalability that you expect from Confluent for Kafka.
Learn more →
I'm putting the final touches on my deep dive into Apache Paimon blog post, and a formal specification written using Fizzbee. This one was both a lot of hard work but also a lot of fun. Posting tomorrow hopefully 🤞
Just got back from
#tlaconf
at
#StrangeLoop
where
@lemmster
and I had our "Obtaining statistical properties by simulating specs with TLC" talk. Slides are available () but I'll probably get around to writing a blog series about the technique.
I'm working on a new replication protocol. If Flexible Paxos and Cheap Paxos had a kid, and also Raft and Apache BookKeeper had a kid and then those kids grew up and they had a baby, that baby would be my protocol.
I'm working on a series that analyzes real-world serverless data system architectures, with a particular focus on how different systems do multi-tenancy with good tenant isolation.
At Vueling Airlines I built an architecture mapping tool that maintained the whole graph of services, databases, queues, object storage, etc and their relationships in Neo4j, all built programmatically from config files and other sources. It was the only way to understand it all.
When I go to a new cloud environment, people apologize because the architecture diagram is out of date.
Spoiler: Everyone's architecture diagram is out of date.
That is the nature of the universe. Smile, nod, and accept it.
Have a listen:
I'm digging into Apache Paimon now. Unfortunately, there is no spec/protocol doc, so diving into the code. I'm still in the "groping in the dark" phase of code reading. Initial impressions are that if Apache Iceberg, Apache Hudi and Delta had a baby, it might be Paimon.