Michael Drogalis
@MichaelDrogalis
Followers
2,832
Following
516
Media
308
Statuses
5,966
🔥 Founder @ShadowTrafficIO 📰 Writes about bootstrapping
Seattle, Washington
Joined May 2011
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
OAT MAIYARAP X FREENBECKY
• 544930 Tweets
#アイマス20周年イヤー
• 218709 Tweets
#iHERBLivexENGLOT
• 147321 Tweets
No.6
• 116941 Tweets
SNCF
• 87177 Tweets
#それスノ
• 67963 Tweets
フジロック
• 48995 Tweets
ニコニコ復活
• 47988 Tweets
The Killers
• 38884 Tweets
モノノ怪
• 28656 Tweets
#ビリオンスクール
• 23024 Tweets
オリンピックの開会式
• 22147 Tweets
Zico
• 20268 Tweets
#ビリスク
• 19141 Tweets
ピクサー
• 14060 Tweets
トイストーリー
• 11217 Tweets
Pinned Tweet
🔥 It's finally here! I'm excited to announce that
@ShadowTrafficIO
is now available. Head to the home page to get started for free.
For my entire career, I've been baffled by how long it takes to build demos, load tests, and proof-of-concept projects. Everyone's built little
6
38
89
Put a UUID on it. Put a timestamp on it. Put a version on it.
You won’t regret it.
30
243
2K
Put a UUID on it. Put a timestamp on it. Put a version on it.
You won’t regret it.
54
163
2K
I've been working with
@apachekafka
for 7+ years. I find myself doing the same set of activities every time. Here's a set of Kafka productivity hacks for doing a few things way faster than you're probably doing them now.
@confluentinc
🔥
1
136
381
Things I wish I knew as a 22 year-old engineer:
1. Software engineering is mainly about people, not code. Technical expertise will only take you so far.
2. There's a time to hack together a house of cards and a time to build a masterpiece. Learn the difference.
3. Prioritize
9
80
373
My respect for a person goes up by a 1000-fold when I hear things like:
- I was wrong, you're right.
- I'm sorry.
- I don't know the answer.
7
137
275
🔥 I can't believe it: 8 months in, and I've just closed my 3rd enterprise customer.
Will I make it long term? Who knows.
But so far, my revenue per employee is higher than many seed-funded startups, and all of my existing customers have expanded 3-4x their initial use cases.
14
6
202
Thinking about writing something on what’s changed since
@martinkl
’s turning the database inside out talk (spoiler: a lot. The practical side is way different now.) Anyone interested?
14
6
124
Since leaving Confluent, I've been asked by so many people how their company can win at stream processing.
Honestly, I don't know, but one thing's been apparent: latency isn't the killer feature.
Whoever wins is going to figure out what it really is.
18
7
122
I’m incredibly excited to share that Distributed Masonry will be joining
@confluentinc
! It’s been amazing journey, and I couldn’t ask for a better exit. 1/
23
19
113
I’ve scratched an itch I’ve had for ages.
Introducing Voluble, an intelligent data generator for
@apachekafka
. Voluble generates streams of realistic events with support for cross-topic relationships, tombstoning, configurable rates, and more.
13
36
108
Many teams have created best practices for how they should name & describe their
@apachekafka
events. When they do this, a kind of murky categorization emerges, but no one can quite make out where the boundaries are. As it turns out, linguists beat us to it a long time ago. 1/
5
29
108
Every now and again I see criticism for using
@apachekafka
for small/low volume projects. If you host it yourself, sure, there’s an ops burden. But I think this misses the point: commit logs are a *hugely* useful primitive for any volume. At least as useful as database tables.
6
21
86
It’s been a big week, in lots of different ways. :-) Rooting on
@confluentinc
from the couch.
16
0
82
Anything worth doing is worth being terrible at for the first 3 months.
1
34
78
Serious question: are we done with open source?
Projects like Linux aside, just look at what's happened in the data ecosystem.
1. HashiCorp hard-prioritized their commercial offering, creating an irreparable rift with the OpenTofu community.
2. Confluent and Databricks tried
27
6
76
If you're an engineer who's always wanted to start a company, read this.
1
13
75
This weekend, I learned that
@NOAA
is just about ready to be take
@OnyxPlatform
to production as the backing streaming engine for their automated surface observing system (ASOS). ASOS serves as a primary climatological observing network in the US.
5
17
74
What if you could have an instantaneous
@GraphQL
API over your
@apachekafka
topics? I wondered just that a few months ago. Kafka would get so much more reach almost for free. Subscriptions over streams, and queries + subscriptions over tables of state — right to your front-end.
10
21
73
🔥 Can't believe I get to post this: two days after I closed my first enterprise customer, I've just closed my second!
I just want to say thanks to everyone who's been following along and encouraging me. It's scary trying something high risk, but I feel like I'm doing exactly
5
2
72
First day back with Clojure full-time.
Oh how I have missed you.
5
5
67
The big news is out! Going forward, I'll be supporting Onyx full time. Get in touch to work with me!
#clojure
13
33
66
I don’t like the term “technical debt”—it implies you got something in return. Most technical debt I see are just design mistakes.
16
9
65
Happy 6 years of marriage, to the woman who was smart enough to never sign up for Twitter. 🥳
2
0
64
"Should we obfuscate the jar?"
"Nah, it's written in Clojure. That's already obfuscated for most people."
0
17
59
💥 3 months, 3 weeks, and 2 days into my journey, I've closed my first enterprise customer.
When I started my company, I had no users, no product, no code. All I had was a problem statement that I thought might be true.
Since then, I spent nearly all my time doing two things:
7
1
58
Property-based testing is one of those things that gives me childhood excitement every time I use it. I know how it works, but it still feels like magic.
2
12
58
Very proud to be one of the few open source Clojure projects to grow large enough to take outside investment.
6
12
56
In other
@confluentinc
news, I'm happy to share that I'm now leading up the stream processing product team! Want to work on Kafka Streams and KSQL with me? My group is hiring. DMs open!
4
10
56
My post on streaming+ChatGPT () is dense (it’s a complex subject to unpack!), so here’s a quick thread on just the essentials of what is going on here.
🧵…
1
22
46
Introducing Metamorphic: a complex event processing library for Clojure & ClojureScript.
0
24
54
Just got my copy of Mastering
@kafkastreams
and
@ksqlDB
. If you’re looking for a complete guide to the practical aspects of stream processing, look no further. Brilliant writing
@kafka_book
!
0
14
54
Announcing Pyrostore: streaming storage that never forgets.
Introducing Pyrostore: a new streaming storage product that complements Kafka with inexpensive, virtually limitless storage. Replay your entire dataset seamlessly, losslessly, on demand.
1
14
42
1
26
53
So happy to finally take all the wraps off what my team has been working on at
@confluentinc
.
@ksqlDB
is the start of a really powerful idea to make a substantial dent in the complexity of building stream processing applications. 1/
Announcing ksqlDB: the event streaming database for stream processing apps. ksqlDB combines the power of a stream processor with the familiarity of a relational database. Integrates natively with
@apachekafka
. Learn more from
@jaykreps
:
2
112
197
1
15
53
Regular programming is like chess; Stream processing is like poker.
With the former, all the data you need is visible. With the latter, you never know when the next piece of data will arrive—if at all. It's a game of hidden information.
That’s what makes SP so fun to work on.
4
12
52
9 years later, and Hammock Driven Development is still one of the most influential conference talks you can watch.
2
10
50
Today we added 8 new
@apachekafka
tutorials to learn stream processing, including my favorite: rekeying a stream by a function (easy to pick the wrong key when starting and hard to change it later)
@confluentinc
0
18
51
In all my time working with
@apachekafka
, the hardest part has always been step 1: getting something working end-to-end. We're fixing that. Here's a growing collection of use cases with idiomatic development/automated test/prod deploy patterns.⚡️
We’re excited to announce Tutorials for
@apachekafka
, a collection of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions. Learn more in our latest blog post by
@MichaelDrogalis
:
2
87
208
1
15
51
Right on. My
#KafkaSummit
Europe talk "How ksqlDB works" was accepted. If you liked my visualization blog series going through the basics, you'll love what I have in store for this one.
2
4
51
Goodbye ZooKeeper :)
2
21
49
Classic database queries vs. streaming database queries.
Processing a query is like solving a puzzle. With a classic DB, you can only see the face of the puzzle (the results) after all the pieces (the data) fit together. With a streaming DB, you can see the face as it evolves.
2
15
49
If I could give any piece of advice, about almost any endeavor, it'd be this: finish what you're working on.
2
5
48
Neat, learned how to capture HD animations for Twitter. More action on how streaming joins work coming soon.
1
7
48
🔥 Enterprise customer number 4 has entered the chat.
I'll post updated revenue numbers after a few more closes, but right now I want to tell you guys a story.
From 2020 to mid-2023, I'd been puttering on a completely different start up idea. I probably spent 200+ hours of my
1
0
47
If you blinked, you could be excused for missing the herding effect to Apache Iceberg.
In less than a year, all the major players—Snowflake, Confluent, Dremio, and of course Databricks—have professed Iceberg as the table format of choice.
While public sentiment changed almost
2
7
47
New thing out tomorrow. I think it's one of the most interesting things I've worked on this year.
3
4
47
People say nothing can prepare you to have a child or lose a parent, and it's true.
I love you mom. Everything good in my life started with you.
Honestly, going back to work, much less doing any of this in public, is the last thing I want to do. But my mom always encouraged me.
0
0
45
Huge update to just landed! The team put a ton of work into making this the best place to learn
@apachekafka
, and it really shows. Neat animations, too. :)
1
10
44
People sometimes perceive the fact that
@ksqlDB
only runs on
@apachekafka
as a limitation, but that's backward. Because connectors are a dedicated part of Kafka, all the problems of talking to external systems are centrally managed.
The result: 200+ ksqlDB compatible connectors.
0
9
43
This is cool: in about 20 lines of Python, I have a public AWS-hosted URL that can stream synthetic data to any Kafka cluster.
I think if I can build some traction around this, it would be a game-changer for tutorials.
No more messing with setting up tedious sample data. Just
4
6
43
First revenue just landed in my bank account. Feels so good. 😇
3
0
43
Building a new kind of database at Confluent has been fun, but a few of us wanted to talk about why it's not just a slight deviation on the norm.
Here comes a 3-part blog series over the new couple of days about the architecture, SQL layer, and runtime.
1
9
42
Me early in my career: cool I get to name something!
Me now: oh no.
1
2
42
🔥 Just released: orchestrate an entire series of streaming, synthetic data generators.
In the video, I show how I populate 4 Kafka topics in precise order:
🌱 First, seed a users stream with base information
💰 Second, generate overlapping login and transaction events
📎
2
4
41
Want to work on something that matters for almost every
@apachekafka
installation? I’m hiring a Product Manager for the stream processing team at
@confluentinc
. Come work on a team with two start-up founders and a founding PM.
DMs are open. :)
0
11
41
Team Drogalis: +1. Thanks to everyone who came out with us!
9
0
41
For all the good things databases can do, there’s something critical they can’t. I wrote an essay about the query your database can’t answer, and what it might take to get there.
2
13
40
🚕 This is cool: I converted the New York City taxi data set, which is a set of batch files, into a stream of infinite rides using ShadowTraffic. Works for
@apachekafka
and
@PostgreSQL
.
All I had to do was swap in the right generators for geolocation, ride cost, and a few
3
2
41
Me: I should print up company t-shirts
Also me: prints exactly one
Anyone who knows me personally will attest this is very on-brand.
2
1
40
One of the most impressive parts of
@confluentinc
's tiered storage is that it's completely seamless - something that's been historically hard to achieve. No background jobs, no special clients, no hacky segment copying - simply Kafka with infinite storage.
0
17
40
Infrastructure projects come in all shapes & sizes, but nearly all of them are hard to explain how they work. This is a trend we want to buck with
@ksqlDB
because you cannot rely on what you do not understand. In that vein, here's something new from me.
3
9
40
A little personal news—I'm joining
@benstopford
's team at Confluent in the Office of the CTO! I'll be working on one of my favorite things, making our tech easier to learn/use.
Feels delightfully full circle as Ben was a big reason
@PyrostoreIO
joined Confluent 4 years ago. :)
1
2
39
If you’re a completionist like me, here’s something you can’t hear often enough: give up on poorly written books. The achievement of finishing it isn’t worth the time. Use it on something better.
1
4
38
Wife: Are you free?
Me: I just have one more bug to fix...
Wife: (Reassuringly) Don't worry... There's definitely more than one left.
2
4
38
One of my favorite interview techniques is asking people to spend a meaningful amount of time talking through tradeoffs. You learn so much watching someone scan the surface area of a problem.
1
4
37
If you follow me because you like reading about my open, solo start-up journey, this post is for you. Everyone else, keep scrolling, because this one is raw.
Having done this for almost a year now, here are a few things I now know to be true:
1. Imposter syndrome doesn't go
4
1
37
Been cooking this up for a while now. The Future of Event Stream Processing
0
22
37
Building a successful open-source project is hard, but the formula is straightforward:
1. Make something people find useful
2. Make a website that clearly explains what it is
3. Write A+ documentation
4. Spend a lot of time helping people when they get stuck
That's it.
4
4
36
Gonna log off till Monday.
I've dramatically underestimated how much rest I've needed after a difficult life event, so time to go take a bunch of naps and get away from the computer.
1
0
36
While I'm at it, I get asked once in a while about how to make an open source project go big. A lot of it is blunt effort, but there are a few things that will help your chances *a lot*:
1
13
35
Bumped into this drawing today and can't take my eyes off it. Stunning.
2
3
34
The animation on
@ShadowTrafficIO
's home page is uniquely cool, but here's the secret: it's not a toy purpose-built for the website.
It's powered by ShadowTraffic ACTUALLY running in the browser.
Why? I wanted a marketing asset that's a true purple cow. No one else has anything
6
3
35
❤️ As I spend today putting the finishing touches on things, I just want to say thank you to everyone who's supported me over the last 3 months.
Building in public is fun, but it doesn't come naturally to me. I'm just some guy cranking out software alone in my home office.
4
1
35
Been poking at Clojure for the first time in ~5 years. Great to see how many familiar faces are still around doing great work.
2
0
35
Pretty cool surprise — the
@apachekafka
tutorial site I helped start is on the hotel keycards at
#KafkaSummit
!
1
2
34
Transformation complete!
One year ago I decided to put my ~20 year competitive distance running career on pause to see what I could do on a bike. I bought a Specialized road bike, an old Kinetic trainer, and got on
@GoZwift
.
Jan: FTP <200
Dec: FTP 260, hanging with cat A races
5
0
31
Happy 4th birthday
@OnyxPlatform
! You got a major reliability and perf upgrade this year, and tons of usability improvements. This is also the first year that I spent more time as a (happy) user than a contributor. More in store for you in 2018!
0
10
33
🔥 It's alive!
@ShadowTrafficIO
can now connect directly to Kafka or Postgres and generate matching synthetic streams—without you having to learn anything.
Over the last week, I've been hacking with Meta's CodeLlama-instruct models.
I put together a set of prompts teaching
0
6
33
Incredible
@apachekafka
scale at Tencent. 10 trillion messages/day (4 million/second) using federated clustering. Neat to look inside the mechanics of a giant workload.
1
9
33
After days of profiling and grinding, new ShadowTraffic write speeds:
- ~100K events/s to
@apachekafka
, up from 8K/s
- ~150K rows/s to
@PostgreSQL
, up from 3K/s
It's now a heck of a load more suitable for load/stress testing!
1
2
32
If you liked the stream processing animations that I built out earlier in the year, good news—I'm back at it. Some new content coming up at
#KafkaSummit
Europe, and fresh material for an upcoming video series.
3
4
32
Happy 3rd birthday
@OnyxPlatform
. Couldn't ask for a better adventure and a more polite community to have it with. :)
1
7
32
Just wanted to say thanks to everyone who's supported us over the last few years. Starting a company is the second hardest thing I've ever done, and getting Pyro off the ground would've been impossible with everyone's help and encouragement.
1
2
32
Traits I've learned from the best engineers I've worked with:
1. Obsess about details when it matters and ignore them when it doesn't.
2. Understand the business reason for everything you work on. The best technical plan might not be the best overall plan.
3. Prioritize the
1
8
31
💰I made an easy way to test Debezium change data capture against Postgres.
The problem: to get interesting data out of Debezium, you need to orchestrate a streaming series of table inserts, updates, and deletes.
The solution: ShadowTraffic's `stateMachine` generator. Tell it
1
7
33
If you're an engineer who's always wanted to start a company, read this.
1. You will never feel ready—just begin. No amount of schooling or other experience will put you at ease.
2. You don't need investors to start. Find what problem you want to solve by talking to as many
1
2
30
To push myself as an entrepreneur, I’m going to try something really hard:
I’m launching 4 startups, with 4 products, in quarters—each one run by just me, as a solopreneur.
I wrote a little about why I’m doing this.
Wish me luck. 🔥
9
3
30
My most prized skill is being able to clearly communicate complex ideas. How did I get it?
By having to continually defend why on earth I program in Lisp.
2
1
30
🫣 ShadowTraffic's growing a UI. Runs in the browser.
1
0
30
🧐 With all the new
@ApacheFlink
services coming out, I wonder if people would find it useful to have a consistent set of streams to test their perf/reliability/usability head to head.
I know lots of people are baking off
@RisingWaveLabs
vs.
@Decodableco
vs.
@DeltaStreamInc
vs.
4
1
28
Haven't talked much about what I've been working on since I joined
@confluentinc
, but
@jaykreps
is starting to take the wraps off it today. Check out his
#KafkaSummit
keynote, live streamed at 9:30 AM PST today:
1
10
29
One of the best things programming teaches is how to think in trees. Trees are everywhere.
In coding, modules combine to form trees of execution. In writing, paragraphs, sentences, and words form a tree of ideas. In visual design, objects create a tree of directed attention.
5
5
28