Jack Vanlightly @vanlightly
Likes breaking ideas and systems, writing, picking systems apart @confluentinc Ex @Splunk, @VMware https://t.co/voJWmL5iBE Credit: ESO/B. Tafreshi jack-vanlightly.com Barcelona, Spain Joined November 2016-
Tweets2K
-
Followers5K
-
Following250
-
Likes1K
Yeah we just wrote a paper (LogDrive) about some of the internals of K2. Oh wait, this is another K2...
Cloudflare is working on K2, a durable event-stream primitive. It's a named, append-style log of "records" you produce to and consume from, with a configurable retention window, writable from a Worker binding or over HTTP. Basically Cloudflare Kafka.
My TLA+ spec for Cursor's Continuity is done. github.com/Vanlightly/s3-… Next I will evaluate Tobias Lütke's walgit to see if it makes any interesting changes to the design.
Well the "similar to Buffer" was really only the wal index (aka manifest) the rest is quite different.
I'm looking at Cursor's Continuity blog post (cursor.com/blog/git-at-an…). It seems very similar to OpenData Buffer, with compaction added on. Though the packfiles also look like they're numbered? Though that seems unnecessary as the reference file determines the ordering. Are there any more details than the blog post? I think I'll take my Buffer spec and add compaction (WAL GC) to it.
This spec completely abstracts away the realities of git and its storage format. It simply models a sequence of values (representing commits) and ensures that all replicas see and apply the same sequence of values, which also are consistent with the pushes and fetches of each client. In that simplified model, everything is looking good so far.
Just putting finishing touches in the Continuity spec. I implemented a simplified compaction via snapshotting the repo and updating the wal index with the snapshot id (which represents the position of the snapshot in history and the compaction frontier). I've mostly left out GC of packfiles (it uses some magic to determine which can be deleted), as Cursor's blog post doesn't cover that. I'll look at walgit next as that is an implementation inspired by the Cursor blog post.
Continuity compaction is very hand-wavy in the blog post. Only the primary does it, but at the same time, it's a soft "primary", more like multi-master with soft affinity. There is a compaction frontier. Adjacent pack files get merged. That's about it. I'll just have to figure something out.
I made good progress on the Cursor Continuity TLA+ spec. Reads and writes done. Next compaction. The visualizations in Cursor's blog post are helpful, but it's still not quite enough to fully describe the protocol. For example, as far as I can figure, a rebase is not only needed after a write conflict. Two replicas can write their pack file concurrently but then update the WAL index serially (without a CAS conflict): 1. r1 writes pack file p1 2. r2 writes pack file p2 3. r1 reads the wal index 4. r1 appends p1 to the index, CAS writes it and commits locally 5. r2 reads the wal index 6. r2 see's that a new entry was added and must now rebase 7. r2 appends p2 to the wal index, CAS writes it and commits locally Ultimately, very similar to OSWALD which is also multi-writer. If a replica sees that it is behind, it must catch up before appending its own op.
Thinking about it, only compaction would trim the reference file, so we don't need numbered packfiles. Finding orphan packfiles is a separate concern, which for now I won't tackle. So I'll reuse the write path of the Buffer spec but the rest will be different.
Hmmm, numbering the packfiles may help with GC as one of the problems with Buffer is knowing when an object can be deleted. How does a GC tell the difference between a newly updated object not yet added to the manifest and a previously commit object that was removed from the manifest? Buffer relies on time: it encodes a timestamp into the object ids and uses a deletion grace period. So not strictly safe as far as dist sys protocols go but probably fine given a large enough grace period.
The architecture description is not detailed enough to write a TLA+ spec of the protocol and S2 Lite defers the heavy lifting to SlateDB, so for the purposes of this survey, which is the explore WAL-on-S3 protocol designs, I can't include S2. Not saying S2 isn't a great system though!
@jeffreyhuber Ok, added to my list thanks
After a few days off I'm back on my project to survey WAL-on-S3 designs. This morning I finished a TLA+ spec of a variant of the SlateDB WAL protocol that uses a single CAS-write manifest file instead of numbered files protected by put-if-absent writes. After some coffee, I'll consider the next design on the list that I'll model. github.com/Vanlightly/s3-…
@lcm_in_ai Yep, modeled that already github.com/Vanlightly/s3-…
Looks like I can't write a spec for S2.dev as the code is closed source. There is s2-lite which is open and uses SlateDB, so I'm going to assess whether there's a serious protocol there or not.
Ok, S2 Lite delegates single-writer enforcement to SlateDB’s fenced WAL, so acknowledged appends should remain linearizable with competing Lite nodes. But Lite assumes a single node: nodes answer check-tail from cached state (so a stale node wouldn't discover it's fenced), which then impacts reads (stale tail can cause a node to say an address is Unwritten when it is actually Written). So Lite isn't linearizable under multi-node use. Any automation trying to keep one node running would need to be careful so that the worst case would be stale prefix reads (monotonic, prefix-consistent failover), which "can" be ok for streams. SlateDB does the heavy lifting and that's already modeled so I'm moving on.
SlateDB WAL protocol TLA+ spec: github.com/Vanlightly/s3-… Next I'm going to play with a CAS-write manifest version of it (instead of numbered manifests). Just because, I'm on extended vacation and I enjoy this stuff 😄
It finished after 8h 32min, 3.3 trillion states, all properties held. Tomorrow I'll publish the spec. I might also explore a variant of the protocol without numbered manifests (for those that don't need that).
Finally doing a long TLC run of the SlateDB spec. 3 writers, 2 gc processes, and just 3 values to write. So far TLC has been running for 6 hours with 16 workers, queue size peaked at 140M, explored 2.7 trillion unique states so far, with 90M on the queue still, probably another 3-6 hours to go. Then just gotta clean up the spec and decide which WAL-on-S3 to verify next.
Gunnar Morling 🌍 @gunnarmorling
71K Followers 300 Following Technologist @Confluentinc · Ex-lead of Debezium · Spec lead of Bean Validation 2.0 · Creator of Hardwood, kcctl, JfrUnit, MapStruct · Java Champion · 🚴
Marc Brooker @MarcJBrooker
28K Followers 750 Following Distinguished engineer at AWS. AI, agents, databases, and serverless. Views are my own.
Alex P @ifesdjeen
16K Followers 1K Following Distributed and Storage Systems. Apache Cassandra Committer and PMC member. Author of Database Internals @therealdatabass. Discord: https://t.co/8LwhZom9eQ
Robin Moffatt 🍻�... @rmoff
10K Followers 643 Following Shitposting and Memes. 🌐 https://t.co/WparjfmCF5
Gwen (Chen) Shapira @gwenshap
30K Followers 11K Following Co-founder of @niledatabase. Making SaaS global, elastic and chill. Find me at: https://t.co/uyuHg400cp
Murat Demirbas (Distr... @muratdemirbas
11K Followers 259 Following MongoDB Research: Distributed Systems, Databases, Formal Methods ex-AWS, ex-Prof SUNY Buffalo. Blog at: https://t.co/IqZPhhtcCC Opinions my own
Dominik Tornow @DominikTornow
12K Followers 631 Following Founder @resonatehqio | Distributed Async Await | Think Distributed Systems | https://t.co/TQGtUTQMvY
Martin Kleppmann @martinkl
50K Followers 939 Following Find me at @martin.kleppmann.com on Bluesky, @[email protected] on Mastodon. Author of @intensivedata, Associate Professor @Cambridge_CL. he/him
Pekka Enberg @penberg
20K Followers 1K Following Founder & CTO @tursodatabase; previously @ScyllaDB and Linux kernel. Author of "Latency" (https://t.co/XWRFq71WaJ).
ahmet alp balkan @ahmetb
51K Followers 390 Following infra @baseten 🌹 oss: https://t.co/LbCjmbdKkJ did large-scale baremetal compute in a past life
Kelly Sommers @kellabyte
51K Followers 368 Following 🇨🇦 Backend Brat. Distributed Diva. Relentless Learner.
Matthias J. Sax 🦦 @MatthiasJSax
4K Followers 492 Following Battle Born | Software Engineer @ConfluentInc | GM of @KafkaStreams 🦦 | @TheASF Member, Committer & PMC member | Go @BuffaloBills 🦬 | Home Means Nevada
Ryanne Dolan @DolanRyanne
974 Followers 544 Following Streams and Pipelines @LinkedInEng @TwitterEng @Cloudera @Hortonworks
🕹️ Alexander Gal... @emaxerrno
4K Followers 3K Following Founder & CEO of @RedpandaData. Building the Agentic Data Plane - https://t.co/xyvsqqV5OM. 🇨🇴
Ivan Prisyazhnyy @joh... @JohnKoepi
896 Followers 410 Following software engineer, distributed systems, open source, linux, cooking, music, and sports. @discuss.systems, @fosstodon.org
Richard Seroter @rseroter
37K Followers 2K Following Sharing tech ideas from people smarter than me. Senior Director and Chief Evangelist @googlecloud, writer, speaker.
Jaromir Hamala @jerrinot
2K Followers 2K Following I play with distributed systems by day and poke various runtimes by night. Engineer @QuestDB, @Hazelcast alumnus.
@norootcause.surfingc... @norootcause
13K Followers 5K Following Lorin Hochstein Student of complex systems failures, resilience eng, cognitive sys eng. Will talk your ear off about @LFISoftware. @[email protected]
Adi Polak @AdiPolak
15K Followers 903 Following Productionizing AI systems @ Confluent • Author • Databricks MVP • Ex Microsoft, Akamai..
Sai @cydatio
45 Followers 2K Following
nixpiper @nixpiper
109 Followers 4K Following
borar @Savlambda
417 Followers 692 Following
MK @KabeerHas
79 Followers 5K Following
whocares @whocaresxam
0 Followers 122 Following
Sanjay Nadaraja @SanjayNadaraja
256 Followers 1K Following
Suraj Ranganath @surajranganath
140 Followers 900 Following
Souvik Banerjee @Souvik1997_
112 Followers 2K Following HFT, Apple Core OS kernel security. UT Austin CS+Math 🤘 working on sandboxes and AI agent security. https://t.co/iVf0BjQXOY
Dan Palmer @therealdaneel
1K Followers 3K Following CTO @teller. Building the best banking APIs in the world. Join us: https://t.co/Kp0yMRYRmb
Aarav Jaiswal @aar4vj
25 Followers 2K Following
dziban @dzibanisms
20 Followers 202 Following Figuring out how to use AI responsibly and with strong fundamentals. *Nothing* on this account is written by AI. Working at @PostHog
Sushant Dinesh @_sushant94
117 Followers 212 Following Co-founder & CEO @ Aretta AI: the verification layer for AI agentic code | Prev: Postdoc/PhD @ EECS UC Berkeley
Dan "18pF flip-flop" @dcominottim
2K Followers 6K Following Passionate about computer HW and SW design & architecture. Walking the Way of the Fool.
Eli Grubb @keygrubb
205 Followers 1K Following rotating keys @bitwarden. merkle tree enthusiast. fan of movies, utah, c(ryp ∪ inema)tography.
Alex J Best @AlexJBest
613 Followers 4K Following Researcher in maths+formal methods+ml. Using formal verification to train models for mathematics, software, hardware, general reasoning @harmonicmath
cedric @saucisoupe
58 Followers 742 Following systems programming/async/backend/rust, getting into verification
Megan Reynolds 👩�... @meganreyno
5K Followers 5K Following first check infra VC / https://t.co/URmyndIemd founder
Shreyas @Shreyas_Mishra2
0 Followers 57 Following
big pep @pepyakin
1K Followers 622 Following Protocols, databases, execution (wasm, RISC-V). Ex Irreducible, NOMT, Polkadot.
Ayush @the_humble_lazy
123 Followers 2K Following Finding myself Tech, businesses, poetries, hip hop, music | Build Baby Build
Ajai Pasupuleti @PasupuletiAjai
3 Followers 539 Following
Michael Lennon @Michael80578148
24 Followers 137 Following
alias @loadingalias
785 Followers 3K Following Founder @ alias-research | cargo-rail | rscrypto | privacy advocate | Member @ Zed Guild 🦀
koper @lukaszkoper19
2 Followers 104 Following
Bogdan Kanivets🇺�... @bkanivets
107 Followers 324 Following Software engineer @apple. Opinions are mine.
kdp @fr_its_me_kdp_
6 Followers 136 Following Computer Science Engineering Student. Passionate Software Engineer and into new tech news
Bennet Leff @LeffBennet16107
0 Followers 47 Following
Chaitanya @chaaai
14K Followers 1K Following Founder @workersio (YC F26) - All things software reliability
Fadil Khodabuccus @fkhoda
102 Followers 2K Following Principal Engineer @Checkout. Opinions are my own.
William Rudenmalm🇪... @w_hgm
9K Followers 2K Following writing Rust 🦀 - ml, comp vision + llms - work w/ coding agent and sandboxes @lovable_dev
James @lcm_in_ai
1K Followers 727 Following Founder https://t.co/rBaTVh2zif shipping AI in enterprise
Ricardo Moreno @rimorpo
30 Followers 775 Following Software Developer. Proud parent. Opinions are my own.
Chenglong Chen @ChenChengl97106
7 Followers 478 Following
Gabin 🌱 @g48in
10K Followers 1K Following
Adrian Yates @OceanYangs21679
73 Followers 740 Following
Bill Njoroge @bnjorogedev
592 Followers 475 Following agent-driven ci @ https://t.co/2alOVD2y1t. the just-in-time ci guy
Van Veluwen @vanveluwen
44 Followers 112 Following Cofounder of Hurdle, Editor of TLDR Data, writing, droning, programming, not a serious person most of the time yada yada yada
Gunnar Morling 🌍 @gunnarmorling
71K Followers 300 Following Technologist @Confluentinc · Ex-lead of Debezium · Spec lead of Bean Validation 2.0 · Creator of Hardwood, kcctl, JfrUnit, MapStruct · Java Champion · 🚴
Marc Brooker @MarcJBrooker
28K Followers 750 Following Distinguished engineer at AWS. AI, agents, databases, and serverless. Views are my own.
Jaana Dogan ヤナ �... @rakyll
170K Followers 1K Following ML/AI Systems at Google. Simplicity and optimism. Personal opinions. 🇺🇸🇪🇺
Cindy Sridharan @copyconstruct
47K Followers 165 Following
Alex P @ifesdjeen
16K Followers 1K Following Distributed and Storage Systems. Apache Cassandra Committer and PMC member. Author of Database Internals @therealdatabass. Discord: https://t.co/8LwhZom9eQ
Andy Pavlo (@andypavl... @andy_pavlo
42K Followers 211 Following Database Research @ClickHouseDB. Associate Professor of Databases @CarnegieMellon.
Aleksey Shipilëv @shipilev
28K Followers 582 Following In love-hate relationship with machines. Doing questionable things, so you don't have to. Personal: astrophotos + basic gloom. Work: everything OpenJDK at AWS.
Robin Moffatt 🍻�... @rmoff
10K Followers 643 Following Shitposting and Memes. 🌐 https://t.co/WparjfmCF5
Gwen (Chen) Shapira @gwenshap
30K Followers 11K Following Co-founder of @niledatabase. Making SaaS global, elastic and chill. Find me at: https://t.co/uyuHg400cp
Murat Demirbas (Distr... @muratdemirbas
11K Followers 259 Following MongoDB Research: Distributed Systems, Databases, Formal Methods ex-AWS, ex-Prof SUNY Buffalo. Blog at: https://t.co/IqZPhhtcCC Opinions my own
Dominik Tornow @DominikTornow
12K Followers 631 Following Founder @resonatehqio | Distributed Async Await | Think Distributed Systems | https://t.co/TQGtUTQMvY
David Fowler @davidfowl
148K Followers 1K Following Distinguished Engineer at Microsoft, Creator of NuGet, SignalR, https://t.co/PiydKAsC8g Core, Aspire and Barbadian 🇧🇧, Tennis Player, Father, other half @symonefowler
Martin Kleppmann @martinkl
50K Followers 939 Following Find me at @martin.kleppmann.com on Bluesky, @[email protected] on Mastodon. Author of @intensivedata, Associate Professor @Cambridge_CL. he/him
Pekka Enberg @penberg
20K Followers 1K Following Founder & CTO @tursodatabase; previously @ScyllaDB and Linux kernel. Author of "Latency" (https://t.co/XWRFq71WaJ).
Viacheslav Biriukov @brk0v
8K Followers 402 Following Linux System Engineer 💻 ex-Cloudflare, ex-Apple, ex-Facebook. Distributed systems and networks. Golang, Rust, Zig, eBPF, C.
Kelly Sommers @kellabyte
51K Followers 368 Following 🇨🇦 Backend Brat. Distributed Diva. Relentless Learner.
Matthias J. Sax 🦦 @MatthiasJSax
4K Followers 492 Following Battle Born | Software Engineer @ConfluentInc | GM of @KafkaStreams 🦦 | @TheASF Member, Committer & PMC member | Go @BuffaloBills 🦬 | Home Means Nevada
Ryanne Dolan @DolanRyanne
974 Followers 544 Following Streams and Pipelines @LinkedInEng @TwitterEng @Cloudera @Hortonworks
third-party evaluatio... @mycoliza
23K Followers 1K Following despite it all, i still love these silly computing machines. making a better computer at @oxidecomputer. a girl, in case you couldn't tell.
Gabin 🌱 @g48in
10K Followers 1K Following
ba0tiao @baotiao
3K Followers 3K Following Manager of Alibaba Cloud RDS Team, Opinions are my own. MySQL && PostgreSQL Developer blog: https://t.co/IuBfiXnsMX 我就逼逼... 熟人勿 fo
Tobias Ziegler @Tobias__Ziegler
2K Followers 487 Following Optimizing Distributed (Database) Systems @TigerBeetleDB — the financial transactions database designed to power the next 30 years of transaction processing.
Maximilian Kuschewski @maxikuschewski
271 Followers 588 Following Optimizing Performance at @TigerBeeetleDB. Formerly Ph.D. Student at @TU_Muenchen.
Colin Breck @breckcs
4K Followers 855 Following Principal Engineer at Tesla. Working on databases, distributed systems, and edge computing platforms for energy. Views are my own.
Johnson Shi @johnsonshi86
930 Followers 5K Following Senior Product Manager @Microsoft @Azure ⚡️👨💻 Previously SWE at Azure Linux. Scaling up AI Infrastructure for inferencing and training on Azure 🌥️
Pavel Tcholakov @PavelTcholakov
939 Followers 1K Following Turing machine tamer, also interoperates with human beings. Love hi-perf systems and teams! Working on @restatedev – previously engineering @stedi, @awscloud.
ME/CFS Science @mecfsskeptic
6K Followers 492 Following In-depth analysis of research on myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS).
Daniel Lemire @lemire
38K Followers 2K Following Software performance expert. Ranked in the top 2% of scientists globally (Stanford/Elsevier 2025) and among GitHub's top 1000 developers. Father, husband.
Agents Rurals @agentsruralscat
79K Followers 323 Following Agents Rurals. Generalitat de Catalunya. En cas d’incidència, truqueu sempre al 112. 📌 Normes de participació: https://t.co/nOFkfeNxUp
Shriram Krishnamurthi... @ShriramKMurthi
24K Followers 4K Following @BrownCSDept/@BrownUniversity • @BootstrapWorld • @PyretLang • @racketlang • Unreasonably excited about compsci, education, cycling, cricket, human experience.
Hannah Davis @ahandvanish
56K Followers 4K Following Research, algorithmic music, anti-bias in AI data. #LongCovid research & advocacy @patientled. DMs rarely checked
Almog Gavra @almoggavra
1K Followers 335 Following building and writing about databases & object storage 🛠️ main @ https://t.co/ex4dkqOCdR | blog @ https://t.co/bGsuPwdAkL | side gig @ https://t.co/rkUANXa8TV
Piotr Gankiewicz @spetzu
2K Followers 398 Following Founder & CTO @laserdatainc | Apache Iggy PMC | Building next-gen message streaming 🦀
Tyler Hillery @_TylerHillery
2K Followers 888 Following Software Engineer (Storage) @supabase. Creator of https://t.co/p769oCRmDS. Co-organizer of https://t.co/JIJdxbc2gc. Peeling back the layers of abstraction.
Ben Dicken @BenjDicken
48K Followers 910 Following databases → @planetscale videos → https://t.co/J5soMyf6XC writing → https://t.co/YpfkF0JaV8
Clifford Richardson @CorvusCrypto
410 Followers 103 Following exabyte scale data strategy by day job. safety critical aviation/marine systems by passion. Not involved in crypto; short for Corvus cryptoleucus :D
Marco Slot @marcoslot
1K Followers 663 Following Mostly tweets about Postgres, Snowflake Postgres, and Postgres extensions. Formerly Crunchy Data, Microsoft, Citus Data, AWS, TCD, VU
Sunny Bains @TiDB @sunbains
5K Followers 261 Following swe@PingCAP - The company behind TiDB. Oracle/MySQL/InnoDB team lead in a past life
Papadopoulos Teo @dogvile
3K Followers 5K Following Tweets about Web Development, UX, Data Visualization, Data Science, Infographics, BI, Big Data, Science, Complexity, Networks Science
Alex Miller @AlexMillerDB
3K Followers 211 Following Databases. See also @[email protected] or @alexmillerdb.bsky.app
Phillip Carter @_cartermp
6K Followers 358 Following Copilot Platform @GitHub. Formerly Observability @Honeycombio and F# lang/.NET tools @Microsoft
Charity Majors @mipsytipsy
82K Followers 592 Following honeycomb, observability, the second edition of "Observability Engineering" is coming out soon. find me on substack, bsky, LI, and occasionally here. 🐝🏳️🌈🦄
Pawel Hajdan @pawelhajdan
951 Followers 97 Following I help Fintech Founders build high-performing engineering teams / Ex-Google Tech Lead (50+ engineers) / Ex-Citadel
Vinoth Chandar @byte_array
2K Followers 234 Following Founder @Onehousehq — Data Infra for Agents. Creator of @apachehudi, built the first #DataLakehouse @Uber. Databases, Dist. Systems. LinkedIn, Confluent.
Mim @mim_djo
13K Followers 3K Following #MicrosofFabric user advocate, interests in Small Data & Self Service #Microsoftemployee since Dec 2023 , but my tweets are my own
Neelesh Salian 💻 @nssalian
2K Followers 70 Following Software Engineer. Opinions and views shared here are my own. Most posts are jokes. I post about things related to data and tennis.
Ethan Mollick @emollick
391K Followers 594 Following Professor @Wharton studying AI. New book, Co-Existence, coming October 20. Preorder here: https://t.co/hsDYf1xU9K Substack: https://t.co/UIBhxu4bgq
Noah Pepper @noahmp
6K Followers 2K Following https://t.co/JpIPsnZrTN https://t.co/c3telwLzfp @govtechsg Previous: @stripe, @twitter, @luckysort Topics: Startups, investing, building
Sarah Catanzaro @sarahcat21
16K Followers 2K Following “All methods are sacred if they are internally necessary” (GP @amplifypartners, prev @canvasvc; Head of Data @Mattermark; @palantirtech; @c4ads)
LaurieWired @lauriewired
163K Followers 303 Following researcher @google; serial complexity unpacker; https://t.co/Vl1seeNgYK ex @ msft & aerospace
Matthias Boehm @matthiasboehm7
1K Followers 620 Following Prof at TU Berlin and BIFOLD; research on ML systems and data management. @matthiasboehm7.bsky.social
François Chollet @fchollet
733K Followers 829 Following Co-founder @ndea. Co-founder @arcprize. Creator of Keras and ARC-AGI. Author of 'Deep Learning with Python'.
Shreya Shankar @sh_reya
56K Followers 807 Following Incoming assistant professor @CSDatCMU @CMUDB. Putting LLMs in databases and BI tools. Created https://t.co/PmuOqAXVgS and https://t.co/8MQt4na2cj.
Andreas Kipf @andreaskipf
1K Followers 481 Following Professor @utn_nuremberg. Formerly @AWS, @MIT_CSAIL, @TU_Muenchen. Focused on building efficient, easy-to-use data systems.
Peter Kraft @petereliaskraft
6K Followers 107 Following Co-founder @ https://t.co/Tc5E74e83I • Stanford PhD • Database Geek • Building https://t.co/HJhvPqxL2p
DataScienceWeekly @DataSciNews
35K Followers 866 Following An in-depth look at the world of Data Science. Sign up for our free weekly newsletter featuring curated news, articles and jobs related to Data Science...
Ovais Tariq @ovaistariq
3K Followers 1K Following Building @tigrisdata ⏩ https://t.co/XdvyNOzBCr | globally distributed object storage. I post strong opinions about infra. Before this, I ran storage at Uber.
Kevin Liu @kevinjqliu
241 Followers 472 Following



















