I started working at Commenda as an intern and enjoyed working with Spencer and Yaacov so much that I joined them full time. Working at a company where decisions are open sourced really teaches you a lot
It is easier to be curious about most ideas if you just dive a little bit deeper than the surface level.
I am looking for smart generalist interns who are great at absorbing ambition and want to produce some of their best work.
docs.google.com/document/d/1K6…
Completed the project I've been working on for a while!
Weibo is a stream processing engine built in Go. I was inspired by Apache Flink to build this.
Let's go through the whole pipeline briefly:
Producer - The service or application that produces the data stream which eventually gets ingested into the pipeline.
Durable Store - For example, Kafka. The data is stored durably for some time, and multiple consumers can subscribe to and process it.
Operators - These are basically functions that we can run against the data stream, for example KeyBy, Filter, windows, etc.
Sink - This is the place where the processed data eventually ends up.
Each pipeline is considered a job, and every job runs inside a Docker container that can be hosted on any infrastructure.
There are many more things going on inside the architecture, like watermarks, sliding windows, checkpoints, metrics, barriers, and more.
Check out the docs website and the repo to understand how everything works in more detail!
(If you wanna use this in your product, please let me know and we can work on it together. My DMs are open!)
141 Followers 220 FollowingHead of Product Engineering @neatlogs.
I write about why agents break in production - traces, evals, cost, and the feedback loop nobody owns.