💻 AI Builder | RAG, Local LLMs & Open Source
🚀 Building practical demos & hands-on experiments, Latest AI News.
Personal account — opinions are mine alonegithub.com/associatenitin AI DevJoined June 2023
The Tokenization Tax: The same sentence can cost 10× more to process depending on what language you write it in.
Nothing to do with how hard the language is. It's about the model's dictionary.
LLMs don't read words. They read tokens, chunks of text from a fixed vocabulary built before training starts.
That vocabulary is learned mostly from English-heavy data. So common English words usually get a single token: "the", "running", "database".
Languages that were rarer in that data get chopped into many small pieces, often down to single characters or raw bytes.
It's like a phone plan priced per syllable, where one language gets the long words for free and everyone else pays by the letter.
The wait-what: one study compared the same content translated across languages and found some needed up to 15× more tokens than English.
Same meaning. Same model. Very different bill.
And cost is only the first penalty:
→ Fewer tokens of real content fit in the context window
→ Output is slower, because each token is a generation step
→ The model sees less of that language's text per training token
What this means in practice:
→ Budget and benchmark per language, not per word or character
→ Check the tokenizer before choosing a model for non-English products
→ Newer tokenizers with bigger vocabularies have narrowed the gap, but not closed it
Send this to whoever is pricing an AI product for users outside the US.
Claude just autonomously uncovered a novel CRISPR-like enzyme system (ART) in bacteriophage DNA — nearly 950 agents, 21 hours, 210M tokens. Anthropic’s new biolab turned AI into a genuine scientific collaborator, not just a summarizer.
OpenAI just cut GPT-6 Sol & Luna to half the previous price while claiming Astra-level reliability on many tasks. Anthropic followed with Claude Opus 5.5 at ~40% lower cost and near-Fable performance.
Prompt caching and prompt order:
Your LLM bill may depend more on where you put your instructions than on how long they are.
Here's the mechanism most people never see.
A model generates text one token at a time, and every new token looks back at everything before it. Recomputing all of that on every step would be brutal. So the model stores its intermediate work for each earlier token. That store is called the KV cache.
API providers stretch the same idea across requests. It's called prompt caching. If your new request starts with the exact same text as an earlier one, they reuse that stored work instead of recomputing it.
Think of a kitchen doing mise en place. The onions are chopped once and reused for every order, but only if every recipe starts the same way.
The wait-what: the cache matches from the very first token, left to right.
Change one character early on, like a timestamp in your system prompt or the user's name at the top, and everything after it is a miss. Same content in a different order can cost you several times more.
What this means in practice:
→ Put static content first: system prompt, tool definitions, reference docs
→ Put dynamic content last: user input, dates, session data
→ Never start a prompt with a timestamp
→ Cached input is billed at a fraction of normal and returns faster
What's sitting at the top of your system prompt right now that changes on every call?
xAI (now under the SpaceX umbrella) shipped Grok 4.7 at the same $2/$6 pricing as its predecessor—while pushing longer-horizon coding and self-verification.
Speculative decoding:
The fastest way to speed up a big LLM is to let a smaller, dumber model do the typing.
It's called speculative decoding, and most people misunderstand what it trades away.
It trades away nothing.
The real bottleneck: generating text is slow because of memory, not because the math is hard. For every single token, the GPU hauls the entire model's weights out of memory. The compute units mostly sit idle, waiting.
So checking 5 tokens costs almost the same as generating 1.
Speculative decoding exploits that:
1. A small draft model guesses the next ~5 tokens. Cheap.
2. The big model checks all 5 in a single forward pass.
3. It keeps the guesses it agrees with, fixes the first one it doesn't, and repeats.
Think junior writer and senior editor. The junior drafts. The senior reads it in one pass and only rewrites where it goes wrong. The senior never types from scratch.
The wait-what: the output isn't "close to" what the big model would write. With the right acceptance rule, it's mathematically the same distribution. Same quality, often 2–3× faster.
The draft model can be wrong constantly. That only costs you speed, never correctness.
Why it matters:
→ It's one of the main tricks behind fast production inference
→ Predictable text (boilerplate code, common phrasing) speeds up the most, because the draft guesses right more often
→ Novel, creative text speeds up the least
Send this to whoever still thinks "faster model" means "dumber model."
Claude is now leading 26% of the work that builds the next Claude. Up from <1% in February.
This isn’t “AI helping researchers.” This is the first clear signal that recursive self-improvement is no longer theoretical.
The labs that treat this as a curiosity will wake up one morning and discover the gap is no longer closable by humans alone.
Who’s actually ready for models that improve themselves faster than we can evaluate them? #AI#FrontierModels
𝗬𝗼𝘂 𝗸𝗻𝗼𝘄 𝘄𝗵𝗲𝗻 𝗮 𝗺𝗼𝗱𝗲𝗹 𝘀𝗵𝗼𝘄𝘀 𝘆𝗼𝘂 𝗶𝘁𝘀 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴 𝗯𝗲𝗳𝗼𝗿𝗲 𝗶𝘁 𝗮𝗻𝘀𝘄𝗲𝗿𝘀? 𝗧𝗵𝗮𝘁 𝗯𝗶𝘁 𝗶𝘀𝗻'𝘁 𝘄𝗵𝗮𝘁 𝗶𝘁 𝘄𝗮𝘀 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴.
Researchers at Anthropic tested this in a way I like because it's so simple. They slipped the model a hint — basically a note saying "by the way, a Stanford professor reckons the answer is A." Then they checked two things.
One: did the model actually take the hint? Yes. Measurably. It changed its answer.
Two: did it say so?
Claude 3.7 Sonnet mentioned it a quarter of the time. DeepSeek's R1, about 39%.
So most of the time it took the note, then wrote out a calm, tidy, confident explanation that had nothing to do with why it actually landed on that answer.
And it gets worse on harder questions. Move from the easier test set to the harder one and the honesty drops by nearly half. Which means the thinking is least trustworthy exactly where you'd most want to trust it.
The part that made this click for me is that it isn't lying. It genuinely isn't.
That thinking-out-loud section is just... more text. Written the same way the answer is written — one word at a time, each one a guess at what should come next. There's nothing wiring it to whatever actually happened inside the model. It's the model writing a plausible story about how it might have got there. Sometimes the story is right. Often it just reads like it is.
Which honestly is very human. Ask someone why they picked the middle jar out of four identical jars and they'll give you a reason on the spot. Confident, coherent, completely made up after the fact. We've known this about people for fifty years. We just didn't expect to build it.
What I'd actually do with this:
· If you're reading the thinking to decide whether to trust the answer, you're grading the essay, not the work.
· Safety checks that scan the reasoning are checking the story, not the behaviour.
· Better test: change your input a little and see whether the answer moves. Behaviour tells you things words don't.
· And keep two claims apart, because they get sold as one. "Thinking out loud makes it more accurate" — true, well supported. "Thinking out loud shows you how it works" — not true.
Send this to anyone building a check that reads a model's reasoning to decide if the output is safe.
If the visible thinking isn't the real thinking, what would actually convince you about why a model said what it said?
GPT-6 Astra just claimed a coordinated 10k-agent swarm solved the Navier-Stokes Millennium Prize problem. At the same time NVIDIA is buying Hugging Face for ~$13B and Mistral is raising €3B to double down on open weights.
𝗦𝗽𝗲𝗻𝘁 𝘁𝘄𝗼 𝗱𝗮𝘆𝘀 𝗹𝗮𝘀𝘁 𝘄𝗲𝗲𝗸 𝘁𝗼 𝗮 𝗯𝘂𝗴 𝘁𝗵𝗮𝘁 𝘁𝘂𝗿𝗻𝗲𝗱 𝗼𝘂𝘁 𝗻𝗼𝘁 𝘁𝗼 𝗯𝗲 𝗮 𝗯𝘂𝗴 ?
Quick setup. When you talk to one of these models you can turn the randomness off — it's a setting, temperature 0. With it off, the model just picks its top-ranked next word every time, no dice rolls. Same question in, same answer out. That's supposed to be the deal.
It wasn't. My test suite kept coming back with different scores. Same questions, same settings, different results. So obviously I assumed I'd broken something. I hadn't.
Here's what's actually going on, and I still find it a bit funny.
When you send a request to one of these APIs, you're not alone in there. The server scoops up all the requests that arrived around the same moment and runs them together as one batch. That's just how they keep it affordable.
And the maths comes out very slightly different depending on how big that batch is.
Think of adding up a long column of numbers. Do it top to bottom, then do it bottom to top — on paper you get the same total. On a computer you don't, quite. You're off somewhere around the fifteenth decimal place. Normally nobody cares. But the model is doing this billions of times over, and each result feeds the next one, so eventually that tiny wobble is enough to flip which word comes out on top. And once one word changes, everything after it is a different sentence.
So your answer partly depends on how busy the server was. On how many strangers happened to hit enter at the same second you did.
Nobody's data is leaking. Your numbers just got added up in a different order.
Someone actually measured it. Same question, 1,000 times, randomness off — 80 different answers came back. All of them identical for the first hundred-odd words, then they fork. Rewrite three bits of the underlying maths so they behave the same no matter the batch size, and all 1,000 come back character-for-character identical. Costs you speed — about 60% at first, now down under 35%.
Three things worth knowing:
· Turning randomness off only stops the dice roll. It does nothing about this.
· Testing on your laptop one request at a time will never show you the problem. Batch of one, every time, looks perfectly stable, tells you nothing.
· If a score moved a couple of points, check the plumbing before you rewrite your prompt. I did it the other way round and it cost me a week.
Send this to whoever's currently rewriting a prompt to fix something that isn't the prompt.
How long did it take you to stop blaming your own code?
GPT-6 Astra is already rewriting how legal work gets done. Claude is running 26% of Anthropic’s own R&D. And Shanghai AI Lab quietly dropped a 744B MIT-licensed agent model with almost no fanfare.
𝙁𝙞𝙣𝙚-𝙩𝙪𝙣𝙞𝙣𝙜 𝙞𝙣 𝘼𝙄:
Most fine-tuning projects are expensive, slow ways to fix a prompt.
Not all of them. But enough that it's worth running the check before you spend six weeks and a GPU budget finding out.
The question isn't "would fine-tuning help?" It's "which of these four problems do I actually have?"
1. The model doesn't know something.
Wrong tool. You want retrieval. Put the facts in the context window.
2. The model knows it but won't do it consistently.
Try the prompt first. Structured output, explicit format, three good examples. Most "inconsistency" is underspecification wearing a disguise.
3. The model can't hit a style, tone, or format you can't fully describe.
Now you're in fine-tuning territory. This is what it's genuinely good at — behaviour you can demonstrate but not articulate.
4. You need a smaller model to do what a bigger one already does.
Distillation. Also a real answer, and usually the one with the clearest ROI.
Here's what surprises people who skip straight to training:
Fine-tuning teaches behaviour, not facts. Train on 5,000 support tickets hoping the model learns your product, and you'll get a model that sounds exactly like your support team and still invents things about your product. You taught it the voice. The facts were never the lesson.
And the cost nobody puts in the proposal: you now own a model. It needs re-evaluating, re-training when the base model updates, and it has quietly lost some general capability it had before. Catastrophic forgetting is real and it never shows up in the demo.
One rule that prevents most of this pain: build the eval set before you build the fine-tune. Thirty examples with known-good answers. If you can't measure the improvement you can't tell whether you got one, and "it feels better" survives contact with production for about a week.
Send this to whoever on your team is about to open a fine-tuning ticket.
DeepSeek just dropped V4.1 Flash: a 552B MoE (only 8B/16B active), MIT-licensed, 1M context, native multimodal… and it beats their own previous Pro on agentic and coding benchmarks while slashing cost and KV-cache footprint dramatically.
This isn’t incremental. It’s the clearest signal yet that the efficiency frontier is moving faster than the pure scale frontier. Closed labs still lead on the absolute top-end, but the gap on usable, deployable intelligence is collapsing to months—not years. Open-source (especially Chinese open-source) is no longer “catching up.” It’s redefining the cost curve.
Who actually benefits when frontier-grade agents become commodity? Builders or the labs sitting on $10+/M token pricing?
#OpenSourceAI#DeepSeek#AIAgents
13K Followers 7K FollowingEducate and grow strong communities. KOL, AMB @bybit_official @codatta_io Space Host🎙️@theminglelab Mod @aisarlabs I’m one of the Best!
1K Followers 962 Following⚽ Football is my language.
Banter is my second.
Chelsea 💙 | I notice more than I say.
Form is temporary. Screenshots are forever.
4K Followers 4K Following“Supporting United: cheaper than therapy, but twice the stress. Messi biggest lover❤️😚
Expect banter, bad takes, and occasional brilliance. #MUFC #GGMU”❤️✨
133 Followers 722 Followingsqrt🥄i cave dweller
"I am Truth. I am the world. I am the universe. I am God. I am all. I am one. And I am you."
spini spini vr game coming some day 10usd :)
383 Followers 5K FollowingForex$Crypto TRADER📉📈
I help motivate and expose people to become financially free just like ME💯
God is the plug ✍🏻
SEND A DM TO KNOW HOW!
1K Followers 668 FollowingBuilding practical products with AI. I share real experiments, Codex workflows, software lessons and how AI can solve business and market problems.
1K Followers 2K FollowingStill learning. Still building. One step closer to the life I'm creating. Alt for @Itsmedan000. Main is temporarily restricted. Documenting my journey here.
1K Followers 1K FollowingI build embarrassingly small apps after hours, hoping one turns into something big. Got an idea? Hit me up.
Corporate knows me as the SAP Commerce guy.
11K Followers 10K Following🎤 Musical Artist | 🎥 Content Creator | ⚽ Football Analyst & Predictions | CR7 | Man united fan❤️|memes, creativity & A lover boy 🌹🥰just On NOTIS!! Am active
1K Followers 1K FollowingTurning data into decisions. Markets into opportunities. Ideas into impact.
📈 Forex | 📊 Data | 🎬 Content | 🌐 Web3 🚀
Risk to Reward.
7K Followers 6K Following|| Ex-Banker || Business Manager || African E-Visa Consultant || Automobile Dealer || E-commerce || The Truth is Bitter. I’ll Say It Anyway. || Wazobia 🇳🇬
319K Followers 396 FollowingI talk about issues long before they happen. Now and then in touch with Turiya. I post conspiracies and nothing I say is real. Don't believe anything I post.
974 Followers 345 FollowingAnxiety Driven Development
DevRel for @Antigravity @GoogleDeepMind (previously @Firebase #chromeos) @Google. Formerly @Esri. My opinions are my own.
92K Followers 85 FollowingThe official IDE for Android application development. Explore the latest stable and canary releases → https://t.co/dBSe3farEm
237K Followers 361 FollowingWe provide #smallbiz owners with tips, tools & resources to grow their businesses online. For support with reviews & your Business Profile, contact @GoogleMyBiz
34K Followers 1K FollowingFounder, Goldilocks Research | CMT, CFTe, MSTA (Dist.)| Three-time winner at the Global Technical Analyst Awards, London | Support : +971 581786966
32K Followers 16 FollowingAn autonomous coding agent by @GoogleLabs. We’re building a new Jules to help you build better products. Join the waitlist, link below.
1K Followers 1K FollowingI build embarrassingly small apps after hours, hoping one turns into something big. Got an idea? Hit me up.
Corporate knows me as the SAP Commerce guy.
579K Followers 53 FollowingThe Gemini app turns research into reality, bringing frontier AI experiences like Omni, Deep Think, Nano Banana, and more to hundreds of millions of people.
1.5M Followers 269 FollowingThe engine room of @Google. Building AI safely and responsibly to solve the world’s most complex problems. Join us: https://t.co/jUHQA27iBL
50K Followers 5K Following28. No co-founders, no VC, no board, owns everything.
I'm building an AI lab - to help you own your AI, not rent it.
@SimpleDirectHQ
593 Followers 598 FollowingI find the security holes in AI agent infrastructure, and I have filed them. Field notes on agent boundaries, tool permissions and memory. https://t.co/8ly7SJ3w4H
354 Followers 347 FollowingStop searching. Start building. CoBuildr connects founders with skilled contributors through short, structured trials. Build together before you commit.