Rodrigo Gonzalez @AutoExAI

Stanford just released the kind of LLM lecture Anthropic pays engineers $750,000+ a year to understand: Not prompting Not RAG How to actually build language models from scratch In this 104-minute lecture, they break down the full stack behind modern LLMs: 06:18 - how language models are built from raw data 21:42 - tokenization and model architecture 39:17 - why scaling laws predict model performance 57:36 - how compute, data, and parameters trade off 1:14:28 - what actually happens during post-training 1:34:11 - the engineering skills needed to build frontier models This lecture is worth more than 100 LLM tutorials Bookmark and watch it today Then read the full guide to building language models from scratch below

rari @0xwhrrari

2 months ago

x.com/i/article/2086…

37 174 1K 4.1M 4K

💯

Nic Cruz Patane @niccruzpatane

11 months ago

I don’t understand how you can own a Tesla and not use FSD! I see so many Tesla owners driving manually when there’s honestly no need. It changes your life—the data shows it’s a 7x improvement in avoiding accidents.

2K 338 4K 13.8M 190

Memory engine for AI Agents with a single line of code. Just call memori.enable() to give any LLM or agent persistent memory. Uses existing SQL database - no vector databases required. 100% open-source.

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Anthropic is so, so over. Just completely defeated. Really sad, so many great people there. My advice: go to DeepMind before it's too late.

(1/n) 🚨 BERTs that chat: turn any BERT into a chatbot with diffusion hi @karpathy, we just trained a few BERTs to chat with diffusion — we are releasing all the model checkpoints, training curves, and recipes! Hopefully this spares you the side quest into training nanochat with diffusion for now 🙂. It’s both a hands-on tutorial for beginners and an example showing how to use our complete toolkit (dLLM) for deeper projects. Code: github.com/ZHZisZZ/dllm Report: api.wandb.ai/links/asap-zzh… Checkpoints: huggingface.co/collections/dl… Motivation: I couldn’t find a good “Hello World” example for training a minimally working yet useful diffusion language models, a class of bidirectional language models capable of parallel token generation in arbitrary order. So I tried finetuning BERTs to make it chat with discrete diffusion—and it turned out more fun than I expected. TLDR: With a small amount of open-source instruction-following data, a standard BERT can gain conversational ability with diffusion. Specifically, a finetuned ModernBERT-large, with a similar number of parameters, performs close to Qwen1.5-0.5B.

Andrej Karpathy @karpathy

11 months ago

Nice, short post illustrating how simple text (discrete) diffusion can be. Diffusion (i.e. parallel, iterated denoising, top) is the pervasive generative paradigm in image/video, but autoregression (i.e. go left to right bottom) is the dominant paradigm in text. For audio I've

266 524 5K 871K 3K

This simple Claude Code hack has reduced token usage by ~90%. It adopts the "Code Execution with MCP" concept published by Anthropic. Remove preloaded MCP tools from context and use Python to execute tools via bash instead. BTW, this can be optimized much further. Insane!

First tools, then memory... ...and now there's another key layer for Agents. Karpathy talked about it in his recent podcast. Tools help Agents connect to the external world, and memory helps them remember, but they still can't learn from experience. He said that one key gap in building Agents today is that: "They don't have continual learning. You can't just tell them something and they'll remember it." This isn't about storing facts in memory, but rather about building intuition. For instance, when a human masters programming, they don't just memorize syntax. Instead, they develop heuristics, learn edge cases, understand context, and build genuine expertise/skills through repeated interaction. But current agents typically start from scratch every time. Karpathy mentioned one possible path forward, which is to provide Agents with some kind of "distillation phase" that takes what happened during interactions, analyzes it, generates synthetic examples, and updates their understanding via RL. This is similar to how humans consolidate experiences into learning. Composio is actually building the infrastructure to solve this and provide a shared learning layer for Agents to evolve. Think of it as the "skill layer" that gives Agents an interface to interact with over 10k tools while building practical knowledge from those interactions. Interestingly, this direction also aligns with what Anthropic is exploring, codifying repeated agent behaviors as skills .md files. Both approaches point toward a similar design pattern where agents progressively turn experience into reusable, composable skills. So when one agent learns how to handle specific API edge cases, that knowledge becomes available to every other agent via Composio's collective AI learning layer, resulting in Agents that don't just automate but rather develop real intuition. This is what Karpathy meant by continual learning, where Agents don't just memorize, but accumulate skills as they interact. I have shared the Composio GitHub repo in the replies!

Google released another whitepaper 🔥. it's a masterclass on building intelligent optimized sessions and long-term memories for agents that actually work. It covers: > context engineering best practices > comparing memory and RAG > memory-as-a-tool pattern > agent-to-agent (A2A) protocol > session compression strategies > memory for self-improvement i hope they keep up the streak. you can read it here for free: kaggle.com/whitepaper-con…

keyboard_arrow_left Previous keyboard_arrow_right Next

Wild. By far the most complete Claude Skills repo yet 🤯 @composio’s Awesome-Claude-Skills packs 100`s of ready-to-use workflows: ↳ PDF tools, changelog generation ↳ Playwright automation ↳ AWS/CDK tools, MCP builders ... and much more! Free and open-source. Repo in 🧵↓

🚨 Carnegie Mellon just dropped one of the most important AI agent papers of the year. It’s called “Training Proactive and Personalized LLM Agents.” Here’s the wild part... they didn’t train agents to just complete tasks. They trained them to talk better. Most AI agents are task junkies: they execute, they don’t interact. These new ones do three things simultaneously: → Productivity – actually finish the job → Proactivity – ask smart clarifying questions → Personalization – adapt tone, style, and behavior to you They built a full interactive world called UserVille, filled with simulated users each with unique personalities and quirks (like users who only reply in JSON, or only answer A/B/C questions 🤯). Then they trained agents using a new RL framework called PPP (Productive, Proactive, Personalized). Results? +21.6% higher performance than GPT-5 across complex engineering & research tasks. Agents started asking fewer, sharper questions and mirroring user preferences automatically. This is the future: Not just agents that do things but agents that understand who they’re doing them for. Paper: arxiv. org/abs/2511.02208v1

KV cache compression techniques ▪️KV caching (basic) – stores previously computed Keys and Values in memory and calculates attention only for new tokens. ▪️ Quantization – represents KV cache with fewer bits. ▪️ Low-rank decomposition – compresses the KV cache into smaller spaces. ▪️ Slim Attention – stores only Keys and recovers Values from them using math tricks. ▪️ XQuant – quantizes and stores only the layer input activations (X), and recalculates Keys and Values from X on the fly during inference. Read about XQuant method (the newest one) and other methods with their limitations in this overview: turingpost.com/p/xquant

I’ve been beta testing @GoogleAI Co-Scientist for some time. It generates novel ideas, hypotheses & scientific proposals. It’s unbelievably good; together with the other AI models I discuss here, it fundamentally changes how science is done & will make the world a better place!❤️

Our paper on autonomous scientific research is accepted to Findings of #EMNLP2025! 🎉 We introduce Agent Laboratory, a framework that accelerates scientific discovery by teaming human researchers with LLM agents.

Nice paper that is basically an empirical survey of all the recent RL algorithm improvements.

Day 3 of replacing my Google Home with Claude Code Giving Claude a voice with an 11 Labs MCP server! Code walk through 👇

@adxtya_jha Or you could’ve paid for a better model

It’s true

@shadcn I was just thinking of this earlier, one of my repos looks like this and it’s getting annoying to look at

i'm going to say something that most of you won't understand but now, with the new crop of LLMs, coding is mostly solved, and its all systems engineering. and here is something i learned from being a distributed systems wagie for a long time: make your systems parametric