RunAnywhere (YC W26) @RunAnywhereAI

@PrismML's Ternary Bonsai 2 dropped this week. 27B parameters in about 6 GB. • Ternary weights, 1.72 bits each • Built on Qwen3.8-27B • 98% of full precision quality (PrismML evals) • A 4-bit quant 3x the size scores about the same Its model card lists CUDA, Metal and CPU. No NPU. So we ported the model to run on the @Qualcomm's Hexagon NPU through QHexRT, our inference runtime. Same runtime that already runs the first Bonsai family on Android. In the video, the same model running three ways: • Hexagon NPU through QHexRT • Android CPU through @ggml_org's llama.cpp • MacBook Pro M4 through @Apple's MLX NPU build isn't public yet. Shipping soon.

2.5B model, offline, solving a puzzle it first called impossible. MiniCPM5 2B from @OpenBMB. 128K context, top open model under 4B when it landed. Searched Hugging Face inside the RunAnywhere app, downloaded, ran.

Apple killing it with on-device this launch. When we started with @RunAnywhereAI , a lot of people questioned the idea. Aren't models on the phone just a gimmick? But the hardware is getting there, the models are getting there, and we're heads down building towards making it easy for anyone to actually run those models on that hardware, on whatever chip they happen to have. Models living where your data already does is finally becoming a reality.

NeoHorse 1 4B dropped this week. It was running on my phone the same day. 4B params that plan, call tools, and finish the job. It trained on agent runs, not chat logs. Thinks first, answers second. Airplane mode the whole time. RunAnywhere has Hugging Face search built into the app. Type the model name, pick your quant, download, run. Day one, every time.

AI agents are not just for big models in the cloud. This is LFM 2.5, a 2.6B model, running a full agentic loop on a 3 year old iPhone 15. airplane mode the whole time. reads my calendar, reasons over the gaps, books the slot. remembers what i tell it, survives a force quit. funny part: Apple Intelligence will never ship on this iPhone. the chip is one generation too old. the model does not care.

A 20 billion parameter mixture of experts model, running on a phone's @Qualcomm Hexagon NPU. Qualcomm said that next year's Snapdragon NPU will run MoE models up to 30B. This is Deepgrove's Maple Preview, 20B parameters, 256 experts, on this year's chip, in the RunAnywhere app. How a 20B model fits in a phone: every token wakes 8 of the 256 experts, about 1B parameters of work, while the other 19B sit in memory waiting their turn. Ternary weights get the whole thing to 6 GB on disk. The routing, the experts, attention and the output head all run on the NPU. The silicon has been able to do this for a while. What was missing is a runtime that puts a model like this on it. That is QHexRT, and it ships inside the RunAnywhere SDK.

Nobody knows what models their phone can actually run. Everyone asks "which model should i use" and the honest answer is: depends on your device, and nobody has tested it on yours So we built a benchmark suite to test any model on your own device. Ran a quick benchmark on my iPhone 15 here are the results btw: LFM2.5 230M on ANE: 100 tok/s short, held 65 tok/s at 512 tokens LFM2.5 350M on ANE: 88 short, 48 long LFM2.5 1.2B on the GPU: 13.6 short, 11 long Granite 4.1 3B on the CPU: 13.1 short... 4.6 long that granite line is the whole point. it started fine and fell 65% as the phone heated up. you would never know that from a 5 second demo. the neural engine rows didn't move between 256 and 512. stop guessing. benchmark. pick the one that fits your phone. apps.apple.com/us/app/runanyw…

@deepseek_ai's DeepSeek R1, @Microsoft's Phi MoE, @nvidia's Nemotron, @LiquidAI's LFM2.5 and Qwen3.8 27B, all running on a laptop's @Qualcomm Hexagon NPU. None of them were built for it. 57 more across language, speech, vision and embeddings, all ready to drop into your app. Why the NPU: on our bench rig it reads a 365 token prompt in 67 ms vs 958 ms on CPU. Over 60 back to back passes it keeps 97% of its speed while the CPU keeps 69% and loses half its clock. Ten minutes in, the CPU has halved. The NPU has barely moved, at a third of the energy. The CPU is fine for a demo. The NPU is for hours of work. All of it ships in the RunAnywhere SDK. One API, and your app gets the NPU path on Snapdragon without touching a kernel. github.com/RunanywhereAI/…

Qwen3.8 27B, working through multi-step problems on a laptop NPU. Every matrix multiply runs on the @Qualcomm Hexagon. Nothing about this model was designed for an NPU. It is the stock quant of an ordinary frontier model. Everything that makes it run here is on the runtime side, and most of that is kernels we wrote ourselves. People ask why we put this much effort into NPU ops when the CPU already runs the model. Because the CPU runs it for a demo, and the NPU runs it for hours. Here is what we measured on our bench phone (Snapdragon 8 Elite Gen 5, same Hexagon v81 generation as this laptop), NPU against CPU, same base model on both sides: - Reading a 365 token prompt: 67 ms to first token vs 958 ms. About 14x. - 60 passes back to back: the NPU kept 97 percent of its throughput. The CPU kept 69 percent and its clock fell from 4.4 GHz to 2.4. - Energy on that run: 65 mJ per token vs 251. Just under 4x. - Ten minutes of summarising an inbox: the CPU dropped from 50 messages a minute to 27. The NPU went 124 to 112, at a third of the energy per message. The CPU benchmarks fine for thirty seconds and then falls off. The NPU throttles far less and keeps going on long runs. That is what the silicon was built for, and it is why a 27B model belongs on it. The catch is that NPUs want models built for them: one numeric format, purpose trained, a bespoke graph per model. A stock quant is heterogeneous on purpose, different tensors at different widths, none of them chosen with HVX in mind. So we write the ops. Every matmul stays on the NPU, with no quiet fallback to the CPU for the awkward layers. QHexRT is that runtime, and it ships inside the @RunAnywhereAI SDKs. An app on Android or Windows on Snapdragon gets this path without touching a kernel. Same laptop, same chip, same weights. The difference is the runtime underneath.

@samact_coded that's why we have a shared cpp native core!

Been posting individual pieces of RunAnywhere for months. Here's the whole thing in one place. RunAnywhere SDKs is a complete inference suite for any dev who wants to run AI on the user's device. ->8 SDKs: Swift, Kotlin, Flutter, React Native, Web, Electron, Python, and rcli for the terminal. Same API on every one. ->Every platform: iOS, Android, macOS, Windows, Linux, and the browser. ->Every modality: LLM, vision, speech to text, text to speech, embeddings, RAG, structured output, tool calling, and a full voice pipeline. ->Multiple backends under one API: llama.cpp, MLX, sherpa, ONNX Runtime, Core ML. The highest priority engine that fits the device wins, your app never branches. ->Our own runtimes for neural acceleration: QHexRT runs LLM, vision, speech and TTS directly on the Snapdragon Hexagon NPU. NeuRT does the same on the Apple Neural Engine, iPhone and Mac.Pick the model, the acceleration is automatic. ->A console to manage it all: deploy models to your API keys, see every registered device and its hardware, and track usage, latency and errors per modality across the fleet. One SDK. Every modality. Every device. Check it out here: github.com/RunanywhereAI/…

OpenAI and Anthropic have been recently showing off their computer use. But honestly, I don't feel comfortable with cloud models streaming screenshots of my PC back and forth to click desktop buttons. So I built one of my own over @RunAnywhereAI's RCLI running completely locally, sending nothing to the cloud. Here is a quick clip of it performing a quick QA of the desktop app I vibe coded for this demo.

@aronchick Yes, via our CLI tooling: github.com/RunanywhereAI/…, it has a server support similar to @ollama . ANE support is limited but would love any feedback :)

Running inference end to end on the Apple Neural Engine, on an iPhone 15. The Neural Engine is the cheapest place on an Apple chip to spend a token. We ran Qwen3-0.6B on an M4 Max, once on the Neural Engine and once on the GPU, same 300 tokens. On the Neural Engine it drew 7.8x less power, 1.4 W vs 11.4 W, and used 6.7x less energy per token, 20.7 mJ vs 138. That is battery and heat. Every op of this model runs on the Neural Engine, 717 of 717. Zero fall back to CPU or GPU. We verify that per op before a bundle ships, because Core ML will silently run your model on the CPU and never tell you. The runtime is NeuRT, and it is inside the RunAnywhere SDK right now. Pick an ANE model and it runs on NeuRT, no engine code, no Core ML plumbing. The model in the video is LFM2.5 350M, in the RunAnywhere app: apps.apple.com/us/app/runanyw…

Perplexity open-sourced the engine behind this yesterday. It is called Lily and it is worth reading. It runs one model, Qwen3.6-35B-A3B, on one chip generation, M5, with greedy decoding only. No streaming, no sampling, no tools. That narrowness is the whole trick: the kernels are tuned to one fixed shape, and it buys about 1.3x over MLX on decode. We built the opposite thing. One SDK, every device. The exact same model file runs on any Apple silicon Mac since 2020, alongside hundreds of other models, with streaming, tools, vision and voice. Good engineering either way. Happy to see more of it in the open.

Perplexity @perplexity_ai

3 weeks ago

Introducing hybrid compute in Perplexity Computer. Computer can start a task in the cloud, then move to a local model that runs on your Mac. Use it for steps involving private files or sensitive data. Available today in the Perplexity Mac app.

103 124 1K 286K 394

@Mirabuildz Give it a shot: IOS app: apps.apple.com/in/app/runanyw… Mac App: github.com/RunanywhereAI/…

Running inference on a Mac and streaming it directly to an iPhone over the local network. You can run models bigger than what your phone can hold, without sending a single bit to the cloud, at negligible latency. One click plug and play using the RunAnywhere apps.

@TonyZ278264 Both chips are running the same model LFM 2.5 350M

We pitted the NPU against the CPU on a real workload Same phone, Same model. 10 minutes straight. The Job: read emails and write a one line summary continuously in the bg, while someone scrolls through the inbox. Summaries in 10 minutes: NPU : 1,159 CPU : 297 That is a 3.9x difference! The cpu starts at 50 emails/minute. By minute 5 it throttles and drops to 27 and stays there.That is a 46% loss in throughput in the first 5 minutes. The NPU drops 10% and holds. So while models on a CPU may benchmark gloriously, running AI on device in the real world needs silicon that does not die in the first 5 minutes of use. Build on device AI for the real world: github.com/RunanywhereAI/…