LocalAI @LocalAI_API

I added kimodo.cpp animation gen & playback in latent3d.space. So you can describe a motion, then it'll generate it using a @LocalAI_API instance and play it back on the physically simulated character. Of course there are issues with finding animations that are physically possible, but that is part of the fun... maybe.

lnkd.in/p/g-T6EZXM Lucy's coming together, 100+ setting configurations, multiple subsystems, 4 different operator modes, and cloud integration is still safe. Norton and Microsoft security systems were not bypassed. @LocalAI_API @GeoStl @WashUengineers

🚨🚨🚨BRAKING NEWS!!!🚨🚨🚨 I recorded how alien artificial super intelligent agents could communicate just using speakers and microphones!

You can make any LLM into a Jev style classifier using @LocalAI_API . In our implementation you can use hybrid inference as well to do small chunks using autoregressive generation. This allows quickly choosing a function and filling in a parameter. One disclaimer though, calculating confidence scores is not well defined on a regular LLM, or at least I didn't find a way to do it. Practically speaking it chooses "I don't know" fairly reliably though. I'm not sure what Jev does on that front.

Richard Palethorpe @jichiep

2 months ago

Can you have a voice assistant run on CPU? Yes, even on Rpi5, but with a catch. parakeet.cpp for transcription and Piper for TTS are so fast that part is easy. The LLM is harder because autoregressive generation is slow, but prefill is pretty fast and we only (sort of) need

1 2 12 3K 12

kimodo.cpp got between a 1.5 and 10x speedup depending on quantization level. Also it's soon coming to @LocalAI_API

@LocalAI_API cute

Qwen3.8 on @LocalAI_API vibe coded firmware for talking to it via an ESP32.

@gr3yg00 😍 This is so cool!

My first from scratch entirely with @LocalAI_API and qwen3.8-27b agents built app. It provides the agents a way to upload ESP32 binaries so I can easily flash them to hardware and test changes made by the agents.

@gr3yg00 we are never down!

I can't stress enough how well Qwen3.8-27b works for me locally as an agent in @LocalAI_API . Trying out the ResearchAgent from agenthub.localai.io atm and it performs amazingly. Gonna try giving it some more tools...

@SaintBobby303 I'm using Opus and Sonnet for work. Privately I run local models with @LocalAI_API. Had hardware issues recently, but server is up and running again. - imported agenthub.localai.io/#agent=CodingA… as agent - configured qwen3.8-27b as its model - gave it gh key and a repo Works great :)

The support for @MiniMax_AI H3 improved in vllm.cpp. The audio now renders much better. See the new version!

Ettore Di Giacinto @mudler_it

2 months ago

vllm.cpp runs @MiniMax_AI 's MiniMax-H3 now. 33.1B, video and audio out of a single model, and I'm still a bit stunned that it works we reimplemented both VAEs from the checkpoint's remote python, so there's no torch anywhere in the process. And you drive the whole thing over

6 4 38 6K 15

Rocm support on vllm.cpp is improving every day. Mike is really pushing it!

Mike @anothervariable

a month ago

These pre-fill speeds on vllm.cpp rocm/hip on R9700s are starting to look gouda 😁 And there is still more juice to squeeze with more kernel tuning, that number will hit 3k tok_s :) at 41k context. The goal is trying to squeeze as much out as we can up to 250k context. The

0 0 10 2K 1

this week was really crazy in AI. And I have only a DGX box for this, not sure how long it will last. What I'm doing in parallel for vllm.cpp: - Qwen 3.8 27b benchmarks (I'd like to take some numbers to show you) - Qwen 3.8 2.4t fitting on a single Nvidia DGX box (yes, we will have it, even if slow) - @Lightricks LTX2.5 support - Index-tts support - @MiniMax_AI Music 3 - @NVIDIAAI 's new Nemotron release - Dspark just landed - dots3-preview support in vllm.cpp And since I'm having issues in overbooking the box with multiple agents, I'm doing a resource-controller api in parallel to control this..

a week ago I shared what I've been working on, vllm.cpp, vLLM's serving stack in C++ with no python in it. I got really a lot of feedback, and it seems I'm not the only one who wanted this to exist. We reached 800+ commits, 280 stars in just so little time. but where is it going? I got a lot of requests for what to support and what not to, so I took some time to write down why vllm.cpp exists, what it will be, and how you can contribute and it's already not just me. contributors showed up with hardware I don't own and did what I could not do because I don't have the hardware: - @DevAutomata got the @AMD backend going - @anothervariable got gemma-4-26B already running on two R9700s - @lu_zero_ did the whole Tenstorrent bring-up (!) - @jichiep has been looking at the mamba kernels - @filipsajdak got it building on Jetson Thor We are building a high-performance inference stack that runs everywhere and that you can embed in your own software. If you have hardware I do not have, that is the single most useful thing you can bring. If you want to know where are we going, how you can contribute, the vllm.cpp manifesto is in the link in the replies

I hate to post a negative result, but this is animate-any-mesh.cpp. I can't get it to produce reasonable animations even on the reference mesh (the dragon). The C++ results are identical to the Python. However there appears to be a new version on the way so I'm going to publish the code and wait. I've seen it many times in AI where the end results look terrible, but the researcher knows the probability distribution contains something great and it's worth pushing.

LocalAI 4.8 contains 3d generation for the first time via the trellis2cpp backend. There is a /3d REST endpoint and UI. Speaking of UI, this release also got a new UI! Although to be fair most releases get a new UI. There is also a remesh utility to make the models usable without resorting to a third party application because TRELLIS.2 produces weird meshes. There are many other things in this release from 20+ contributors. Including vllm.cpp that @mudler has been posting about.

vllm.cpp runs @MiniMax_AI 's MiniMax-H3 now. 33.1B, video and audio out of a single model, and I'm still a bit stunned that it works we reimplemented both VAEs from the checkpoint's remote python, so there's no torch anywhere in the process. And you drive the whole thing over /v1/videos the clip is our LocalAI llama. I handed it in as a reference image and H3 did the rest. the audio is generated too, unmute it! 4-bit, on a jetson thor. 5 seconds took 28 minutes, but the results are great!

Blog post: localai.io/blog/what-land…

Installing a model also stopped meaning "pick a quantization". One gallery entry can offer several builds of the same weights. LocalAI drops what your hardware can't run, drops what won't fit, installs the largest that's left. Q4 on the big box, IQ1 on the laptop, same command

LocalAI 4.8 is out. We just crossed 48.3k stars! - `local-ai chat` is a TUI agent harness now. - image to 3D, audio-driven talking avatars - voice cloning first-class in the UI - vLLM's serving architecture in C++20, no Python 386 PRs, 22 days, 25 contributors, 11 new👇