Not sure if the measurement method used for this is actually valid or reliable enough, but for reference, Codex 6.1 Sol Standard runs at 19.9 tok/s according to a benchmark I recently saw: https://www.reddit.com/r/codex/s/b4mUszSOv1
The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster.
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
I would guess the assumption is that "agentic work" should be happening asynchronously in the background on a machine that's also used for foreground interactive work, where the slower the inference is, the longer the machine's GPU and [V]RAM are "hogged" by it at a time, and so the more the machine's performance seems to degrade.
This could be a game changer for me. The best I've been able to do so far is ~8 tok/s with Qwen 35B A3B, 32GB RAM. But then again I just got an OpenCode Go plan yesterday and have yet to figure if I can even consume my quota on DeepSeek Flash as the need to plan things well is bottlenecking me. Might be a while before I try the local thing again.
Based on how vibe-coded this whole thing is, I’m guessing the newer models weren’t in the training data and the agent just picked what it knew existed.
I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
Seems like a useful innovation on the desktop level, but on the server level, isn't it already pretty normal to see machines with 24x high-performance (i.e. individually high-bandwidth) NVMe drives in them, that can then be used in software RAID0 to achieve pretty massive read bandwidth?
There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.
What a waste of time.
Strata already does this with Qwen3.8-Flash-Next: https://github.com/Niko1221/Strata
Which seems sufficient for mortals who are not running gas town, but want a local model.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
https://hfviewer.com/compare/qwen3.6-27b-vs-qwen3.8-27b
I don’t want to bother reading anything sub-Fable, and this one looks sub-Fable.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
Maybe next you can run an LLM off of some old MFM drives