Strata: Qwen 3.8 Flash Next (125B MoE) on consumer GPUs
C · Weak signal Open-source project README with self-measured speed figures on two machines; useful signal on local inference but no independent benchmark or quality evaluation.
Summary based on the full text·Original: about 8 minutes to read
In short
Strata is an MIT-licensed open-source project on GitHub (user Niko1221) that runs Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model from the Qwen team, on a gaming PC with a 12 GB or larger NVIDIA or AMD card. It splits the model across graphics card, system RAM and SSD and exposes OpenAI- and Anthropic-compatible local APIs. The README reports usable chat and coding speeds on two ordinary consumer machines, with nothing leaving the PC.
Key numbers
- 94 tokens/sanswer speed of the Q2_0 size on an RTX 5070 (12 GB) with 64 GB RAM
- 24,576experts in the model, of which 10 are used per token
- 1.6-1.8xspeed-up from a small draft model guessing tokens for the large model to check
- 91%of the full model's SWE-bench Verified score kept by the Coder variant with half the experts removed (authors' figure)
As reported by the source.
English·Web page·opens github.com
What it reports
- On an RTX 5070 (12 GB), Ryzen 5 7600, 64 GB RAM: 94 tokens/s answer speed and 2,650 tokens/s prompt reading at Q2_0; 53 tokens/s and 1,620 tokens/s at IQ3_S.
- On an RX 9070 XT (16 GB), Ryzen 9 3900X, 47 GB RAM: 60 tokens/s and 1,160 tokens/s at Q2_0.
- Requirements: 12 GB VRAM or more, 32 GB RAM or more (64 GB runs every size), about 80 GB disk; the model download is about 70 GB.
- The model has 24,576 experts and uses 10 per token; the most-used experts sit on the GPU, all of them in RAM, a lookup table on the SSD.
- Speculative decoding (a small helper guesses, the large model checks) gives answers 1.6-1.8x sooner.
- The Coder variant drops half the experts, fits 32 GB RAM and keeps 91% of the full model's SWE-bench Verified score, as measured by its authors; it is weaker outside code.
- First load takes 1-3 minutes and puts 35-55 GB into RAM; a 30,000-token first prompt takes about 1 minute.
How they got it
Speeds were measured by the project on two named PCs with engine versions 0.1.26 and 0.1.36, using 4K-token answers and 32K-token prompts. Quality figures for compressed variants come from the compressors (ISTA-DASLab, UkisAI, Unsloth), not from Strata.
Limits
- No quality comparison between heavily compressed sizes (2-3 bit) and the full model, apart from the Coder figure.
- Speeds are single-request, self-reported on the project's own machines.
- Hobby project with donation funding; enterprise support, security review and licence terms of each model need separate checks.
Why it matters for companies
A model of this size usually needs a server; running it on a desktop with 64 GB RAM changes the cost and data-residency picture for pilots where data cannot leave the building. Compatibility with existing agent tools means teams can test local models without rewriting integrations. Output quality at 2-3 bit compression is unproven here, so a local setup should be tested on the company's own tasks before replacing hosted models.
Summary based on the full text. Read the original for the full argument and data.Read the original on github.com