We fine-tuned a 235B model on 4 GPUs. Then we didn't need it
B · Useful, limited Vertexcover engineering post with per-tenant F1 tables, linked bug reports and a candid account of not shipping; useful practice, but one task and self-reported results.
Summary based on the full text·Original: about 8 minutes to read
In short
An engineering post (16 October 2026) from Vertexcover. To cut the cost of fine-tuned OpenAI and Gemini models for document entity classification, the team fine-tuned the open Qwen3-235B-A22B mixture-of-experts model on one machine with four H100s. Accuracy improved for every tenant and beat the API models at about a third of the per-token cost. They never deployed it: serving a 235B thinking model cheaply enough was a large project, and newer open models solved most workloads without fine-tuning.
Key numbers
- 75.47 to 83.10average F1 across nine tenants, base vs fine-tuned Qwen3-235B-A22B
- 26 hoursone epoch on about 3,600 examples on four H100 GPUs
- about a thirdper-token cost of the fine-tuned open model relative to the fine-tuned API models it replaced
- 4 of 5training setups that failed before a single training step ran
As reported by the source.
English·Web page·opens blog.vertexcover.io
What it reports
- Task: find entities in long structured documents and label them from a fixed set; scored per tenant with F1.
- Base vs fine-tuned F1 across nine tenants, from 58.17 to 67.77 (tenant I) up to 72.88 to 84.84 (tenant F); average 75.47 to 83.10, +7.63.
- The model is 470 GB in 16-bit and about 120 GB in 4-bit; the machine had 320 GB of GPU memory and 960 GB of system RAM.
- Failures: Axolotl ran out of memory both ways; Unsloth hit a CPU-tensor bug in a GPU kernel (since fixed); Answer.AI's FSDP+QLoRA could not start on any Transformers version that supports Qwen3-MoE; LlamaFactory with FSDP loaded four full copies into RAM because Transformers silently disables low-memory loading when quantising.
- What worked: LlamaFactory with DeepSpeed ZeRO-3 and CPU offload, holding the 4-bit base in system RAM and streaming layers to the GPUs. Slow, but the run only had to happen once.
- The author's lesson: fine-tune when the model must learn something only you have (customer formats, internal conventions); if the base model is just not good enough yet, the next release may close the gap.
How they got it
First-hand engineering record. Open models were benchmarked on the task before choosing Qwen3-235B-A22B-Thinking; reasoning traces were kept in training data. The post links the relevant issues in Axolotl, Unsloth and LlamaFactory and the guard in Transformers' source.
Limits
- One classification task, one epoch, about 3,600 examples; no held-out details or variance reported.
- Cost comparison is per token; serving cost, the reason it was not shipped, is not quantified.
- Dated two days before the review window; surfaced via Hacker News.
Why it matters for companies
Fine-tuning a frontier-size open model on owned hardware is feasible and can beat paid fine-tuned APIs on accuracy and token cost, but the work sits in tooling and serving, not training. Model releases can erase the gain within weeks. Before funding a fine-tune, check whether the gap is proprietary knowledge or temporary capability, and price inference before training.
Summary based on the full text. Read the original for the full argument and data.Read the original on blog.vertexcover.io