---
id: "lib-2026-10-16-we-fine-tuned-a-235b-model-on-4-gpus-then-we-didn"
type: "library"
title: "We fine-tuned a 235B model on 4 GPUs. Then we didn't need it"
lang: "EN"
org: "Vertexcover"
source_type: "Practitioner post"
date: "2026-10-16"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-16-we-fine-tuned-a-235b-model-on-4-gpus-then-we-didn/"
original_url: "https://blog.vertexcover.io/fine-tuning-235b-moe-four-gpus"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "Vertexcover engineering post with per-tenant F1 tables, linked bug reports and a candid account of not shipping; useful practice, but one task and self-reported results."
summary_basis: "full-text"
original_minutes: "8"
tags: ["fine tuning", "open models", "inference cost", "deployment"]
---

# We fine-tuned a 235B model on 4 GPUs. Then we didn't need it

Tier B (Useful, limited): Vertexcover engineering post with per-tenant F1 tables, linked bug reports and a candid account of not shipping; useful practice, but one task and self-reported results.

Summary based on the full text.

## In short

An engineering post (16 October 2026) from Vertexcover. To cut the cost of fine-tuned OpenAI and Gemini models for document entity classification, the team fine-tuned the open Qwen3-235B-A22B mixture-of-experts model on one machine with four H100s. Accuracy improved for every tenant and beat the API models at about a third of the per-token cost. They never deployed it: serving a 235B thinking model cheaply enough was a large project, and newer open models solved most workloads without fine-tuning.

## Key numbers

- 75.47 to 83.10: average F1 across nine tenants, base vs fine-tuned Qwen3-235B-A22B
- 26 hours: one epoch on about 3,600 examples on four H100 GPUs
- about a third: per-token cost of the fine-tuned open model relative to the fine-tuned API models it replaced
- 4 of 5: training setups that failed before a single training step ran

## What it reports

- Task: find entities in long structured documents and label them from a fixed set; scored per tenant with F1.
- Base vs fine-tuned F1 across nine tenants, from 58.17 to 67.77 (tenant I) up to 72.88 to 84.84 (tenant F); average 75.47 to 83.10, +7.63.
- The model is 470 GB in 16-bit and about 120 GB in 4-bit; the machine had 320 GB of GPU memory and 960 GB of system RAM.
- Failures: Axolotl ran out of memory both ways; Unsloth hit a CPU-tensor bug in a GPU kernel (since fixed); Answer.AI's FSDP+QLoRA could not start on any Transformers version that supports Qwen3-MoE; LlamaFactory with FSDP loaded four full copies into RAM because Transformers silently disables low-memory loading when quantising.
- What worked: LlamaFactory with DeepSpeed ZeRO-3 and CPU offload, holding the 4-bit base in system RAM and streaming layers to the GPUs. Slow, but the run only had to happen once.
- The author's lesson: fine-tune when the model must learn something only you have (customer formats, internal conventions); if the base model is just not good enough yet, the next release may close the gap.

## How they got it

First-hand engineering record. Open models were benchmarked on the task before choosing Qwen3-235B-A22B-Thinking; reasoning traces were kept in training data. The post links the relevant issues in Axolotl, Unsloth and LlamaFactory and the guard in Transformers' source.

## Limits

- One classification task, one epoch, about 3,600 examples; no held-out details or variance reported.
- Cost comparison is per token; serving cost, the reason it was not shipped, is not quantified.
- Dated two days before the review window; surfaced via Hacker News.

## Why it matters for companies

Fine-tuning a frontier-size open model on owned hardware is feasible and can beat paid fine-tuned APIs on accuracy and token cost, but the work sits in tooling and serving, not training. Model releases can erase the gain within weeks. Before funding a fine-tune, check whether the gap is proprietary knowledge or temporary capability, and price inference before training.

Original: https://blog.vertexcover.io/fine-tuning-235b-moe-four-gpus