Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup
B · Useful, limited A candid deployment retrospective from a small software firm (Grayhat Developers), accepted as an IAAI-27 poster; the savings are operational estimates with no baseline, by the authors' own account.
Summary based on the full text·Original: about 12 minutes to read
In short
An arXiv paper (5 November 2026) from Grayhat Developers, a software company of under 40 people, on adding self-hosted models next to Claude and Codex. Local models could not replace commercial coding agents: too slow or too weak on repository-level work. They were useful for coordination: pulling context from Outline, Plane and ERPNext into files the coding agents read, and turning agent output into tickets. The first internal rollout failed on setup friction, not model quality.
Key numbers
- $325monthly commercial AI spend (Claude and Codex seats plus credits) before local models
- 25.93B+recorded tokens across those services
- 30-40%estimated share of commercial AI workload spent on coordination: context transfer, docs and ticket sync
- 27-36%estimated commercial AI workload displaced by local models, about $88-117 a month
- 8 of 28version 1 feedback comments about setup and onboarding, the largest category
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Before: seven Claude Team/Claude Code seats (about $175 a month), seven Codex subscriptions (about $140) and about $10 of credits, about $325 a month for over 25.93 billion recorded tokens. Five Claude Pro seats could run out mid-billing period.
- About 20% of a usage window went to documentation sync and a further 10-15% to task and project sync, so 30-40% of workload was coordination.
- Kimi and DeepSeek on consumer hardware were frustrating replacements for complex coding.
- Hybrid set-up: a Gemma model on llama.cpp (RTX 4080 16 GB machine) connected through self-hosted MCP servers to Outline, Plane and ERPNext.
- Version 1 (LobeHub, user-configured) was not adopted; 11 interviews produced 28 comments, 8 on setup and onboarding.
- Version 2 (centrally configured LibreChat with pre-set MCP connectors) improved feedback and use; setup complaints fell from 8 to 3.
- Estimated about 90% of coordination work handled locally, displacing 27-36% of commercial AI workload (about $88-117 a month) against $10-30 a month local running cost.
How they got it
A first-person retrospective organised around the Kübler-Ross stages. Usage shares come from repeated workflow observation, not token measurement. Feedback counts mix structured interviews (v1) and unstructured Discord feedback (v2).
Limits
- No formal baseline; several changes happened at once; the authors call the cost figures experience, not evidence.
- Tiny absolute amounts: the savings are about $100 a month, before engineering time and electricity.
- The local/commercial split is manual, not an automated router.
Why it matters for companies
The split of work is the useful part: frontier agents for hard coding, small local models for context shuffling between internal tools, which is also where internal data is most sensitive. The failure of version 1 is a familiar lesson for any internal AI platform: if staff must configure connectors and authentication themselves, they will not use it. Central configuration mattered more than the model.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- AI enablement, Self hosting, Cost