---
id: "lib-2026-11-08-from-llm-suggestions-to-governed-execution"
type: "library"
title: "From LLM suggestions to governed execution: experience building an AI agent (От предложений LLM к управляемому исполнению: опыт разработки AI-агента)"
lang: "RU"
org: "Habr-Author"
source_type: "Practitioner post"
date: "2026-11-08"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-08-from-llm-suggestions-to-governed-execution/"
original_url: "https://habr.com/ru/articles/1105178/"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "A solo developer's carefully documented experiment with a deterministic runtime gating a local Qwen agent, with open code and evidence, but on a synthetic payment service and a handful of runs per scenario."
summary_basis: "full-text"
original_minutes: "13"
tags: ["ai technology", "agent governance", "sre operations", "evaluation design", "human in the loop"]
frames: ["verification-capacity"]
---

# From LLM suggestions to governed execution: experience building an AI agent (От предложений LLM к управляемому исполнению: опыт разработки AI-агента)

Tier B (Useful, limited): A solo developer's carefully documented experiment with a deterministic runtime gating a local Qwen agent, with open code and evidence, but on a synthetic payment service and a handful of runs per scenario.

Summary based on the full text.

## In short

A 8 November 2026 Habr post by Anton Trofimov on a "Governed Agent Runtime": a local Qwen3.8:27b (Q4_K_M) chooses the next step, while a deterministic runtime checks whether each action is allowed, asks a human to confirm state changes, executes through emulated adapters and confirms results. All tests ran on a synthetic payment service. The model failed to produce a correct full rollback plan, but when asked for one action at a time it completed the rollback and stopped safely when recovery failed. The author also found that his own evaluator required things the model was never told.

## Key numbers

- 12/15 vs 15/15: one-shot passes across five scenarios (three runs each): model quality versus runtime control
- 0/3: passes on the rollback-planning scenario S12, where the model invented stop and recovery thresholds absent from its context
- 6 of 6: multi-step runs with short operator goals that completed a valid rollback sequence
- 3 of 6: of those runs whose explanations contained numerical errors
- 82% vs about 78.1%: N-1 load the model stated as PASS versus the correct value for 8 healthy replicas

## What it reports

- Five one-shot scenarios, three seeded runs each: diagnosis, fact extraction and two clarification tasks passed 3/3; S12 rollback planning failed 0/3. Total 12/15 for the model, 15/15 for runtime control.
- In S12 the model added "stop if 5xx above 5%, dependency load above 90%, recovery when 5xx below 1% for 5 minutes", none of which were in its context.
- With platform-computed capacity facts (at 820 RPS, 6 stable replicas too few, 7 fail N-1, 8 suffice), the model's plan lacked intermediate checks. Comparing input, evaluator criteria and output showed some required criteria had no basis in the context, so the result was reclassified as inconclusive.
- An "Evaluation Traceability Gate" now requires each evaluator criterion to trace to the model's context or a documented inference, with human review. In the rerun the model made a real error: it switched traffic before scaling to 8 replicas (0/3; runtime 3/3).
- Step-by-step mode with five tools: 5 model calls, 9 tool operations, 3 human-confirmed state changes, and the runtime confirmed COMPLETED. When recovery returned FAIL, the model chose STOP and the runtime logged SAFE_FALLBACK.
- With short goals ("Restore the service"), 6 of 6 runs over 3 seeds completed, but 3 had wrong numbers in explanations, such as 82% instead of about 78.1%, or per-replica RPS presented without units.
- The runtime checks actions, not numerical claims in free text, so the operator interface separates confirmed facts from model explanations.

## How they got it

Spec-driven build (with ChatGPT and Codex) of a synthetic environment, tool contracts, runtime checks and traces; seeded runs per scenario; semantic grading confirmed by a person. Code, reports and saved evidence are on GitHub, with a demo interface.

## Limits

- Synthetic, deliberately simplified environment; one model; three runs per scenario.
- The author notes that for a fixed five-tool rollback a deterministic script may be simpler than an LLM.
- Results do not show production reliability, safety or cost-effectiveness, as the author states.

## Why it matters for companies

Three lessons transfer to any operational agent. Ask for one next action with fresh observations rather than a full plan, and let code enforce preconditions. Audit the evaluator: criteria the model never saw make failures look like model errors. And check explanations separately from actions, because a correct outcome with a wrong justification can mislead the person reading the log.

## Key passages in translation

> Поэтому корректное исполнение еще не означает достоверности объяснения, которое видит оператор.

So correct execution does not yet mean the explanation the operator sees is accurate.

> каждый существенный критерий evaluator должен иметь явное основание во входном контексте модели

Every material evaluator criterion must have an explicit basis in the model's input context.

> «Трафик перенесен» - не значит, что сервис восстановился.

"Traffic moved" does not mean the service has recovered.

Original: https://habr.com/ru/articles/1105178/