---
id: "lib-2026-10-02-knowledge-or-calculator-decomposing-the-skill"
type: "library"
title: "Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-02"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-02-knowledge-or-calculator-decomposing-the-skill/"
original_url: "https://arxiv.org/abs/2610.03564"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Controlled benchmark study with 17,820 episodes across 9 models and deterministic verifiers, cross-checked on a second harness, though from a small academic and independent team."
summary_basis: "full-text"
original_minutes: "40"
tags: ["ai technology", "enablement", "agent skills", "finance"]
---

# Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows

Tier A (Strong evidence): Controlled benchmark study with 17,820 episodes across 9 models and deterministic verifiers, cross-checked on a second harness, though from a small academic and independent team.

Summary based on the full text.

## In short

An arXiv preprint (2 October 2026) by Jermyn Zhen Yong Bek, Zhuang Qiang Bok and Zhongtian Sun (Independent, Deep Insight Labs, University of Kent and Cambridge). It introduces FinSkillBench, a benchmark of verifiable investment-management tasks, to test whether "skills" (packaged procedures and tools) improve AI agents. Curated, human-built skill packages raise scores substantially; skills the agent writes for itself do almost nothing. Most of the gain comes from executable tools rather than written instructions.

## Key numbers

- +16.2 points: mean score gain from curated skill packages (0.366 to 0.528)
- +0.5 points: gain from skills the agent wrote for itself within the episode
- +19.5 vs +5.6: points added by executable tools alone versus procedural documents alone
- 0 of 33: mandate-adherent portfolios when the agent ran a nine-stage chain on its own handoffs

## What it reports

- FinSkillBench: 2,603 point-in-time episodes across 12 subtasks in portfolio construction, risk management and fundamental analysis; 17,820 episodes run across 9 models and 3 conditions.
- Curated skills: mean 0.366 to 0.528 (+0.162, 95% CI +0.152 to +0.171); self-generated skills: +0.005.
- Documents alone add +5.6 points, tools alone +19.5 points; together they are subadditive.
- Gains are largest in portfolio construction (+0.278) and risk management (+0.218), smallest in fundamental analysis (+0.054).
- Self-generated skills raise median tokens per episode from 19,952 to 40,034 and median latency from 59.8 s to 76.8 s.
- 3,736 episodes (21.0%) hit the turn budget; Phi-4 scored 0.000 on all 1,980 episodes.
- A scheduling bug caused 61% of gpt-5.4's curated script runs to time out; repaired, its curated mean rises from 0.520 to 0.680.
- In a nine-stage chained workflow, the agent produced near-optimal books (optimality 0.934) that were never mandate-adherent (0 of 33), while every per-stage check reported success.

## How they got it

Tasks use a 30-stock US equity universe with hidden, regenerable ground truth and deterministic verifiers. Agents ran with no skills, curated skills, self-written skills, documents only or tools only, at temperature 0.7 with up to 12 turns and one trajectory per cell. Results were checked across 10 scoring variants, bootstrap analyses and a second, independent agent harness.

## Limits

- Curated packages were built for this benchmark, so some gain reflects alignment with its verifiers.
- One trajectory per cell; the chained-workflow result is a single-model, single-mandate case study.
- Effect sizes depend on how the harness exposes tools and data.

## Why it matters for companies

Asking an agent to "write its own procedure" costs twice the tokens and adds little. The value sits in tested tools and precise data-access guidance that someone in the business builds and maintains. The chained-workflow result is the warning for operations: each step can pass its own check while the end result breaks the original requirement. Controls should check outputs against the source requirement, not against the agent's own reading of it.

Original: https://arxiv.org/abs/2610.03564