---
id: "lib-2026-10-17-when-code-becomes-an-llm-call-an-empirical-study"
type: "library"
title: "When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems"
lang: "EN"
org: "arXiv (Nanjing U., XJTU, SMU, UCL)"
source_type: "Paper"
date: "2026-10-17"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-17-when-code-becomes-an-llm-call-an-empirical-study/"
original_url: "https://arxiv.org/abs/2610.24545"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Software engineering researchers from Nanjing, Xi'an Jiaotong, Singapore Management University and UCL audit 496 model call sites in 40 open-source agent systems with a pipeline validated by 17,428 human judgments and runtime trigger tests; a strong empirical study, limited to open-source code at one commit each."
summary_basis: "full-text"
original_minutes: "45"
tags: ["ai technology", "agent engineering", "reliability", "code quality"]
---

# When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems

Tier A (Strong evidence): Software engineering researchers from Nanjing, Xi'an Jiaotong, Singapore Management University and UCL audit 496 model call sites in 40 open-source agent systems with a pipeline validated by 17,428 human judgments and runtime trigger tests; a strong empirical study, limited to open-source code at one commit each.

Summary based on the full text.

## In short

An arXiv paper (17 October 2026) by Shihao Weng, Longtian Wang, Yang Feng, Federica Sarro, Xiaofei Xie and colleagues. Agent systems replace pieces of program logic (choosing a tool, judging whether an action is allowed, writing code to run) with calls to a language model. The model's behaviour is not fixed and can be steered by its input, so the guarantees must come from checks around each call. The authors compared what each call site should enforce with what the code actually enforces across 40 popular open-source agent systems. Most call sites have checks; almost none have adequate ones.

## Key numbers

- 48%: share of model outputs that reach control flow or code execution; 73% of those never reach a person
- 82% vs 5.3%: call sites with checks before and after the model call vs call sites that satisfy their whole expected contract
- 67%: prompt constraints that are only requested from the model, not enforced in code
- 63%: expected constraints left unmet across the call sites
- 38: gap mechanisms shown to trigger at runtime, all stopped by a small code fix

## What it reports

- 496 call sites in 40 systems (13.4 million lines, four languages, six domains).
- 48% of model outputs reach control flow or execution, and 73% of those never pass a person.
- Programs enforce 78% of their constraints in code, but in prompts 67% of constraints are only requested from the model.
- 82% of call sites are checked before and after the call, yet 63% of expected constraints are unmet and only 5.3% of call sites satisfy their whole contract. Ten recurring patterns explain 99% of the gaps.
- 38 gap mechanisms were triggered at runtime; a small code fix enforcing the expected constraint stopped all 38.
- Example: one coding agent runs any model reply containing a "repl" code fence as Python with no approval or sandbox, though its documentation calls the feature sandboxed.
- Every system had gaps, and gap rate did not depend on system size.

## How they got it

A three-stage multi-agent pipeline: locate call sites and trace where each output flows; derive an "expected contract" from a code-free functional specification by an agent that never sees the code, and an "implemented contract" from the code by an agent that never sees the expected one; then compare and test. Every step was reviewed by people (17,428 human judgments; agreement AC1 0.91 or higher), and gaps were confirmed with runtime trigger tests.

## Limits

- Expected contracts are a normative reconstruction; other analysts might admit slightly different constraints, though code-blind review and leakage checks are reported.
- Open-source systems at one commit each; closed-source agents and frameworks may differ.
- Trigger tests used one substitute model and up to ten framings, showing a mechanism can fire, not how often it does in practice.
- The 290 exploitable-output count is a lower bound.

## Why it matters for companies

This is the clearest evidence so far that "we have guardrails" says little about whether they work. Anything a model decides that feeds execution or control flow needs enforcement in code (allow-lists of tools, schema checks, approval before execution), not instructions in the prompt. Security and engineering reviews of agent products, bought or built, should inventory each model call site, ask what it is allowed to trigger, and check that the constraint is enforced after the reply, not merely requested before it.

Original: https://arxiv.org/abs/2610.24545