---
id: "lib-2026-11-05-what-the-middleman-delivers-answer-behavior-tool"
type: "library"
title: "What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays"
lang: "EN"
org: "arXiv (Wu et al.)"
source_type: "Other"
date: "2026-11-05"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-05-what-the-middleman-delivers-answer-behavior-tool/"
original_url: "https://arxiv.org/abs/2611.07650"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "A Queen's University security group audits 20 LLM API relays against official APIs over two rounds, with explicit decision rules, calibration mixtures and balance reconciliation; sample is non-random and services are anonymised."
summary_basis: "full-text"
original_minutes: "50"
tags: ["ai technology", "procurement", "billing"]
---

# What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays

Tier A (Strong evidence): A Queen's University security group audits 20 LLM API relays against official APIs over two rounds, with explicit decision rules, calibration mixtures and balance reconciliation; sample is non-random and services are anonymised.

Summary based on the full text.

## In short

An arXiv paper (5 November 2026) by Xiaodong Wu, Jianbing Ni and colleagues at Queen's University tests third-party relays that resell access to GPT-6 Astra and Claude Opus 5, often below official prices. Low-price relays were more often flagged for answering differently from the official API, four ignored instructions disabling tools, and five reported thousands of extra input tokens for identical requests. Where balances could be checked, charges matched reported usage to the cent, including the surplus tokens at one service.

## Key numbers

- 7/10 vs 2/10: low-price vs same-or-higher-price relays flagged for answer-behaviour differences in round one (p = 0.07)
- four: relays that returned tool calls despite instructions disabling them (round one)
- 4,200 to 60,000: extra input tokens reported by five relays for requests identical to official API calls
- 3,332: screened code and setup responses, with no confirmed injection
- 7/35: relay-model pairs that gained or lost a flag between rounds

## What it reports

- Sample: 20 relays (10 below official prices, 10 at or above), found via LLM recommendations, community lists and web search; requests sent from the USA.
- Answer behaviour: knowledge-boundary questions compared with batch-matched official responses; 7/10 low-price and 2/10 same-or-higher-price relays flagged in round one; the price-group difference is not significant at 5% (p = 0.07).
- Retest 11-18 days later on 35 relay-model pairs: 7 gained or lost a flag under the original rules, 6 under matched correction families; six services flagged in both rounds.
- Tool permissions: four relays returned tool calls despite explicit restrictions in round one, none in round two; calls were not executed.
- Content: no confirmed injection in 3,332 screened responses to a benign code and setup workload.
- Billing: five relays reported 4,200 to 60,000 more input tokens than the official API; at all six services with visible balances, deductions matched reported usage at listed prices, so the surplus at relay R11 entered the bill.
- In a pilot, removing an explicit refusal instruction coincided with Opus scores rising from 6/40 to 20/40, a sign of prompt sensitivity.

## How they got it

Client-side measurement through public APIs and paid accounts, with official API references collected per batch. Controlled mixtures of official and substitute responses calibrate what the accuracy test can detect. Token counts were compared on identical requests and reconciled with account balance changes.

## Limits

- Non-random sample of English-language, card-payable relays; not the whole relay market.
- An accuracy difference does not identify the model behind it; wrappers, settings or caching can cause it.
- Benign standalone requests may not trigger attacks that depend on tools, credentials or long sessions.
- Services are anonymised and full logs are private, so results cannot be fully rescored.

## Why it matters for companies

Any team buying model access through a reseller or gateway should treat the relay as a supplier to audit: compare answers against the official API periodically, enforce tool permissions in the client rather than trusting the request flag, and reconcile billed input tokens against an official count. A matching price list proves nothing about what is billed. The same checks apply to internal AI gateways that sit between staff and model providers.

Original: https://arxiv.org/abs/2611.07650