---
id: "lib-2026-10-14-same-context-different-agents-a-behavioral-study"
type: "library"
title: "Same Context, Different Agents: A Behavioral Study of Semantic Layers, Orchestration and Compaction in Data Agents"
lang: "EN"
org: "arXiv (Oracle AI, USC, UPenn)"
source_type: "Paper"
date: "2026-10-14"
week: "2026-W42"
url: "https://nosov.ai/library/lib-2026-10-14-same-context-different-agents-a-behavioral-study/"
original_url: "https://arxiv.org/abs/2610.20959"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Oracle AI, USC and UPenn (Dan Roth) run a controlled test of all eight combinations of three context techniques across seven LLMs and 1,514 tasks, with behaviour traces, accuracy and cost reported per model."
summary_basis: "full-text"
original_minutes: "40"
tags: ["ai technology", "data agents", "semantic layer", "enterprise data"]
---

# Same Context, Different Agents: A Behavioral Study of Semantic Layers, Orchestration and Compaction in Data Agents

Tier A (Strong evidence): Oracle AI, USC and UPenn (Dan Roth) run a controlled test of all eight combinations of three context techniques across seven LLMs and 1,514 tasks, with behaviour traces, accuracy and cost reported per model.

Summary based on the full text.

## In short

An arXiv preprint (14 October 2026) from Oracle AI with USC and UPenn tests three context-engineering techniques for data agents: a semantic layer that describes the data, orchestration that splits work between a planner and specialists, and compaction that condenses history. Seven LLMs fall into three behaviour groups: investigators that check the data, premature answerers that answer without looking, and early stoppers that give up. Each technique helps only the group whose habit it repairs, and can hurt the strongest agents.

## Key numbers

- 1,514: long-horizon data tasks from five benchmarks used in the evaluation
- 68% to 31%: investigator accuracy on runs where a semantic layer changed their tool use
- 23 points: accuracy gain orchestration gives agents that answer before looking, at four times the cost
- 1.2 to 1.8 times: rise in dollar cost from compaction, despite smaller prompts

## What it reports

- All eight on/off combinations of the three techniques, seven LLMs from four providers, five benchmarks, 1,514 tasks.
- Semantic layer: halves give-ups for agents that stop early, but investigators treat it as the answer and stop checking; their accuracy on affected runs falls from 68% to 31%.
- Orchestration: gives premature answerers 23 points of accuracy at four times the cost; makes investigators overconfident, costing about 5 points.
- Compaction: keeps each prompt near three times the first prompt (vs fourteen times without), but total input barely changes, output grows 2.3 to 3.5 times and dollars rise 1.2 to 1.8 times. 42% of GPT-5.5's DataClaw runs reach 100+ calls with it, none without.
- Best configuration for Opus 5.5 and GPT-6 Astra is no technique at all (68.8% and 69.6% baseline); GPT-5.5 jumps from 21.4% to 58.1% with orchestration.
- Combined techniques give smaller gains and losses than their sum. Picking the wrong slice of data after inspecting it stays the most common error in every setting.

## How they got it

Each technique is switched on or off over a fixed baseline agent with the same tools, limits and prompt. The authors score accuracy, cost and benchmark process measures, then classify every run's trace into behaviours (give-ups, loops, tool-use switches). LLMs also act as annotators and judges.

## Limits

- One representative design per technique; other semantic-layer or orchestration designs may behave differently.
- Data question answering only; software engineering and maths may show other groups.
- Model behaviour changes with each release, so group membership is a snapshot.
- Oracle sells data and agent platforms; the paper does not test a commercial product.

## Why it matters for companies

Semantic layers and multi-agent setups are often sold as default upgrades for "talk to your data" projects. This paper shows the effect depends on the model in use: the same layer that rescues a weak model makes a strong one skip verification. Teams should measure the failure their agent has before adding context machinery, and check answer correctness and evidence, not prompt size or call counts. Compaction saves context window, not money.

Original: https://arxiv.org/abs/2610.20959