Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
A · Strong evidence Tsinghua preprint pairing an analysis of real user-agent logs with a new verifiable benchmark tested on eight frontier models, with data and code released; clear method, though the benchmark has 64 tasks.
Summary based on the full text·Original: about 45 minutes to read
In short
Kaiming Liu, Yuxuan Liu and colleagues at Tsinghua University argue that agents should treat a user's instruction as evidence about what is needed, not as the full objective. Users write instructions from partial understanding, and the real requirements often sit in the code, logs and records. In real coding-agent logs, more than half of first instructions missed requirements that later proved necessary. On a new benchmark built around this gap, frontier models lost 53 to 67 points of success when given the partial instruction instead of the full spec.
Key numbers
- 56%first instructions in 100 SWE-chat trajectories that omit requirements later needed
- 52%useful follow-up requirements that could have been inferred from the project
- 53 to 67 pointssuccess-rate loss for eight frontier models moving from full spec to partial instruction
- 16 to 34score points recovered by prompting the model to treat the instruction as evidence
- 64tasks in TiGanEval (40 software engineering, 24 research)
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 56% of first instructions in 100 SWE-chat trajectories omit requirements that later emerge; 52% of useful follow-up requirements were inferable from the project.
- Worked example: a user asks for payment retries; logs show timed-out payments were usually already charged, so the true requirement is "charge each order exactly once".
- TiGanEval: 64 tasks where requirements are fixed in advance and discoverable from code and runtime behaviour, with a simulated user who revises the instruction after seeing results.
- Eight frontier models, including GPT-5.6-Sol, lose 53 to 67 points from full spec to partial instruction; GPT-5.6-Sol loses 53.
- Feedback over four submissions does not close the gap. A plan-mode harness does not either.
- Prompting the model to treat the instruction as evidence recovers 16 to 34 points.
- Most failures are evidence-use failures: fixing the reported instance but missing the rule, never opening the deciding record, or ignoring output from their own replays.
How they got it
Manual analysis of 100 SWE-chat trajectories. Tasks were generated by TiGanTrans, which converts fully specified tasks into partial-instruction versions, then verified and revised by human annotators. Models were compared under full, partial and literal-execution conditions, with an evidence-use prompt, a plan-mode harness and budgets of up to four submissions.
Limits
- 64 tasks is a small benchmark; per-domain results rest on 40 and 24 tasks.
- The simulated user is a model, not a person.
- The SWE-chat analysis covers 100 trajectories.
Why it matters for companies
Most agent failures at work will look like this: the agent does what was asked and misses what was needed. Better prompting and planning harnesses help only partly. Teams should give agents access to the records that hold the real constraints (logs, tickets, past incidents) and review outputs against outcomes, not against the original request. Writing fuller specifications up front still pays.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org