What Did the AI Take On? Characterizing Cognitive Delegation in LLM Reasoning
A · Strong evidence KAIST HCI study led by Juho Kim's group with 992 step-level annotations and three taxonomies; careful method, but 24 young frequent users and one model.
Summary based on the full text·Original: about 70 minutes to read
In short
Yoonsu Kim, Sean Kim, Kihoon Son, Saelyne Yang and Juho Kim (KAIST, Carnegie Mellon) looked inside reasoning-model responses step by step and asked users which steps they had actually requested. Users judged about half of all reasoning steps as work the AI took on by itself, and that share grows as a response unfolds. Most AI initiative was welcome, but users wanted to be consulted on critical, unrequested decisions and wanted less effort on minor, repetitive ones. The authors propose interfaces that mix delegation protocols within one response and make AI-initiated decisions inspectable and revisable.
Key numbers
- 48.6%share of LLM reasoning steps users judged as AI-initiated, work they had not asked for
- 31.1% to 61.4%AI-initiated share from the first to the final fifth of a response
- 63.9%steps users were happy to fully delegate without notice
- 82%share of 'overreaching' steps where users wanted to be consulted before the AI acted
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 24 frequent LLM users (12 female, 12 male, mean age 25.75) produced 992 retrospective annotations of reasoning steps and tool actions.
- Three tasks, eight participants each: pitch slides (Creation), an energy explainer (Information), a cover-letter revision (Advice).
- 48.6% of steps were seen as AI-initiated, rising from 31.1% in the first fifth of a turn to 61.4% in the last fifth.
- 62.7% of AI-initiated steps contributed as participants wanted.
- Unwanted and minor steps were more often AI-initiated than wanted and critical steps (70.0% vs 41.1%; 75.8% vs 44.4%).
- Users preferred full delegation for 63.9% of steps. Of the 358 steps (36.1%) needing involvement: 63% consultation before acting, 29% adjusting how much the AI did, 8% notification afterwards.
- "Generating" steps drew the most demand for involvement (62%, mostly consultation, 55%); "Repeating" 56%, mostly "do less" (44%); Understanding 15%, Planning 21%, Recalling 26%.
- Even among satisfactory critical AI-initiated steps, 28% still called for involvement; 23 of 29 requests for after-the-fact disclosure came from these.
How they got it
Participants did a 20-minute task with gemini-3.5-flash plus web search and a PowerPoint tool via MCP, then spent 90 minutes reviewing the logged reasoning trace step by step, labelling who initiated each step, how critical it was, whether it matched intent and how they would have wanted to be involved. Interviews followed. Taxonomies were built inductively from the annotations.
Limits
- Small, young, university-recruited sample; one task per category, one model, one interface.
- Judgements were retrospective, after seeing the output, so hindsight bias is possible; stated preferences may change when interruptions carry real cost.
- Design sketches are proposals, not tested interfaces.
Why it matters for companies
When staff hand a task to an agent, many consequential choices get made silently in the middle of the run. This study gives a vocabulary for which choices people want flagged: new content generation and unrequested critical decisions. Blanket "human in the loop" approval on every step would be rejected as burdensome; selective consultation is closer to what users want. Tool selection and internal agent design can use this as a checklist.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org