Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows
A · Strong evidence University of Minnesota NLP group releases an IRB-approved dataset of real professionals working with agents on their own problems and tests frontier models against those professionals' ratings.
Summary based on the full text·Original: about 35 minutes to read
In short
Kolab, an arXiv preprint (15 October 2026) from the University of Minnesota, records 50 professionals in 40 occupations using an agent (Claude) on their own long tasks, with their ratings of collaboration and output quality. It then asks frontier models to forecast whether people adopt, modify or reject agent output, and to judge collaboration quality from the recording. Models overestimate collaboration and cannot separate good sessions from bad ones.
Key numbers
- 115+ hoursscreen recordings of professionals collaborating with agents
- 50professionals from 40 occupations, each paid $100 plus a time bonus
- 0.492Claude's F1 for forecasting whether a professional will adopt, modify or reject its output
- 3.3 to 3.9range of Claude's predicted satisfaction ratings across the whole 1-5 human scale
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 115+ hours of video, 100 problem sets, 508 steps across 90+ software tools; most participants used Sonnet 5.5.
- Burstier workflows go with longer independent work before turning to AI; more concurrent human-agent activity goes with lower satisfaction.
- Forecasting adoption: Gemini 3.1 Pro 39.1% F1, GPT-6 51.3%; Claude reaches 0.676 when the response is observable but 0.492 when forecasting.
- Gemini 3.1 Pro labels 52% of true MODIFY and 71% of true REJECT as ADOPT. Gemini 3.8 Flash predicts ADOPT 83.8% of the time vs 65.9% actual.
- Rating prediction is flat: Gemini 3.1 Pro predicts 4.83 for a true 1 and 4.95 for a 5; Claude's predictions span only 3.3 to 3.9. GPT-6 separates low and high ratings best.
- Longer workflow context gives little or no consistent improvement.
How they got it
Participants recruited via word of mouth, Prolific and Upwork did two one-hour recorded sessions, then updated workflow documents and rated each step. Workflows were annotated as graphs of human and agent actions with adoption labels. Four agentic models and two open-weight video models were tested on video, graph and chat inputs.
Limits
- 50 participants with a single default agent; Claude may have an advantage reading its own interface.
- GPT-6 was evaluated on fewer instances due to cost.
- Ratings are self-reported and skew positive.
Why it matters for companies
Agents cannot yet tell whether their work was used, changed or thrown away once it leaves the chat, nor judge whether the person was satisfied. Usage dashboards and model-graded "success" will therefore overstate value. Companies measuring AI deployment should track what happens to outputs downstream and ask staff directly, rather than trusting automated quality scores.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Work and AI