Can AI Agents Help Robotic Engineers? Lessons from Two Focus Days in Industry
B · Useful, limited University of Passau case study with a crossover design, real industrial tasks and assessment by the firm's own team lead; careful and transparent, but four engineers and 42 task solutions cannot support general conclusions.
Summary based on the full text·Original: about 40 minutes to read
In short
A preprint (arXiv, 23 October 2026) by Salim Fares and Steffen Herbold, University of Passau, on a robot-programming firm, AKE Technologies, trying a coding agent on its daily work. Four experienced engineers solved six typical tasks with and without Claude Code running an on-premise open-weight model (GPT-OSS 120B). The agent made them slower and their code worse on most tasks; it helped only on highly repetitive trajectory code. Results improved after eight weeks and a reconfigured agent, and prior AI experience mattered more than robotics experience.
Key numbers
- 3 of 12 vs 9 of 12tasks finished within the time limit with the agent versus without AI on the first focus day
- 1 of 9 vs 5 of 9tasks finished in time with the agent versus without AI on the second focus day
- 25 vs 11programming-standard violations with versus without the agent on day one (16 vs 15 on day two)
- 5 of 8 vs 1 of 7violation-free solutions with versus without the agent on day two, after the agent was reconfigured
- 8 weeksgap between the two focus days, during which engineers could use the agent in daily work
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Day one: 3 of 12 tasks finished with the agent versus 9 of 12 without; agent work took at least as long on every task. Team-lead ratings averaged functionality 1.7 versus 2.3, code quality 1.8 versus 2.5, readability 2.1 versus 2.8.
- Engineers used the agent for every change, even adding a comma, so it regenerated whole modules; long conversations drifted. Prepared skills were almost never invoked.
- Between days the researchers replaced the skills with automatic project context: the ABB RAPID manual, task descriptions and a CLAUDE.md file with coding rules.
- Day two: 1 of 9 finished with the agent versus 5 of 9 without, but standards adherence reversed (5 of 8 agent solutions violation-free versus 1 of 7 without). Functionality still lagged.
- Trajectory generation was the one clear win: the most AI-experienced engineer finished in 15 minutes, as fast as the most senior engineer working without AI.
- The engineer who rarely used AI finished no task with the agent.
How they got it
AB/BA crossover over two full focus days: each engineer did each task type once with and once without the agent, with different task variants on day two. Time-boxed tasks (30-60 minutes) were defined and graded by the firm's head of robot programming; a validator checked standards; questionnaires and chat logs were collected. Inference ran on one local H100 because customer data cannot go to the cloud.
Limits
- Four engineers (three on day two); no statistical generalisation, as the authors state.
- One model, chosen informally among five open-weight candidates.
- The terminal interface was unfamiliar to engineers used to graphical tools, which conflates tool design with agent capability.
- The agent had no access to the simulator's visual feedback.
Why it matters for companies
For engineering teams outside mainstream software, an agent dropped in without training and domain context costs time. The fixes this study points to are cheap: load reference manuals and coding rules into the project context automatically, start with repetitive, well-specified tasks, and teach people when to delegate and when to edit by hand. It also shows that a confidential, on-premise setup is technically workable for a small firm, and that one-day productivity tests miss the learning curve.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org