OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
A · Strong evidence HKUST-GZ, HKU and industry co-authors build a benchmark from paid client engagements with client-accepted expert solutions as the reference, hidden feasibility checks, five runs per model-task pair and released data; small (31 instances) but well designed.
Summary based on the full text·Original: about 45 minutes to read
In short
An arXiv paper (16 October 2026) by Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu and colleagues at HKUST-Guangzhou, the University of Hong Kong, Tencent Hunyuan and several firms. OR-License tests whether AI agents can do operations research work from raw client materials, as a consultant would: ask the client for missing facts, formulate the optimisation problem, solve it and deliver. Agents are compared with the expert solutions clients actually accepted. When they deliver something feasible, several agents match or beat the experts' objective values. They fail often enough that none matches the experts once infeasible deliveries score zero.
Key numbers
- $3.68 milliontotal project value of the 10 paid operations research engagements behind the 31 benchmark instances
- 88.39%best strict feasibility (Grok 4.6 in the Claude Code harness); the range runs down to 45.16%
- 0.87best Deployment-Adjusted Quality against an expert reference of 1
- about 86%share of the best model's gap to experts caused by infeasible deliveries rather than weaker objectives
- 24% to 45%share of complete hypothesis-to-update research chains across models
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 31 instances across 10 industries, from paid engagements worth $3.68 million in total, de-identified but keeping original requirements, data formats and ambiguity.
- ORSim, a simulated client, answers business-fact questions (up to three facts per query) without giving modelling expertise.
- Feasibility ranged from 45.16% (Hy3) to 88.39% (Grok 4.6); conditional quality on feasible deliveries ranged only from 96.92% to 101.01% of expert value.
- Best Deployment-Adjusted Quality was Grok 4.6 at 86.55% (0.87 in the abstract). Of its 13.45-point gap, 11.61 points came from infeasible deliveries and 1.84 from weaker objectives.
- Other DAQ scores include GPT-5.6 Sol 85.02%, Gemini 3.8 Flash 82.94% and Claude Opus 5 77.44%.
- Complete research chains (hypothesis, test, evidence, update) ran from 24% (Muse Spark 1.3) to 45% (Grok 4.6). GPT-5.6 Sol formed no testable hypothesis in 41% of opportunities.
- Failures span problem formulation (wrong objectives, state transitions, information structure) as well as implementation; useful client questions continued into solving and validation, not only at the start.
How they got it
Eleven frontier models ran in the Claude Code harness (a Codex harness is an ablation), five runs per model-task pair, with ORSim exposed as a shell command. Hidden checkers verify hard constraints and recompute objectives from submitted decisions; tasks are weighted equally. Trajectories were coded into eight error categories and a research-chain scheme.
Limits
- 31 instances is small, and spreads are within-task standard deviations, not standard errors of the benchmark mean.
- Conditional quality is computed on each model's own feasible subset, so it does not compare optimisation skill across the same tasks.
- The simulated client may be easier or more consistent than real clients.
- The abstract's "Claude Code configuration" refers to the harness; the top model in it was Grok 4.6.
Why it matters for companies
For planning, scheduling and logistics work, the result says agents are close on solution quality but unreliable on delivering something that respects all business rules. The bottleneck is turning messy requirements into the right formulation. Teams piloting agents for optimisation should invest in automated feasibility checks against hard constraints and keep an expert reviewing the formulation, not just the output. Score pilots the way this paper does: zero credit for an infeasible plan, however good the objective looks.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org