GPT-6 Astra aces embodied benchmarks, struggles on real robots
GPT-6-Astra具身能力遭大考！传统基准拿满分，真机频碰壁
B · Useful, limited Jiqizhixin's detailed summary of a SenseTime-led technical report with NTU and the University of Tokyo, testing GPT-6 Astra on 12 simulated benchmarks and 8 real robots; the method is laid out, but real-robot trial counts are small and the report also promotes SenseTime's own harness.
Summary based on the full text·Original: about 10 minutes to read
In short
A Jiqizhixin article (19 October 2026) summarising "Has GPT-6-Astra Achieved Embodied Intelligence?", a technical report by SenseTime with Daxiao Robotics, Ropedia, Nanyang Technological University and the University of Tokyo. Given demonstrations in context and a larger step budget, OpenAI's general model scores at or near the top on most simulated robot benchmarks. On eight real robots without demonstrations it handles rigid pick-and-place but fails at fine contact and long tasks, and runs slowly and expensively. The report concludes that the model has crossed a capability threshold but is far from practical deployment.
Key numbers
- 100%GPT-6 Astra success on LIBERO-Long++ with in-context demonstrations and an extended step budget
- 27.6% vs 81.9%RoboDojo success at the standard step budget versus 10x the budget
- 4+ hours, $55time and cost for a real robot to put a pen in a pen holder
- 0 of 11successful plug insertions across three real robot platforms
- ~90%share of run time on the Unitree G1 humanoid spent on model deliberation
As reported by the source.
Chinese·Web page·opens jiqizhixin.com
What it reports
- Simulation (12 benchmarks): LIBERO-Long++ 100%, LIBERO-Pro 99.3%, SIMPLE 98.8%, LIBERO-Ultra-Long 98.9%, DexJoCo 87.3%, RoboDojo 81.9%. Weaker on BEHAVIOR (37.0%) and long-horizon navigation LHPR-VLN (28.9%).
- Two conditions drive the scores: a frozen in-context knowledge pack built from allowed training demonstrations, and up to 10x the standard step budget. RoboDojo rises from 27.6% to 81.9%, Bench2Dex from 25.6% to 61.5%.
- Cost: LIBERO-Long++ episodes average about 1 million tokens and close to an hour; LIBERO-Ultra-Long about 9 million tokens and nearly two hours; some RoboDojo episodes exceed three hours.
- Failure types: misjudging physical success (a cup "on the hook" that fell), irreversible actions (spilled liquid), and undoing completed steps.
- Harness tests on RoboDojo: open-loop planner plus VLA 5.83%; closed-loop review of VLA proposals 25.83%; with in-context learning 54.17%, at about 6.9x the tokens. Relative motion commands beat joint angles (18.06% vs 9.72%).
- Real robots (no demonstrations, 60-minute limit): rigid pick-and-place 17 of 18; cloth folding 2 of 4; plug insertion 0 of 11; long-horizon search 9 of 23. A dexterous hand's joint reached 62°C and stalled. Putting a pen in a holder took over 4 hours and $55.
- Household simulation: the model burned eggs by switching off the stove about 90 seconds late, left a fridge open and flooded a sink.
- SenseTime's own RoboRSI harness, which revises itself from failure logs, raised RoboDojo-Sim success from 22.48% to 54.17%, so far in simulation only.
How they got it
The underlying report runs standardised simulated benchmarks under declared conditions and zero-shot trials on eight real robot platforms of five types, with operators supervising for safety. The article reproduces the main tables and examples and links the report and code repository.
Limits
- Real-robot results rest on very few trials per task.
- High simulated scores depend on demonstrations and extended step budgets that standard benchmarks do not allow.
- The report is by a company that sells a competing model family and harness, and its own RoboRSI gains are shown only in simulation.
- This digest is based on the media summary, not the full report.
Why it matters for companies
General models now plan and correct robot actions well enough to top benchmarks, but the cost in time, tokens and failed contact tasks rules out unsupervised physical work for now. The report's case against benchmark-only evaluation applies to any agent pilot: score completion time, side effects and safety, not only whether the goal state was reached. Firms looking at robotics should expect the near-term value to come from the harness and control layer around a model, not the model alone.
Key passages in translation
Chinese original
把笔放进笔筒这个对人来说几秒钟就能完成的任务，总耗时 4 个多小时，消耗 55 美元。
Putting a pen in a pen holder, a task a person does in seconds, took more than 4 hours in total and cost 55 dollars.
Chinese original
插入插头的操作，在多款真机交叉验证 11 次，一次都没成功。
Plug insertion was cross-checked 11 times on several real robots and did not succeed once.
Chinese original
用户不会因为机器人跑分漂亮，就接受它做早餐时忘记关火。
Users will not accept a robot that forgets to turn off the stove while making breakfast just because its benchmark scores look good.
Summary based on the full text. Read the original for the full argument and data.Read the original on jiqizhixin.com
- Tags
- Embodied AI, Robotics, Evaluation