RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale
B · Useful, limited Alibaba and university co-authors describe the production inference engine behind about 100 models; credible operating detail, but outcome figures are self-reported from one company with few measured incidents.
Summary based on the full text·Original: about 50 minutes to read
In short
An arXiv systems paper (31 October 2026) on RTP-LLM, Alibaba's open-source inference engine used across Alibaba Cloud, Taobao, Tmall, Cainiao, Amap, AliExpress, Lazada and other businesses. The design separates model-specific execution from shared request handling, state management and resource coordination, so new models change little shared code. The paper describes how models are onboarded and how failures are contained, with a few production measurements.
Key numbers
- about 100models supported by the shared production engine
- hundreds of trillionstokens processed per day
- 14-28 daysusual time from a community model release to production readiness
- below 0.01%daily request error rate on nearly all later days of an 80-day MaaS cluster observation
- 40.9%share of failed requests due to unreachable workers, connection resets or RPC timeouts
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- About 100 models served, hundreds of trillions of tokens a day, across heterogeneous hardware and service-level targets.
- Share of lines changed per model onboarding: DeepSeek-V4 3.17% in the model layer and 0.84% in the serving layer; Qwen3.5 0.14% and 0.0003%.
- Onboarding timeline: integration and canary from one day to about a week; performance optimisation about three weeks for DeepSeek-V4 and two for others; 14-28 days in total from community release.
- Release process: regression validation, limited production rollout comparing errors, output abnormalities, latency and resources with the serving version, and rollback on regression.
- An 80-day MaaS cluster: request volume grew while error rate fell, staying below 0.01% on nearly all later days.
- Failed requests: 40.9% connectivity and RPC timeouts, 34.6% execution errors and exceeded deadlines, 19.4% memory or KV-cache allocation and overload.
- Prefix cache hit rates of up to 80-90% in some applications.
How they got it
Architecture description and lessons from operating the engine, with code-change counts, onboarding timelines and telemetry from a single production cluster.
Limits
- Self-report by the operator; no comparison with vLLM, SGLang or other engines under equal conditions.
- Timelines depend on team size and are not controlled comparisons, as the authors note.
- Reliability data cover one cluster and one service.
Why it matters for companies
New open models arrive monthly, and onboarding each one is recurring work, not a one-off integration. The separation of model-specific code from shared serving is the pattern to look for in any internal platform or vendor, along with canary rollouts and rollback. The two-to-four-week lag from release to production at a firm with Alibaba's resources is a realistic floor for others.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org