Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy
B · Useful, limited PayPal's platform team documents the design and operation of its multi-vendor LLM gateway in unusual detail; the architecture and lessons are concrete, but operating figures are point-in-time self-reports and the only benchmark uses a synthetic upstream.
Summary based on the full text·Original: about 45 minutes to read
In short
A preprint (arXiv, 26 October 2026) by Mithun Sasidharan and colleagues at PayPal describing Cosmos, the single governed entry point for the company's LLM traffic. Every call to any vendor passes through one gateway on Envoy that handles identity, credentials, policy, budgets, data-loss prevention, format translation and usage records. The paper reports the architecture, how governance is rolled out, operating figures and ten lessons.
Key numbers
- ~200 billiontokens per day across more than 100 models from several providers
- 12,000+developers using the gateway, alongside hundreds of production use cases
- 4 ms / 11 msmedian and p99 overhead added by the core request path at over 2,200 requests per second
- 99.999%availability to business as of September 2026
- up to 96%prompt-cache hit rate on the best agentic coding workloads from gateway-side cache shaping
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Two tiers: I/O-bound decisions (authentication, credential brokering, policy, budgets) run out of process; streaming translation, token metering and cache shaping run in-proxy as a Rust WebAssembly filter, so streamed chunks do not wait on external calls.
- Models, providers and prices are validated data, not code; a model goes from staged to generally available without a deployment.
- Policy as code in Open Policy Agent, with role-scoped authoring, approvals, scheduled activation, cohort pilots and shadow mode before blocking. A typical budget rule warns at 90% and blocks at 100%.
- Budgets are views over per-model usage counters, so a new budget applies to spend already recorded in its window without data migration.
- Optional data-loss prevention combines pattern rules with an internally hosted model to block, redact or pseudonymise content before it reaches a provider.
- Operation as of September 2026: about 200 billion tokens a day, peak 60,000 requests per minute from one workload, over 2,000 concurrent voice calls, more than 10 deploys during live voice traffic with no dropped calls.
- Lessons include "wrap, don't rewrite", anchor policy to stable identities such as organisation leaders, and "routing is not the differentiator".
How they got it
Design description by the operating team, production figures from internal monitoring, and an overhead benchmark of the core request path against a synthetic upstream on two 4-vCPU instances. The code base is about 45k lines of Go and 7k of Rust.
Limits
- Self-report from one organisation; figures are point-in-time, not service commitments.
- Overhead was measured against a synthetic upstream, not real vendor latency.
- No cost or spend-reduction figures beyond cache hit rates.
- Centralisation concentrates risk: the gateway holds credentials for every provider, as the authors note.
Why it matters for companies
Any firm using several model vendors runs into credential sprawl, format lock-in and untracked spend. This is a detailed reference design for a central gateway that fixes those while letting model choice change by configuration. The transferable practices are cheap to adopt at any scale: store usage at the finest grain and make budgets views over it, ship every new control in shadow mode, warn before blocking, and tie policies to stable organisational identities so they survive reorganisations.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org