Look Before You Launch: Learned Throughput Prediction for Hardware Selection in a Production Training Fleet
B · Useful, limited Meta engineers report a production throughput predictor routing training jobs across 20+ hardware types; solid practice with ablations, but the savings figure is the operator's own estimate on internal data.
Summary based on the full text·Original: about 35 minutes to read
In short
An arXiv paper (2 November 2026) by Hossein Askari and colleagues at Meta. In a mixed-vendor training fleet, many jobs can run on several accelerator types, and a poor choice wastes compute without any error. LEAP predicts each job's throughput on every candidate type at submission and ranks options by throughput per unit cost. It replaced a lookup table and now covers almost all eligible GPU-hours.
Key numbers
- 77%prediction accuracy over 30 nightly retrains
- 58% to 95%share of eligible training GPU-hours ranked by LEAP over six months
- about 3%estimated saving in training compute cost
- more than 20hardware types in the mixed-vendor fleet; tens of thousands of job attempts routed a day
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Routes tens of thousands of training job attempts a day across hundreds of model types and more than 20 hardware types.
- Accuracy 77% over 30 nightly retrains, higher than the lookup table it replaced and at least as high as every alternative tested, including fine-tuned language models.
- Coverage of eligible training GPU-hours grew from 58% to 95% over six months.
- Estimated saving: about 3% of training compute cost.
- Hardware routing governs roughly half of the fleet's half-year GPU-hours; the long tail matters, with the top 100 model types covering 78% of one day's attempts.
- Each job's own prior throughput is a key input; the paper reports which drivers move accuracy and which do not.
How they got it
A learned prediction service trained on submission-time configuration and execution history, evaluated against the legacy table and alternative models, with coverage and savings tracked in production.
Limits
- Self-report on internal data; the 3% saving is an estimate, not a controlled comparison.
- The accuracy metric's definition matters and is specific to the paper.
- Applies to fleets large and heterogeneous enough to have real routing choices.
Why it matters for companies
Silent inefficiency is common wherever workloads can run on several hardware or cloud options: jobs finish, so nobody notices the waste. A simple learned predictor using each job's own history captured a few percent of compute at Meta, and the same idea applies to cloud instance selection. Language models were not better than a conventional model for this task.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org