Experimentally Validated Digital Twins: Evaluating Workforce Training Programs in the Age of Big Data
S · Shapes the field Raj Chetty, Lawrence Katz and co-authors (Harvard, Opportunity Insights, US Census Bureau) validate machine-learning counterfactuals on tax records for all US adults against 37 RCTs, revisiting a result that has guided program evaluation since LaLonde (1986).
Summary based on the full text·Original: about 75 minutes to read
In short
A paper dated October 2026 (arXiv 2 November) by Raj Chetty, Jamie Fogel, Lawrence Katz, Kadeem Noray, Sonya Porter and James Reisinger. Program evaluation has relied on RCTs because observational estimates diverged from experiments in small survey datasets. With anonymised tax records for all American adults from 1999 to 2025, machine-learning counterfactuals match 37 RCTs of job training programs about as closely as the RCTs match the truth. The authors call estimators that pass this test "experimentally validated digital twins" and use them to evaluate many more sites and years.
Key numbers
- 37randomised trials of workforce training programs used to estimate the bias of the observational method
- $1,717 vs $2,126noise-adjusted RMSE of the digital twins estimates vs RMSE of the RCT estimates themselves
- 60%RMSE reduction relative to the parametric estimators used in prior work
- 0.93correlation between digital twins and RCT estimates across the 37 trials
- $121larger annual earnings impact per percentage point of target-sector job placement
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Digital twins and RCT estimates correlate at 0.93 across 37 trials. Noise-adjusted RMSE (bias only) is $1,717; total RMSE $2,073; the RCTs' own RMSE from sampling error is $2,126.
- Parametric specifications from LaLonde (1986) and Dehejia and Wahba (1999) have noise-adjusted RMSEs of $4,485 and $3,960 on the same trials, about 2.5 times larger; the new method cuts RMSE by 60%.
- The gain comes from combining large samples with flexible models: at baseline complexity, moving from a 0.02% sample to the full population cuts RMSE by 52%; with a model 3.33% as complex, only by 13%.
- RCT effects range from -$1,188 to $27,303 a year; the margin of error of $4,063 is smaller than the $5,699 spread, so the method can rank programs.
- Effects vary widely across locations and over time within the same program.
- Target-sector placement drives gains: $3,040 a year where placement is below 31%, $10,163 where it is 80% or more; $121 per percentage point. Gains persist after participants leave their first job; 90% of Year Up United participants switch jobs within five years.
How they got it
Double machine learning predicts counterfactual earnings from prior work histories, demographics and residence for non-participants. Bias is estimated as the noise-adjusted gap between these estimates and the RCT estimates for the same programs, sites and years, using RCT data from MDRC and Economic Mobility Corporation.
Limits
- Validation covers US workforce training programs; whether the method passes in other domains must be tested domain by domain.
- Requires access to confidential tax data through the Census Bureau.
- Raj Chetty leads Opportunity Capital, which invests in training programs using these methods (disclosed).
- The placement effect rests on an identifying assumption the authors state; it is not experimentally assigned.
Why it matters for companies
For employers and funders of training, the result means program impacts can be measured from administrative data at far lower cost than running trials, provided the method is first validated against some experiments. The strongest practical finding is simple: programs that place people into jobs in the sector they trained for produce the largest earnings gains. The same validate-then-scale logic applies to evaluating internal reskilling with HR and payroll data.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Future of work, Evaluation