Advancing coding in low-resource programming languages (OpenAI; coverage: COBOL score 5.4% to 86.6%)
B · Useful, limited Let's Data Science news write-up of OpenAI's 19 October post; it relays vendor-run results but adds useful verification context (hidden tests, an outside developer's check of the public repo, IBM precedent).
Summary based on the full text·Original: about 7 minutes to read
In short
A Let's Data Science article (20 October 2026) on OpenAI's post "Advancing coding in low-resource programming languages". OpenAI reports that its newest models went from failing almost all of its COBOL migration tests to passing most of them, and released MultiLangBench, a public benchmark that asks models to build a SQLite-compatible engine in 20 languages. The article's point is the split: the headline migration score comes from hidden tests only OpenAI can run, while MultiLangBench can be rerun by anyone. The primary OpenAI page was not reachable (403), so this digest rests on the secondary coverage.
Key numbers
- 5.4% to 86.6%OpenAI COBOL migration eval pass rate, GPT-5.6 Sol low to GPT-6.1 Sol low reasoning
- 61%GPT-6.1 Sol low cost as a share of GPT-5.6 Sol cost on the migration eval (OpenAI estimate)
- 97.9%GPT-6.1 Sol pass rate at high reasoning effort, which OpenAI calls nearly saturating
- 26.8% vs 87.4%MultiLangBench 20-language average, GPT-5.6 Sol vs GPT-6 Astra at maximum effort
- 72.1%GPT-6 Astra on COBOL in MultiLangBench, its weakest of the 20 languages
As reported by the source.
English·Web page·opens letsdatascience.com
What it reports
- COBOL migration eval: models convert COBOL to C++, Java, Kotlin, Python or Rust; hidden tests require byte-for-byte parity on observable outputs, and a migration passes only if all tests pass.
- GPT-5.6 Sol (low reasoning) 5.4%; GPT-6.1 Sol (low) 86.6% at an estimated 61% of the older model's cost; GPT-6.1 Sol (high) 97.9%.
- MultiLangBench at maximum effort, 20-language average: GPT-5.6 Sol 26.8%, GPT-6 Luna 40.3%, GPT-6 Astra 87.4%. GPT-6 Astra scores above 83% in 19 of 20 languages; COBOL is lowest at 72.1%.
- GPT-6 Luna costs an estimated $0.77 per sample, about 78% less than GPT-5.6 Sol.
- OpenAI's post cites COBOL in more than 40% of online banking systems and 95% of ATM transactions, and warns that production use "still requires expert review".
- An outside developer, Gabriel Anhaia, confirmed the public repository covers the SQLite build task under Apache 2.0, and noted he had not run a COBOL migration himself.
How they got it
The article summarises OpenAI's post, TechBriefly and AlphaSignal coverage, and Anhaia's DEV Community review, and adds IBM's 2023 watsonx Code Assistant for Z launch as precedent. No independent testing was done by the publication.
Limits
- The migration figures come from an eval OpenAI designed, ran and has not released; the number of COBOL programs in it is not stated, and no outside group has verified it.
- The article notes the same vendor's earlier model was caught gaming a coding benchmark in June, a problem a private eval cannot reveal.
- Secondary source: the OpenAI primary page returned 403 and was not read.
Why it matters for companies
Legacy migration is a large, real cost line, and these numbers will appear in vendor pitches. The defensible part is the method, not the score: byte-for-byte parity tests on captured production inputs and outputs, with explicit checks for fixed-point truncation versus rounding in financial code. Teams with COBOL or Fortran estates can run MultiLangBench on their candidate models and build their own parity suite before any pilot. Treat 86.6% as a vendor claim until someone outside can rerun it.
Summary based on the full text. Read the original for the full argument and data.Read the original on letsdatascience.com
- Tags
- Evaluation, Openai