How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting
A · Strong evidence Finance and accounting academics (Toronto, UCL, Macau) test GPT-4o referee scores against later top-journal publication on 280 conference papers, with look-ahead controls and replications; clear design, modest sample.
Summary based on the full text·Original: about 50 minutes to read
In short
Hai Lu, Yichen Luo and Luping Yu ask which part of a referee's job a language model can do. GPT-4o, with training data ending before the 2024 conference season, refereed papers from the 2024 AFA and NBER finance meetings using a plain three-sentence prompt. Its scores predicted which papers reached the top three finance journals within two years. All the predictive signal came from its judgements of correctness and clarity; its scores for originality and significance carried none. The authors conclude that models can check whether research is right, but not whether it matters.
Key numbers
- 44%higher likelihood of top-three finance journal publication per 1-SD rise in the model's score, net of author reputation
- 2.9 percentage pointsthe same effect in absolute terms, from a base rate of 6.5%
- 280papers in the main sample from the 2024 AFA meeting and NBER finance meetings
- 90%papers for which the model says it does not know whether they were published, even given the authors
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- A 1-SD higher model score makes publication 44% more likely, or 2.9 points from a 6.5% base rate, controlling for school ranking, publication record and editorial experience.
- The model does not recall outcomes: it says it does not know for 90% of papers, and the score keeps its power after controlling for its answers.
- Originality and significance scores carry no information; the overall score predicts worse than correctness or exposition alone.
- Giving the model a referee guide that stresses importance does not help, and its overall score then stops predicting publication.
- On 41 accounting papers, overall and correctness scores predict top-three accounting publication; originality and significance do not.
- GPT-5 shows the same split.
- Context: about 21% of ICLR 2026 reviews were classified as fully AI-generated.
How they got it
363 conference papers whose PDFs were public; 280 outside the AFA PhD poster session form the main sample. GPT-4o scored each on originality, significance, correctness, exposition and overall. Publication in the Journal of Finance, Journal of Financial Economics or Review of Financial Studies within two years is the outcome. Look-ahead bias was probed by asking the model about each paper's fate.
Limits
- Publication is a proxy for quality and reflects the same human judgement of importance the model fails to match.
- Small samples, especially 41 accounting papers.
- One naive prompt; more elaborate agentic reviewers were not tested.
Why it matters for companies
The split generalises beyond journals. In any review process, such as investment memos, research reports, audit work papers or grant proposals, models are useful as a technical check on correctness and clarity. Judging which proposal matters most remains a human call, and telling the model to focus on importance does not fix that. Design review workflows so the model screens for errors and people decide priority.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Evaluation