---
id: "lib-2026-10-25-a-10-year-old-cnn-outperforms-frontier-llms-for"
type: "library"
title: "A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis"
lang: "EN"
org: "Lynch Lab (Magnus Lynch)"
source_type: "Practitioner post"
date: "2026-10-25"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-25-a-10-year-old-cnn-outperforms-frontier-llms-for/"
original_url: "https://lynchlab.net/llm-skin-cancer"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "A consultant dermatologist and King's College London honorary reader runs a clear, reproducible test on biopsy-proven public images and releases cases, prompts, every reply and scoring code; the sample is small and the comparison has caveats he states."
summary_basis: "full-text"
original_minutes: "5"
tags: ["medical ai", "evaluation", "narrow vs general models"]
frames: ["verification-capacity"]
---

# A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

Tier A (Strong evidence): A consultant dermatologist and King's College London honorary reader runs a clear, reproducible test on biopsy-proven public images and releases cases, prompts, every reply and scoring code; the sample is small and the comparison has caveats he states.

Summary based on the full text.

## In short

A 25 October 2026 post by Dr Magnus Lynch, consultant dermatologist and Mohs surgeon at Guy's Hospital and honorary reader at King's College London. He gave 200 randomly drawn, biopsy-proven skin lesions to GPT-6 Astra, Claude Fable 5.1 and a fine-tuned Inception v3, a convolutional network first published in 2015. The CNN named the right diagnosis out of six for 125 lesions (62%); the two LLMs for 84 and 85 (42%). The LLMs kept almost all dangerous lesions but would have discharged few harmless ones.

## Key numbers

- 200: biopsy-proven lesions from the public MILK10k dataset, each with a clinical and a dermoscopic image
- 62% vs 42%: correct first-choice diagnosis (of six) for the fine-tuned Inception v3 CNN versus GPT-6 Astra and Claude Fable 5.1
- 32 and 22: safe lesions (of 100) the two LLMs would discharge, against 76 for the CNN at its preset threshold
- 68,011: ISIC Archive images used to fine-tune the CNN, with all test lesions and patients removed

## What it reports

- Test set: 100 lesions not safe to discharge (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 safe (50 moles, 50 seborrhoeic keratoses). All were biopsied, so they are harder than typical cases.
- First or second choice correct: CNN 173 of 200, LLMs 134 and 138. Melanoma as first choice: 13, 8 and 11 of 30.
- Unsafe lesions kept: LLMs 97 and 96 of 100; CNN 77 at its preset threshold, 98 or 99 when matched. Melanomas discharged: LLMs 3 and 4; CNN 18 at preset, 2 or 1 when matched.
- Safe lesions discharged: GPT-6 Astra 32, Claude Fable 5.1 22, CNN 76 at preset threshold.
- Each LLM had a favourite answer: GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of 30 melanomas; Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were.
- Asked as a patient with only the clinical photo, both LLMs advised seeing a doctor in almost every case, including 99 and 98 of a further 100 melanomas.
- The CNN was fine-tuned for about twenty minutes on one H100 and runs in a phone browser.

## How they got it

LLMs received both images and a fixed prompt at default settings, asking for a diagnosis or "unsure", confidence, second choice and a discharge decision; each case ran once. Scoring was by script against histology. The CNN was fine-tuned on ISIC Archive images with all test lesions and any other images of the same lesion or patient removed. Cases, prompts, replies and scoring code are downloadable.

## Limits

- 200 lesions and 30 melanomas give wide uncertainty; close results cannot be ranked.
- Single runs per LLM case; answers vary between runs.
- MILK10k has been public since 2025, so LLM training contamination is possible (which would favour the LLMs).
- Inception v3 is a baseline; regulated products perform considerably better. Analysis and drafts were produced with Claude Code, and one tested model is Anthropic's.

## Why it matters for companies

A general-purpose model is not a substitute for a small model trained on the task, even when the general model is far newer and larger. In healthcare and any other high-stakes classification, procurement should ask for task-specific validation with held-out data, not general benchmark scores. The LLMs' safe-but-useless pattern (refer almost everyone) would also defeat the point of triage, sending nearly all harmless cases on for review.

Original: https://lynchlab.net/llm-skin-cancer