How Amazon Marketplace evaluates content quality with Amazon SageMaker AI
B · Useful, limited Amazon's own engineering account with a labelled dataset, a holdout comparison and an ablation; internal data, no public replication, and the post doubles as a SageMaker showcase.
Summary based on the full text·Original: about 15 minutes to read
In short
An AWS Machine Learning blog post on how Amazon scores A+ Content, the rich media block on product pages. The page is rendered as one snapshot and a vision-language model scores it against a four-part rubric. Those scores, plus four OCR readability measures, feed a small CatBoost classifier trained on expert labels. SHAP values then rank what to fix, and an LLM turns them into recommendations for the seller.
Key numbers
- 5,478product detail pages from 10 categories labelled by one expert and four annotators
- 0.94 / 0.95accuracy and F1 of the fused VLM + OCR CatBoost classifier on a 20% holdout
- 0.80 / 0.84accuracy and F1 of the baseline that averages the four VLM scores
- 1.4 pointsaverage quality-score gain (on 10) after selling partners revised pages using the recommendations
- 0.845 to 0.915Cohen's kappa of each annotator against the consensus label
As reported by the source.
English·Web page·opens aws.amazon.com
What it reports
- Rubric dimensions: Visual Storytelling, Feature Presentation, A+ Layout, Conversion Optimization; each scored 0-10 with a confidence and rationale.
- Labels: 5,478 pages, 10 categories, majority vote of one marketing expert and four annotators; classes nearly balanced (49.6% high quality).
- Holdout results (about 1,100 pages): averaged-VLM baseline 0.80 accuracy / 0.84 F1; OCR-only 0.82 / 0.83; VLM-only CatBoost 0.89 / 0.90; fused 0.94 / 0.95.
- On a single-category set of 499 projects, the fusion gain held across two VLM backbones (+0.16 and +0.15 accuracy).
- Most influential features: OCR Character Density (mean |SHAP| 1.1), OCR Overall Readability (1.0), VLM Visual Storytelling (1.0).
- Revised pages gained 1.4 quality points on 10. A worldwide A/B test that changed visibility of low-quality content gave a "statistically meaningful" rise in ordered product sales, with no figure given.
- The VLM stays frozen; only the classifier is retrained, as a CPU-only job.
How they got it
Supervised evaluation on an internal labelled set with a 20% holdout and component ablation. The sales A/B test and the 1.4-point gain are reported without sample sizes or effect sizes.
Limits
- Internal data and labels; no code or dataset released.
- The quality-score gain is measured by the same model that made the recommendations.
- No sales uplift number; the 8-20% sales claim for A+ Content is stated without source.
- Vendor blog promoting SageMaker services.
Why it matters for companies
The design is a cheap and portable pattern for any content-quality or brand-compliance check: keep a frozen general model as a feature extractor, add simple deterministic measures it is weak at, and train a small classifier on a few thousand expert labels. Retraining stays fast as standards change, and SHAP gives the "what to fix" that content teams need. Brands selling on marketplaces should expect their pages to be scored this way.
Summary based on the full text. Read the original for the full argument and data.Read the original on aws.amazon.com