2 sources filed · EN 1 · RU 1
Rollout of real sends went 10%, 30%, 50%, 100%, with dry run as default and sends held when the model layer failed.
605 of 655specialists' closing messages that also invited the buyer to write again, which defeated keyword rules
Benchmark: 1,000 identities built from pseudopeople synthetic records with added noise, half single people and half over-merges.
99.0% at $3.86Claude Opus 5.5 accuracy and cost per 1,000 identities in grading identity-resolution results
2 sources filed · RU 2
Findings from v1: 32 of 120 phrases were not recognised; sellers wanted explanations, not the tables they already see in the marketplace dashboard; feature requests outpaced coding.
32 of 120seller phrases in the first conversations that the rules-based version did not recognise (shown in square brackets in the source)
Human assessors on a domain benchmark (medicine, law, science) cost from several thousand to several tens of thousands of dollars per model, so they cannot be used for routine evaluation of every checkpoint.
~$400cost of scoring one model checkpoint on the full public benchmark suite with a GPT-4 judge