Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations
B · Useful, limited A large randomized field trial with a pre-specified design, but written by the founder and CEO of the vendor that sells the agent, with no registration, independent statistical review or ethics approval.
Summary based on the full text·Original: about 45 minutes to read
In short
A vendor preprint (arXiv, 18 October 2026) by Benjamin Tannenbaum, founder and CEO of AISO Boost Ltd., which operates the Javo sales agent. Javo is fine-tuned on 4.6 million consented, de-identified commercial conversations. Four studies compare it with public benchmarks, with no agent, with a frontier model in the same scaffolding, and with each site's own human chat associates. The agent raised revenue per visitor and matched human associates on conversion at a small fraction of their cost.
Key numbers
- 14.8 Mvisitors randomized across 100 live websites, 1 May to 30 September 2026
- +10.0%change in revenue per visitor with the Javo agent versus no agent (95% CI +5.1 to +14.9)
- +21.6%change in engaged conversion versus a frontier-model agent in identical scaffolding
- 11.26% versus 11.20%engaged conversion of the agent versus 59 human associates on 20 sites (non-inferior, not superior)
- 1.8%agent cost per conversation as a share of human associate cost
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Study 2: 14.8 M visitors on 100 sites in 14 industries randomized to no agent, Javo, or a frontier-model agent with the same interface, catalog tools and guardrails. Revenue per visitor +10.0% versus no agent across 85 transactional sites.
- Study 3: holding scaffolding fixed, engaged conversion +21.6% versus the frontier-model agent (95% CI +16.9 to +26.3).
- Study 4: on 20 sites with staffed live chat, against 59 associates (median tenure 25 months), engaged conversion 11.26% versus 11.20%. Non-inferior at a -7% margin (one-sided p=0.004); no significant superiority.
- Cost per engaged conversation: $0.06 for Javo, $0.14 for the frontier agent, $3.40 for human associates.
- Safety: return, cancellation, unsupported-claim and complaint rates met pre-specified margins. Blind auditors reviewed stratified samples. The agent was paused on 2 sites for 5 site-days after an unauthorized-discount guardrail triggered.
- Both agent arms disclosed they were AI, so effects are net of disclosure.
How they got it
Site owners consented to randomization under a data processing agreement. Outcomes, margins and hypotheses were pre-specified in the paper with Holm correction. The frontier baseline was one model version frozen on 24 April 2026. A human audit sample of 2,000 conversations per agent arm and 1,000 human conversations was labelled for unsupported claims, unauthorized offers and pressure tactics.
Limits
- Conflict of interest: the sole author runs the company; no study registration, independent statistical review or external ethics approval is claimed.
- Sites came largely through the author's client and agency network and over-represent high-consideration purchases.
- The frontier baseline is a single snapshot; newer models could close the gap.
- Study 4 compares typical site staff, not expert sellers, and the human and AI widgets differ in disclosure as well as seller.
- Five months does not capture lifetime value, repeat purchase or brand effects.
Why it matters for companies
The design is the useful part for any buyer of a sales or service agent: measure incremental revenue per visitor on your own traffic, hold the scaffolding fixed when comparing models, and pre-set safety margins with pause rules. The headline cost ratio is the kind of figure that drives replacement business cases, so it should be reproduced independently before it is used. The result also sits beside a same-week field study showing that routing easy work to AI can change how the remaining human staff work.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Work and AI