---
id: "lib-2026-10-27-dayjob-tax-evaluating-ai-agents-on-50"
type: "library"
title: "DAYJOB: Tax, evaluating AI agents on 50 professional tax assignments"
lang: "EN"
org: "Surge AI"
source_type: "Report"
date: "2026-10-27"
week: "2026-W44"
url: "https://nosov.ai/library/lib-2026-10-27-dayjob-tax-evaluating-ai-agents-on-50/"
original_url: "https://surgehq.ai/blog/dayjob-tax"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "Surge AI, a data and evaluation vendor, releases an expert-built tax benchmark with a strict pass threshold and concrete failure cases; the design is credible, but the post gives only the top three scores, the full dataset is gated, and the vendor sells the expert labour behind it."
summary_basis: "full-text"
original_minutes: "9"
tags: ["ai technology", "future of work", "enablement", "business transformation", "professional work", "tax", "agent evaluation"]
frames: ["verification-capacity"]
---

# DAYJOB: Tax, evaluating AI agents on 50 professional tax assignments

Tier B (Useful, limited): Surge AI, a data and evaluation vendor, releases an expert-built tax benchmark with a strict pass threshold and concrete failure cases; the design is credible, but the post gives only the top three scores, the full dataset is gated, and the vendor sells the expert labour behind it.

Summary based on the full text.

## In short

A Surge AI blog post (27 October 2026) introducing DAYJOB: Tax, the third benchmark in its DAYJOB family after healthcare and finance. Agents get a client request and a folder of records and must deliver a return, workpaper, review or advice letter, including issues the client did not raise. An attempt passes only if it meets 95% of the task's criteria. The best model passed 10.8% of attempts. Surge's summary: frontier agents can do the individual steps but not yet carry a professional assignment end to end.

## Key numbers

- 50: tax assignments across eight areas of practice
- 1,789: workspace files, mostly PDFs (1,487), with a median of 30 per assignment
- 95%: share of rubric criteria an attempt must meet to pass
- 10.8%: pass rate of the top model, Claude Sonnet 5.5
- 9.6% ... 9.2%: pass rates of Claude Fable 5.1 and Claude Opus 5.5, second and third

## What it reports

- Leaderboard (pass@1 at the 95% threshold): Claude Sonnet 5.5 10.8%, Claude Fable 5.1 9.6%, Claude Opus 5.5 9.2%. The full leaderboard is on Surge's site.
- Sonnet 5.5 and Opus 5.5 sit at the high-performance end of the cost-versus-score frontier.
- Eight areas: individual and family, small business, partnerships, property and investments, estates and trusts, international, corporate transactions, review and disputes.
- 1,789 files (1,487 PDFs, 117 Excel workbooks, 38 Word documents, plus emails, images, CSV and JSON), up to 123 per assignment; they include a photographed handwritten receipt and a dispatch log showing crews in five states for a client that assumes it operates in one.
- Case: a client compares two sale dates for a condo. Both qualify for the up-to-$250,000 exclusion, but eligibility lapses around August 2027. Astra missed the deadline in all five attempts; Opus caught it in all five.
- Case: a separating couple asks how to split retirement accounts. Opus assumed they were unmarried in all five attempts despite records suggesting an informal Texas marriage; Astra flagged the need to verify in four of five.
- Tasks are written by tax experts from real workflows and go through a three-layer expert review.

## How they got it

Tax professionals (backgrounds at large accounting firms and corporate tax departments) built each assignment, its records, expected outcomes and grading criteria. Models ran each task and were scored against the rubric; the post reports pass@1 and average cost per trial.

## Limits

- Only the top three scores appear in the post; other models and confidence intervals are not given.
- The dataset is available on request, so results cannot be freely reproduced.
- 50 tasks is small, and a 95% threshold makes pass rates sensitive to rubric design.
- US tax law only.
- Surge sells evaluation and expert-labelling services; the benchmark is also marketing.

## Why it matters for companies

Tax, audit and advisory firms are testing agents on exactly this kind of work. The result says that even the best agents rarely produce work a reviewer could sign without changes, and that the typical failure is judgement: missing the issue the client did not ask about, or building advice on an unverified assumption. The two cases show different models failing in different places. For now, agents fit as preparers with mandatory professional review, and evaluation should be on the firm's own case files.

Original: https://surgehq.ai/blog/dayjob-tax