---
id: "lib-2026-11-03-payworthy-can-llms-predict-which-freelance-jobs"
type: "library"
title: "PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?"
lang: "EN"
org: "Upwork"
source_type: "Report"
date: "2026-11-03"
week: "2026-W45"
url: "https://nosov.ai/library/lib-2026-11-03-payworthy-can-llms-predict-which-freelance-jobs/"
original_url: "https://arxiv.org/abs/2611.05759"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Upwork researchers build a benchmark from real postings sampled by a fixed rule, test 18 forecasters, check results across judges and against human raters, and release code; the label is an LLM judge, not real payment."
summary_basis: "full-text"
original_minutes: "35"
tags: ["future of work", "ai agents", "freelancing"]
---

# PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?

Tier A (Strong evidence): Upwork researchers build a benchmark from real postings sampled by a fixed rule, test 18 forecasters, check results across judges and against human raters, and release code; the label is an LLM judge, not real payment.

Summary based on the full text.

## In short

An arXiv paper (3 November 2026) by six Upwork researchers on whether frontier models can predict, before an AI agent tries, which paid freelance jobs it will complete to a client's satisfaction. An agent attempts each of 1,494 real postings from the posting alone, and an LLM judge acting as the client accepts or rejects the work. Told what the agent cannot do, every forecaster ranks jobs well. Told it is a generic capable agent, they do about as well as chance, mainly because they miss that the agent never receives the client's files.

## Key numbers

- 1,494: real freelance job postings in PAYWORTHY, sampled from 18.8M posted 2020-2026
- 0.66-0.86: AUC of 18 models forecasting agent success when told what the agent cannot do
- 0.45-0.56: AUC of the same models when told it is a generic agent with a browser and code execution
- 54.4%: share of agent attempts the LLM judge, acting as client, accepts
- 35%: share of judge-accepted work that two human raters accepted (149 jobs)

## What it reports

- Postings from 12 categories, sampled by a fixed rule from 18.8M postings requesting $69.1B of work; the agent (based on claude-opus-5) works in one response without tools.
- The judge (gpt-5, three votes, Fleiss kappa 0.86) accepts 54.4% of attempts; judges from five model families accept 33-58% of the same deliverables.
- Limits prompt: AUC 0.66-0.86 across 18 forecasters; the fifth of jobs each rates highest is accepted 65-92% of the time. Generic prompt: AUC 0.45-0.56; for 15 of 18 forecasters, their top fifth is accepted less often than a random fifth.
- Two of six stated limits account for most of the gain. Larger models, reasoning models and averaging do not fix generic forecasts.
- The judge accepts 82% of self-contained jobs, 42% of jobs needing the client's files and 32% of jobs needing another medium.
- Human raters with category experience accepted only 35% of 149 judge-accepted jobs; three quarters of their rejections cite missing information.
- Best forecasts still trail repeated attempts (AUC 0.86 vs 0.94). A metadata classifier's top fifth reaches 77%.

## How they got it

Benchmark construction from platform data, an agent run per job, an LLM judge as simulated client, and forecasts from 18 models under two prompts, scored by AUC with bootstrap intervals. Robustness checks: four other judge families, repeated attempts, a tool-using agent with a simulated client, two human raters, and 322 jobs with real paid freelancer work as an anchor. Code is on GitHub.

## Limits

- The outcome is an LLM judge's acceptance, not real client payment; on real paid work the judge accepted only 16.5%, and humans are far stricter.
- Two human raters covered only judge-accepted jobs, so rater agreement and rejected jobs were not measured.
- Models are called under provider aliases that may change; authors work for the platform that supplied the data.

## Why it matters for companies

Before routing work to agents, a triage step is needed, and a model can do it reasonably only if told the agent's actual limits: no client files, no tools, no follow-up questions. Described as a generic capable agent, the same models pick worse than random. Teams building agent intake should write down what the agent cannot access and feed it to the triage model. The human-rater result is also a warning that LLM judges overstate acceptable work, especially for short briefs.

Original: https://arxiv.org/abs/2611.05759