The knows tests web agents on open-ended tasks that require searching the live web, using Google Workspace, and producing a structured artifact such as a document, spreadsheet, or slide deck. Each of the 110 tasks is scored by an evaluator that combines deterministic checks with LLM judgments across multiple checkpoints.\n\nOn state-of-the-art , the best performer fully succeeds on fewer than 3% of tasks. Partial-success scores were roughly 35 to 70%, but agents often failed visual/spatial and source-faithfulness steps, which made the resulting artifacts unusable even when most other steps passed. This matters because real assistant work is not just retrieving information but synthesizing and displaying it well.
Web agents struggle to turn search results into usable artifacts
The KNOWS benchmark tests whether browser agents can research live web information and assemble it into coherent documents, spreadsheets, and presentations.
Academic
University of Utah
Research Digest··2 min read
In one line
Frontier computer-use agents fully complete fewer than 3% of long-horizon web tasks requiring synthesis and artifact creation, and visual/spatial failures often make partial artifacts unusable.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →computer-use agentAn AI agent that interacts with computer interfaces—GUI, terminal, or code—to autonomously perform tasks.50 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real304 papers →
Read it as
Gill and colleagues introduce KNOWS, a benchmark of long, open-ended workflows spanning web research, information synthesis, interface navigation, and artifact creation in Google Workspace.
The authors designed browser-based tasks in which an agent searches the live web and produces a specified document, spreadsheet, or slide deck in Google Workspace.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.