Unpaired evaluations predict model benchmarks with 8x fewer queries

The Product Kernel Perspective Space creates low-dimensional model representations from sparse response data, matching full-evaluation accuracy with a fraction of the queries.

AI Startup

Hayden Helm · Avijit Ghosh · Ben Johnson · Carey Priebe

Helivan · Hugging Face · Jataware · Johns Hopkins University

Research Digest·3 November 2026·2 min read

Illustration: Unpaired evaluations predict model benchmarks with 8x fewer queries

In one line

Representing models by similarities among their responses to similar, not identical, queries predicts benchmark scores from sparse, unpaired evaluation data.

Before you read this

An idea this paper builds on

benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.297 papers →

Read it as

Testing a generative model (software that produces text or other outputs) on every question in every (standard test set) is too costly. Evaluators often test different models on different subsets of questions, leaving a sparse, unpaired record: many scores are missing and responses do not line up question by question.

Product Kernel Perspective Space (PKPS) turns each model into a low-dimensional fingerprint built from its cached responses. It compares each response with responses to similar questions, not only identical ones, so it can use records that do not overlap.

PKPS matched sample scoring with up to eight times fewer queries, and combined with low-rank matrix completion, a missing-entry estimator, cut score-matrix completion error by up to 28 percent.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

benchmark prediction kernel perspective space unpaired evaluation data matrix completion multidimensional scaling benchmark