The authors model requests as a radix tree, where requests with a shared prompt prefix follow the same path.
Auctions can prioritize LLM requests while preserving cache efficiency
The proposed mechanism orders cache-compatible request groups by bids, uses truthful pricing and supports budget-constrained automatic bidding.
Big Tech
Keegan Harris · Siddharth Prasad · Asher Trockman · Nika Haghtalab · Michael I. Jordan
University of California, Berkeley · Toyota Technological Institute at Chicago · Google Research · Inria & École Normale Supérieure
Research Digest··3 min read
Harris et al.
Why this paper
From Google Research and 3 others
In one line
An inference auction where users bid for priority increases welfare without harming latency or cache efficiency.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§