AI chips can hide most of flash memory’s read delays

In GPU simulations, the authors’ prefetching design reaches 92.4% of an idealized low-latency flash system’s performance while using a much smaller buffer than a competing design.

Big Tech

Anish Saxena · Hritvik Taneja · Christos Kozyrakis · Aamer Jaleel

Georgia Tech · NVIDIA Research · Stanford University

Research Digest·13 November 2026·2 min read

Illustration: AI chips can hide most of flash memory’s read delays

In one line

SlipStream prefetcher achieves 92.4% of ideal HBF performance with a 40 MB staging buffer for LLM inference.

Before you read this

Two ideas this paper builds on

large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.306 papers →mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.83 papers →

Read it as

(LLMs) need a lot of memory for their weights. A new technology called High-Bandwidth Flash (HBF) offers much more capacity than current HBM memory, but reads are about twenty times slower, which can slow inference dramatically.

SlipStream is a co-designed hardware and software prefetcher. Software profiles the exact order LLM kernels access weight pages and creates a replay stream. A special warp fetches pages from HBF early. Hardware tracks each page until it is first demanded, preventing prefetch from running too far ahead and using buffer space efficiently.

With a 40-megabyte staging buffer, SlipStream achieves 92.4% of ideal HBF performance. Even with 16 megabytes it reaches 88.8%. This makes HBF practical for GPUs, enabling larger models without huge slowdowns.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

prefetching large language model high-bandwidth flash software-hardware co-design demand-paced staging mixture of experts