When you ask many questions about the same long document, current AI systems often process each question separately. Even though they only read the document once to prepare a shared memory (key-value cache), every time they generate a new word for each question, they have to fetch that entire memory again. This repeated memory access becomes a bottleneck as the number of questions grows.
The researchers developed Parallel Prompting, which combines multiple questions into one request with parallel streams. Each stream generates its answer independently, but all share the document's key-value cache. The system reads the cache only once per generation step, regardless of how many questions are being answered, using a special attention mask that keeps each answer focused on its own question and the document.
On an A100 GPU, Parallel Prompting is 3.7 times faster than existing systems for long documents with many questions, while maintaining answer quality. This speedup matters for real-world applications like legal document analysis or book QA, where dozens of questions are asked about the same text, reducing cost and latency.