Fusing adapters into low-precision GEMMs cuts latency for AI serving and training

EpiLoRA hides adapter compute inside the base model's matrix multiplication, improving decode speed by up to 24% and enabling 27B models to train on a single GPU

Independent

Hieu Tran · Hong Yu

Research Digest·4 November 2026·3 min read

Illustration: Fusing adapters into low-precision GEMMs cuts latency for AI serving and training

In one line

EpiLoRA fuses LoRA adapters into low-precision GEMM kernels, cutting serving latency and enabling RL training on FP8 weights.

Before you read this

Two ideas this paper builds on

reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.256 papers →post-training quantizationQuantizing pre-trained neural network weights and activations to44 papers →

Read it as

When you ask a large language model for text, it's often a base model customized with a small adapter for your specific task. Current systems keep the adapter separate from the main math, so making the main math faster with lower-precision numbers doesn't speed up the adapter.

EpiLoRA is a plugin for the vLLM serving engine. It runs the adapter's first step while the main math loads its weights, then adds the adapter's second step at the very end of the main math's computation. This hides most of the adapter's work behind the faster main calculation.

On Hopper GPUs, decoding a single request took up to 24% less time per token than the fastest vLLM configuration. For many users sharing one model, it delivered more throughput than vLLM on most tested GPU-model pairs.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

low-rank adaptation low-precision general matrix multiplication multi-tenant serving epilogue fusion reinforcement learning post-training quantization