Standard zeroth-order optimization probes a model with parameter perturbations and estimates an update from the resulting loss changes.
Activation-guided subspaces improve forward-only fine-tuning of large language models
AIM-ZO accumulates activation-derived directions across training, then samples smaller subspaces for efficient parameter updates without backpropagation.
Chinese Tech
Yue Xie · Zhi Zheng · Yunpeng Ba · Xuyang Wu · Xialiang Tong · Zhichao Lu · +2 more
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. · City University of Hong Kong
Research Digest··2 min read
Xie and colleagues introduce AIM-ZO, a zeroth-order fine-tuning method that estimates model updates using only forward loss evaluations.
Why this paper
From Huawei Technologies Ltd. and 3 others
In one line
AIM-ZO improves zeroth-order LLM fine-tuning by using forward activations to maintain an evolving subspace for perturbations.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§