Seminar Details
2026-09-15 (14:00) : LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
At Euler building (room A.002)
Duration: 60 minutes
Organized by Mathematical Engineering
Speaker :
Cédric Simal (University of Namur)
Abstract :
The Muon optimizer has been making waves as a contender for being the optimizer of choice for training large scale neural networks. Unlike other methods, it explicitly leverages the matrix structure of parameters by performing steepest descent with respect to an operator norm. Over the last year, several works have adapted Muon to various matrix manifold constraints, and we focus in particular on spaces of low-rank matrices, which is of interest for LoRA, a popular fine-tuning method for neural networks. Our method is invariant under reparametrization symmetries, and achieves learning rate transfer across rank, width, depth and factor rescaling.