2023 • Conference Paper
Coordinate descent on the Stiefel manifold for deep neural network training
Authors:
Massart, Estelle ,
Abrol, Vinayak
Published in:
31st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning.
To alleviate the cost incurred by orthogonality constraints in optimization and model training, we propose a stochastic coordinate descent algorithm on the Stiefel manifold. We compute expressions for geodesics on the Stiefel manifold with initial velocity aligned with coordinates of the tangent space and show that, analogously to the orthogonal group, iterate updates of coordinate descent methods can be efficiently implemented in terms of multiplications by Givens matrices. We illustrate our proposed algorithm on deep neural network training.