Seminar Details
2024-02-13 (14h) : Relationship between sample size and architecture for the estimation of Sobolev functions using deep neural networks
At Euler building (room A.002)
Organized by Mathematical Engineering
Speaker :
Stéphane Chretien (University of Lyon 2)
Abstract :
Beyond the many successes of Deep Learning based techniques in various branches of data analytics, medicine, business, engineering and the human sciences, a sound understanding of the generalisation properties of these techniques is still elusive. Central to these successes are the availability of huge datasets and the availability of huge computational ressources and some of the most recent trends have given paramount importance to the necessity of building huge neural networks with millions of parameters, and most often, of several orders of magnitude larger than the size of the training set. This set-up has however led to many surprises and counterintuitive discoveries. Overprametrisation was recently shown to favour connectivity in a weak sense of the set of stationary points, hence permitting stochastic gradient type methods to potentially reach good minimisers in several stages despite the wild nonconvexity of the training problem as demonstrated by Kuditipudi et al. Relating generalisation to stability, recent theoretical breakthroughs have been able to provide a better understanding of why generalisation cannot even happen without overparametrisation as shown by Bubeck et al. Following the ideas developed by Belkin, a substantial amount of work has also been undertaken in order to study the double descent phenomenon, and the associated benign overfitting property which holds for least norm estimators in linear and mildly non-linear regression, as well as in for certain kernel based methods. In the present paper, we aim at studying the generalisation properties of overparametrised deep neural networks using a novel approach based on Neuberger's theorem.