Abstract:  

I will discuss some of the challenges of high dimensional inference: the type of learning problems that involve large numbers of model parameters and finite sample sizes. Neural networks are often high dimensional in practice, and by applying ideas from random matrix theory and dynamical systems, we can better understand when over-training and overfitting will occur in practice. This approach also helps to explain why generalization error can remain low even in neural networks where the number of parameters dwarfs the number of samples. I then provide some algorithmic lessons and inspirations drawn from these theories for new approaches to initializing and training neural networks.