Description:

Talk Title: How is it Possible to Train Deep Neural Networks?

Abstract:

In 1961, Minsky, one of the founders of AI, perceived a fundamental flaw within the burgeoning field of artificial neural networks. He doubted that such a nonlinear system could be effectively trained using gradient methods, because unless the “structure of the search space is special, the optimization may do more harm than good.” Fast forward to today, and we observe deep neural networks — far more complex than those envisioned at the field's inception — being successfully trained with methods akin to gradient descent. It has, indeed, become evident that the objective function displays a highly benign structure that we are only starting to comprehend. In this lecture, I aim to summarize our current understanding of this enigmatic optimization process. I will discuss several themes, including intrinsic dimensionality, the optimization landscape, and implicit regularization, all within the context of deep networks and large language models.

Speaker Bio:

Jorge Nocedal is the Walter P. Murphy Professor of Industrial Engineering and Management Sciences and (by courtesy) Engineering Sciences and Applied Mathematics at Northwestern University. Nocedal is also the Director of the Center for Optimization and Statistical Learning. Nocedal's main area of research is optimization, with applications in machine learning, engineering design, and the physical sciences. Research activities range from the design of new algorithms, to their software implementation and mathematical analysis. Areas of emphasis include large scale problems (with millions of variables), optimization under uncertainty, and parallel computing.

Learn more about Jorge Nocedal's research

The NSF-Simons National Institute for Theory and Mathematics in Biology Seminar Series aims to bring together a mix of mathematicians and biologists to foster discussion and collaboration between the two fields. The seminar series will take place on Fridays from 10am - 11am at the NITMB offices in the John Hancock Center in downtown Chicago. There will be both an in-person and virtual component.