Northwestern University

Feb
11
Mon 11:00 AM

Grant Rotskoff: Neural networks as interacting particle systems: understanding global convergence of parameter optimization dynamics

When: Monday, February 11, 2019
11:00 AM - 12:00 PM  

Where: Technological Institute, M416, 2145 Sheridan Road, Evanston, IL 60208 map it

Audience: Faculty/Staff - Student - Public - Post Docs/Docs - Graduate Students

Contact: Jawaad Ahmad   847.491.3345

Group: McCormick-Engineering Sciences and Applied Mathematics

Category: Lectures & Meetings

Description:

Title: Neural networks as interacting particle systems: understanding global convergence of parameter optimization dynamics

Speaker: Grant Rotskoff, James S. McDonnell Fellow, Courant Institute, New York University

Special Note: Note the unusual time

Abstract: The performance of neural networks on high-dimensional data distributions suggests that it may be possible to parameterize a representation of a target high-dimensional function with controllably small errors, potentially outperforming standard interpolation methods. We demonstrate, both theoretically and numerically, that this is indeed the case. We map the parameters of a neural network to a system of particles relaxing with an interaction potential determined by the loss function. This mapping gives rise to a deterministic partial differential equation that governs the parameter evolution under gradient descent dynamics. We also show that in the limit that the number of parameters n is large, the landscape of the mean-squared error becomes convex and the representation error in the function scales link n^{-1}. In this limit, we prove a dynamical variant of the universal approximation theorem showing that the optimal representation can be attained by stochastic gradient descent, the algorithm ubiquitously used for parameter optimization in machine learning. This conceptual framework can be leveraged to develop algorithms that accelerate optimization using non-local transport. I will conclude by showing that using neuron birth/death processes in parameter optimization guarantees global convergence and provides a substantial acceleration in practice.

-----

To subscribe to the Applied Mathematics Colloquia List send a message to seminar-join@esam.northwestern.edu

Add Event to Calendar

Add Event To My Group:

Please sign-in