Description:

Abstract: Modern deep learning algorithms involve careful hyperparameter tuning to achieve the best performance. With large amounts of similar tasks and related data available, pre-trained foundation models are ubiquitous, but can we also “pre-train” the hyperparameters? A key challenge is that the network’s performance is a volatile function of the hyperparameters, and the usual assumptions of convexity and smoothness fail to apply. We consider two broad classes of hyperparameters — model hyperparameters that are part of the network architecture and optimization hyperparameters that only impact the training procedure. We develop novel techniques for bounding the pseudo-dimension of a function class involving optimization of piecewise-polynomial functions, implying sample complexity guarantees for configuring model hyperparameters like neural network activation functions. We further show how to provably tune optimization hyperparameters like learning rate and learning schedules, extending the results of Gupta and Roughgarden (2016) beyond smooth, convex regimes to practical non-convex settings including neural networks.

Bio: Dravyansh (Dravy) Sharma is an IDEAL postdoctoral researcher, hosted by Avrim Blum at TTIC and Aravindan Vijayaraghavan at Northwestern University. He obtained his PhD at Carnegie Mellon University, advised by Nina Balcan. His research interests include machine learning theory and algorithms, with a focus on provable hyperparameter tuning, adversarial robustness, and learning in the presence of rational agents. His work develops principled techniques for tuning fundamental machine learning algorithms to domain-specific data, including decision trees, linear regression, graph-based learning and, most recently, deep networks. He has published several papers at top ML venues, including NeurIPS, ICML, COLT, JMLR, AISTATS, UAI and AAAI, has multiple papers awarded with Oral presentations, won the Outstanding Student Paper Award at UAI 2024, and has interned with Google Research and Microsoft Research. He has presented a tutorial at UAI 2025 and an accepted joint tutorial at NeurIPS 2025.