Description:

Bruno Fava (Northwestern):  A Central Limit Theorem for Split-Sample Estimators

Abstract: As artificial intelligence grows in popularity an increasing number of researchers policymakers and industry practitioners are tasked with using the same dataset to both train a model and evaluate some of its properties such as accuracy or fairness. Conducting each task with separate random splits of the data facilitates inference but comes at the cost of using fewer data at each step. Averaging over multiple splits such as with cross-fitting uses more data for each task and leads to better reproducibility properties but inference is challenging due to the statistical dependence among splits. I address this challenge by proving a new Central Limit Theorem for a large class of split-sample estimators under arguably mild and general conditions. Importantly I make no restriction on the model complexity convergence rate or rate of algorithmic stability. I show that inference based on the normal approximation may be invalid if any moment equation converges at a rate faster than sqrt(n) and provide an adaptive test that is asymptotically valid in both situations. Moreover I provide a measure of reproducibility for p-values obtained from split-sample estimators and show its asymptotic validity.