Description:

Title: Applications of Power Constrained Nonstationary Multi-Armed Bandits in Personalized Healthcare.

Abstract: A common challenge for decision makers is selecting actions whose rewards are unknown and evolve over time based on prior policies. For instance, repeated use may reduce an action’s effectiveness (habituation), while inactivity may restore it (recovery). These nonstationarities are captured by the Reducing or Gaining Unknown Efficacy (ROGUE) bandit framework, which models real-world settings such as behavioral health interventions. While existing algorithms can compute sublinear regret policies to optimize these settings, they may not provide sufficient exploration due to overemphasis on exploitation, limiting the ability to estimate population-level effects. This is a challenge of particular interest in micro-randomized trials (MRTs) that aid researchers in developing  just-in-time adaptive interventions that have population-level effects while still providing personalized recommendations to individuals.  We first develop ROGUE-TS, a Thompson Sampling algorithm tailored to the ROGUE framework, and provide theoretical guarantees of sublinear regret. We then introduce a probability clipping procedure to balance personalization and population-level learning, with quantified trade-off that balances regret and minimum exploration probability. Validation on an MRT dataset concerning physical activity promotion shows that our methods both achieve lower regret than existing approaches and maintain high statistical power through the clipping procedure without significantly increasing regret. This enables reliable detection of treatment effects while accounting for individual behavioral dynamics. For researchers designing MRTs, our framework offers practical guidance on balancing personalization with statistical validity. We demonstrate how prior data, such as from pilot studies, can be leveraged to inform adaptive intervention delivery. In cases where interventions differ in burden or risk, tailored strategies can be used to prioritize either participant safety or learning objectives. Importantly, we show that statistical power can still be achieved with reduced exploration, enabling the design of more efficient and participant-friendly trials.

 Bio: Yonatan Mintz is an assistant professor in the Industrial and Systems Engineering department at the University of Wisconsin, Madison. His research focuses on the application of machine learning and automated decision making to human sensitive contexts. One application of his research has been on using patient level data, to create precision interventions . Yonatan is also interested in the sociotechnical implications of machine learning algorithms and has done work on fairness, accountability, and transparency in automated decision making.  In terms of methodology his research explores topics in machine learning theory, stochastic control, reinforcement learning, and nonconvex optimization. Yonatan's work has been recognized as a finalist in the INFORMS Health Applications Society Pierskalla Paper competition, a best poster award from the NeurIPS joint workshop on AI for Social Good, and he has been actively invited to publicly speak about his work in both print and televised media including PBS. His research has been funded by multiple awards from the National Institutes of Health (NIH) and American Family Insurance. Prior to joining UW--Madison, Yonatan was a postdoctoral research fellow at the department of Industrial and Systems Engineering at the Georgia Institute of Technology. Yonatan received his B.S. in Industrial and Systems Engineering with a concentration in Operations Research from Georgia Tech in 2012, and his Ph.D. in Industrial Engineering and Operations Research from the University of California, Berkeley in 2018.