Skip to main content

NUTC Seminar Series: "VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving" - Sikai Chen, University of Wisconsin-Madison

Thursday, October 29, 2026 | 4:00 PM - 5:00 PM CT
Chambers Hall, Ruan Conference Center, 600 Foster St, Evanston, IL 60208 map it

Abstract:

 Reinforcement learning (RL) lets autonomous vehicles learn to drive through trial and error, but its success depends on the reward function, the rule that tells the vehicle what counts as “good driving”. Today these rewards are mostly hand-designed by experts. That takes a lot of effort and often fails to capture what safe driving means in complex traffic. Vision-language models (VLMs), trained on huge amounts of images and text, carry broad knowledge about how the world works, such as how objects, actions, and their outcomes relate to one another. This built-in knowledge offers a new way to tackle this problem. This talk will first discuss the challenges of reward design in RL-based autonomous driving and the opportunities that foundation models bring. It will then introduce VLM-RL, a framework that uses pre-trained VLMs to turn plain-language driving goals into reward signals for RL. Humans learn well from both good and bad examples, and the framework works the same way. It compares what the vehicle sees against a positive goal (e.g., “the road is clear with no car accidents”) and a negative goal (e.g., “two cars have collided with each other”). It then combines this feedback with basic vehicle state information to produce stable, informative reward signals. Next, the talk will show how well the framework works through experiments in the CARLA simulator, where compared with existing methods, VLM-RL lowered collision rates, raised route completion, and generalized to driving scenarios it had not seen before. Finally, the talk will discuss future directions for bringing foundation models into safe, human-centered autonomous driving, including moving from simulation to real world.

 

Bio:

 Dr. Sikai (Sky) Chen is an Assistant Professor in the Department of Civil and Environmental Engineering at the University of Wisconsin-Madison, with a courtesy appointment in the Department of Mechanical Engineering. He also serves as the Associate Director of the university's Smart Highway Research Center (SHRC), leading the Digital Infrastructure, Data Science, and Artificial Intelligence Program. He received his Ph.D. in Civil Engineering, with a concentration in Computational Science & Engineering, from Purdue University. His research centers on three interconnected themes—human users, AI, and transportation—with the goal of developing safe, efficient, resilient, and human-centered transportation systems by integrating human behaviors, interactive autonomy, digital infrastructure, and domain-specific foundation models. Dr. Chen is one of the fifteen Stage 1 winners of the USDOT ARPA-I Ideas and Innovation Challenge, and his research has received multiple national and international recognitions, including the Matthew G. Karlaftis Best Paper Award and the Milton PikarskyMemorial Award (a national award for the best Ph.D. dissertation in transportation studies). He serves as PI or co-PI on research projects funded by USDOT, WisDOT, WARF, and industry partners such as NVIDIA. He is a member of two ASCE national committees (Connected & Autonomous Vehicle Impacts, and Economics & Finance) and the IEEE Emerging Transportation Technology Testing Technical Committee. He previously served as a member and Committee Communications Coordinator of the TRB Standing Committee on Statistical and Econometric Methods (AED60).
 

Audience

  • Faculty/Staff
  • Student
  • Public
  • Post Docs/Docs
  • Graduate Students

Contact

Andrea Cehaic
(847) 491-7287
Email

Interest

  • Academic (general)

Add Event To My Group

Please sign-in