Safety from Weak Feedback: Inverse Constraint Reinforcement Learning for Safe Control and Safe Online Adaptation

This NSF project aims to make learning-based autonomous systems provably safe in the realistic cases where the rules defining "safe behavior" are never written down and cannot easily be specified at all. Today's methods for safe machine learning usually assume someone can hand the machine a precise, moment-by-moment definition of danger. In the real world, machines instead receive only coarse, after-the-fact feedback—whether an entire trip ended safely or came dangerously close to failing—and must work out the hidden safety limits for themselves. The project will establish the mathematical foundations for a machine to discover its own safety limits from weak feedback, to recognize honestly what it does not yet know, and to act, explore, and improve without ever taking unacceptable risks. This will be achieved by developing theory that determines when safety can even be learned from such limited feedback, methods that measure uncertainty specifically where it matters for staying safe, and a novel strategy that turns a shortage of data into a controllable process of cautiously "probing" to reveal the safety boundary. The intellectual merit of the project includes advancing the fundamental understanding of when safety constraints can be learned from limited feedback and how to guarantee safe behavior despite that uncertainty. The broader impacts of the project include strengthening the reliability of autonomous systems in healthcare, transportation, and other safety-critical settings, along with open educational resources, a free online course, public software and benchmarks, and research training for students from high school through the doctoral level. The central technical challenge in safe machine learning is that safety specifications are often unknown and observed only through weak, trajectory-level feedback—a single safe/unsafe label per attempt, rather than a detailed cost at every step. This makes the problem an inverse one: the system must reconstruct hidden safety limits without knowing where along a trip a violation occurred, and available data rarely approaches the true danger boundary. The project addresses this through three connected directions: first, establishing when hidden safety limits can be identified at all and building calibrated confidence sets that capture what remains uncertain; second, using those sets to compute policies that stay safe under worst-case uncertainty while taking small, deliberately low-risk actions that most efficiently reduce it; and third, enabling systems to keep improving after deployment with near-zero safety violations, even as conditions drift. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.