Safe Offline Reinforcement Learning
via Chance-Constrained Policy Filtering
Accepted to the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026)
Video
Abstract
This paper proposes a chance-constrained policy filtering method for safe offline reinforcement learning (RL) designed to limit the probability of safety violation to a user-specified threshold. The proposed approach constructs a safety-filtered policy that accepts or rejects candidate actions based on a learned cost estimator. To support satisfaction of the target safety constraint, we further calibrate the estimator via posterior computation using the offline dataset. The proposed method offers significant flexibility, allowing for dynamic adjustment of constraint thresholds during deployment without additional retraining. We empirically validate our method on offline RL robot navigation tasks in the OSRL benchmark, demonstrating that it meets user-specified thresholds while achieving additional performance gains via end-to-end back-propagation.
This figure illustrates the overall pipeline of our approach.
Our method consists of three components:
(i) a cost estimator that predicts whether executing action is safe (A),
(ii) a filtering rule that constructs a deployed policy by accepting or rejecting actions based on the estimator’s output (A), and
(iii) a posterior computation and calibration procedure that adjusts the estimator so that the resulting filtering rule satisfies the target safety constraint (B).
Finally, we optionally fine-tune the cost estimator via back-propagation to further improve performance (C).