STL‑Based Rewards Improve Model‑Free RL

Singh and Saha present a new framework that derives reward functions directly from formal specifications written in Signal Temporal Logic (STL) for model‑free reinforcement learning agents. The approach uses a quantitative semantics of STL to convert safety and liveness properties—such as “always maintain safe distance”—into numerical values suitable for gradient‑based optimization, enabling agents to satisfy these properties more reliably than methods relying purely on trial and error exploration.¹

The authors evaluate their method on several complex continuous control benchmarks, showing the new semantics outperform existing techniques across multiple tasks. Experimental results indicate higher success rates while respecting safety constraints consistently compared with baseline approaches.¹

By grounding rewards in formal specifications, designers avoid manual tuning that may inadvertently incentivize unintended behaviors. This reduces the risk of reward hacking, where agents exploit loopholes in handcrafted signals. The approach also facilitates modular verification: once an agent satisfies its STL‑derived reward criteria, one can formally prove adherence to the original specification using established model‑checking tools. Such synergy between learning and formal methods is particularly relevant for cyber‑physical systems such as autonomous vehicles or industrial robots, where safety guarantees are paramount.¹

A key caveat noted by the authors—and echoed in expert commentary—is the limited expressiveness of STL. While powerful for many continuous control properties, it cannot encode all desirable agent behaviours, especially those requiring complex combinatorial logic or probabilistic reasoning. Consequently, the reward generation pipeline may not generalize to domains lacking precise temporal constraints. Future work could explore hybrid logics combining STL with other specification languages to broaden applicability.