Prob‑IRM Learns Reward Machines from Noisy Data

Researchers have released a novel approach named Prob‑IRM that tackles one major practical hurdle for reward machine learning—noisy labeling functions that map raw sensor readings to high-level propositions. The technique is described in a paper by Roko Parac, Lorenzo Nodari, Daniel Furelos-Blanco, Leo Ardon, Federico Cerutti and Alessandra Russo, published on 2026/10/22 ¹.

Reward machines (RMs) encode non-Markovian rewards as finite-state automata driven by propositional events. Traditional methods assume these events come from a deterministic mapping of environment states; however, real-world sensors often provide noisy or missing information.

Prob‑IRM addresses this gap by treating noisy proposition histories as training examples for inductive logic programming (ILP). ILP learns an RM that maximises coverage of low uncertainty histories while interleaving policy optimisation with RM refinement. A belief over RM states guides relearning decisions and supplies auxiliary guiding rewards.

The authors evaluated Prob‑IRM across discrete and continuous environments under varying levels of labeling noise. Results show it performs comparably to baselines using handcrafted RMs, indicating the method can recover accurate reward structures without manual engineering.

Expert context highlights why this development matters. Prior work at DeepMind in 2016 demonstrated how even small amounts of sensor noise could severely degrade reinforcement learning performance—a problem long affecting robotics and autonomous vehicles. By leveraging ILP, Prob‑IRM brings a form of Bayesian inference proven effective in decades-old navigation algorithms, allowing the system to infer likely reward functions despite imperfect inputs.

However, experts caution that further research is needed to assess scalability beyond the tested scenarios. Current experiments involve relatively modest state spaces; extending the approach to high-dimensional real-world domains will require additional algorithmic refinements and computational resources. Moreover, while Prob‑IRM matches hand-crafted baselines on synthetic benchmarks, its effectiveness in safety-critical applications remains unverified until extensive field testing.

In practice, Prob‑IRM offers an attractive path toward deploying reward machines in noisy settings such as industrial automation or mobile robot exploration where sensors cannot guarantee perfect fidelity. If future studies confirm its robustness, the technique may reduce the need for costly human-designed reward specifications and accelerate adoption of RL agents in complex environments.

The paper’s contribution lies not only in the specific method but also in framing reward machine learning under realistic assumptions about data quality—a step many researchers have called essential for bridging theory with application. As AI systems increasingly operate amid uncertainty, techniques like Prob‑IRM may become central tools for building reliable, interpretable reinforcement learners.