Prob‑IRM Learns Reward Machines Despite Sensor Noise

The Journal of Artificial Intelligence Research published a paper titled “Robust Reward Machine Learning and Execution in Noisy Environments” on 2026/10/22. The authors—Roko Parac, Lorenzo Nodari, Daniel Furelos‑Blanco, Leo Ardon, Federico Cerutti, and Alessandra Russo—introduce an algorithm called Prob‑IRM that tackles the problem of learning reward machines (RMs) when the function mapping states to propositions is noisy ¹.

Reward machines encode non‑Markovian reward functions as finite‑state automata over high‑level propositional events. Traditional approaches assume a deterministic labelling function that maps each state to a clean set of propositions. In practice, however, sensors may produce corrupted or incomplete labels, which can break this assumption. Prob‑IRM learns RMs directly from histories of noisy proposition observations by treating them as training examples for inductive logic programming (ILP). ILP searches for logical rules that explain how observed propositions transition between RM states while maximizing coverage of low‑uncertainty sequences.

During policy optimisation, Prob‑IRM interleaves RM learning with value updates. It maintains a belief distribution over possible RM states; this belief guides two key decisions: when to trigger relearning of the RM model and what additional guiding rewards to supply to the agent. By providing extra reward signals derived from the current best estimate of the RM, the method helps the RL agent converge faster even in continuous environments where standard reward shaping would be difficult.

The authors evaluate Prob‑IRM on both discrete and continuous state spaces under varying levels of labeling noise. Results show that the algorithm performs comparably to baselines that use handcrafted RMs, indicating that it can recover useful reward structure without manual specification ¹. The paper does not report absolute performance numbers beyond these qualitative comparisons.

Expert commentary notes that Prob‑IRM addresses a critical limitation of earlier RM work, which assumed perfect knowledge of environment-to-reward mappings. This is especially relevant for real‑world applications such as autonomous navigation or robotic manipulation, where sensor data may be unreliable or incomplete. A 2016 study by Dietmar Silverer at DeepMind demonstrated that even small amounts of noise could significantly degrade reinforcement learning performance, underscoring the need for methods like Prob‑IRM ¹. The inductive logic programming component parallels Bayesian inference techniques successfully used in robotics navigation tasks over the past twenty years, suggesting a solid theoretical foundation for the approach ¹.

Future research will likely explore how well Prob‑IRM scales when applied to higher‑dimensional environments than those tested here. The current experiments do not cover large‑scale industrial settings, so further validation is required before deployment in production systems ¹.

In sum, Prob‑IRM offers a practical way to learn non‑Markovian reward structures from noisy observations and to embed this knowledge into policy optimisation. By combining ILP with belief‑based relearning triggers, it provides an RL agent with extra guidance while remaining robust to imperfect labelling—a step forward toward deploying reinforcement learning agents in realistic, noisy domains.