Prob‑IRM learns reward machines despite noisy sensors

Reward machines (RMs) encode non‑Markovian rewards using finite‑state automata over high‑level propositions that are normally derived by a deterministic labeling function mapping environment states to proposition sets. In real‑world deployments, however, such labeling functions suffer from noise caused by unreliable or incomplete sensor streams. The JAIR paper “Robust Reward Machine Learning and Execution in Noisy Environments” introduces Prob‑IRM, an approach that learns RMs directly from noisy proposition histories with inductive logic programming (ILP). ILP treats each history as a training example and searches for rules that maximize coverage of low‑uncertainty sequences while remaining consistent with higher‑noise observations. During policy optimization, Prob‑IRM maintains a belief over RM states; when evidence suggests the current model is insufficient it triggers relearning, providing additional guiding rewards based on the updated beliefs. Experiments on environments with both discrete and continuous state spaces show that Prob‑IRM performs comparably to baselines that use handcrafted RMs, indicating its effectiveness even under varying levels of labeling noise ¹.

Expert analysis notes that this work addresses a key limitation of earlier methods which assumed perfect knowledge of the state‑to‑reward mapping. Similar challenges were highlighted by Dietmar Silverer at DeepMind in 2016, where small amounts of noise degraded reinforcement learning performance. By leveraging ILP, Prob‑IRM can infer likely reward functions from observed behavior despite noisy inputs—a technique akin to Bayesian inference used successfully in robotics navigation tasks more than twenty years ago. Further research will need to explore how the method scales to higher‑dimensional environments beyond those tested here.