New JAIR Papers on Reward Machines, Bayesian Bandits, Explainable RL

The Journal of Artificial Intelligence Research (JAIR) released three papers on October 22, 2026 that address different challenges in AI research. Each paper offers new methods and insights that build on earlier work while opening avenues for future study.

Robust Reward Machine Learning and Execution in Noisy Environments – Roko Parac, Lorenzo Nodari, Daniel Furelos‑Blanco, Leo Ardon, Federico Cerutti, Alessandra Russo describe Prob‑IRM, an approach that learns reward machines (RMs) when the labeling function that maps states to propositions is noisy. The method uses inductive logic programming (ILP) with background knowledge to learn an RM that maximises coverage of low‑uncertainty histories. ILP examples are derived from noisy proposition histories; the learned RM interleaves with policy optimisation so the agent can adapt its reward representation as it learns. A belief over RM states guides relearning and provides additional rewards. Experiments show Prob‑IRM performs comparably to baselines that use handcrafted RMs across discrete and continuous environments ¹.

Expert commentary notes that this interleaving prevents drift caused by changing dynamics observed previously in other RL methods such as those described by Sutton & Barto (2018). The authors argue the technique could improve robustness in domains like robotic control where sensor noise is common.

Empirical Bayesian Multi‑Bandit Learning – Xia Jiang and Rong J. B. Zhu present a hierarchical Bayesian framework for learning across multiple contextual bandits. Their model captures both heterogeneity and dependence among bandit instances through a prior whose covariance matrix is estimated empirically from data. Two algorithms, ebmTS and ebmUCB, incorporate this learned prior into Thompson sampling and upper confidence bound decision making respectively. Frequentist regret bounds are established for both algorithms.

The expert notes highlight that estimating covariance directly from data improves upon traditional conjugate priors used in earlier bandit algorithms. By extending Ghahramani’s 2005 hierarchical models, the new framework maintains theoretical guarantees while offering practical benefits. Such improvements may lead to more effective personalized recommendation engines capable of adapting quickly when user preferences change rapidly.

A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs – Léo Saulières surveys over 250 papers on explainable reinforcement learning (XRL) and proposes an intuitive taxonomy based on two questions: “What” (the target of explanation) and “How” (the method of providing explanations). The survey also identifies related domains that should receive attention and outlines needs such as better evaluation metrics beyond simple human comprehension scores because current measures may be insufficient to assess whether explanations are truly actionable or causally informative—a limitation highlighted by Lipton (2019).

Saulières’ taxonomy offers a useful structure but does not fully capture challenges associated with counterfactual explanations, which involve reasoning about what actions would have happened under different circumstances. The survey correctly points out the need for improved metrics that can evaluate causal informativeness of XRL methods.

These three JAIR contributions illustrate continued progress across AI subfields—from robust reward representation under noisy conditions, through principled Bayesian sharing among multi‑bandits, to systematic reviews of explainability in RL—each building on prior research while pointing toward future directions.