ToM-Dyna-Q: on the integration of reinforcement learning and machine Theory of Mind

Abstract

The capacity to understand others, or to reason about others’ ways of reasoning about others (including us), is fundamental for an agent to survive in a multi-agent uncertain environment. This reasoning ability, commonly known as Theory of Mind, is instrumental for making effective predictions over others’ future actions and learning from both real and simulated experience. In this work, a novel architecture for model-based reinforcement learning in a multi-agent setting is proposed. The proposed architecture, called ToM-Dyna-Q, integrates ToM simulation alongside with the well-known Dyna-Q architecture to account for artificial cognition in a shared environment inhabited by multiple agents interacting with each other. Results obtained for the two-player competitive game of Tic-Tac-Toe demonstrate the importance for a given agent of learning, reasoning and planning based on mental simulation modeling of other agents’ goals, beliefs and intentions.