Quantitative models – for example, Reinforcement Learning (RL), Bayesian inference, or drift-diffusion models – are fundamental tools in cognitive science. Typically, cognitive models are hand-crafted to test specific hypotheses about the cognitive mechanism, making them explainable and interpretable. Empirically, however, “classic” cognitive models often provide poor fit to experimental data, leaving substantial amounts of explainable behavioral variance unexplained. Artificial neural networks (ANNs) provide the inverse profile, allowing for arbitrarily good fit at the cost of opaque mechanisms. Here, we propose a novel kind of cognitive model that unites the predictive power of ANNs with the interpretability of classic cognitive models, creating a class of “hybrid” RL-ANN models. Employing a classic, 4-arm drifting bandit task in humans, we first show that this approach improves model comparison: RL-ANNs are no longer limited to testing specific, hand-crafted cognitive mechanisms, but reveal how much additional variance can be explained by adding any arbitrary cognitive mechanism. We then show that RL-ANNs provide interpretable explanations: We find evidence for RL-like value updating, which, however, is not Markovian, as in classic RL models, but shows long-term history dependence. This finding supports and formalizes previous notions that RL is intertwined with other, non-Markovian memory systems. We furthermore find evidence for a separate, value-independent action selection process that generalizes previous notions of choice stickiness and structured exploration. In summary, RL-ANNs hold the promise of explaining more variance in human behavior than classical methods, without sacrificing explanatory power: They provide tangible insights into value learning, memory, habitual control, and exploration. I would love to discuss possible implications of this research for psychiatric research during the meeting.

Maria Eckstein
Research Scientist
Deep Mind

Maria Eckstein – Predictive and Interpretable: Combining Artificial Neural Networks and Classic Cognitive Models to Understand Human Learning and Decision Making