Publications 2017

  • doi pdf Theory-based computational psychiatry
  • Maia TV* and Huys QJM* and Frank MJ
  • Biological Psychiatry (2017) 82: 382--384
  • Editorial for special issue on computational psychiatry.
  • pdf Bayesian approaches to Learning and Decision Making
  • Huys QJM
  • in: Computational Psychiatry: Mathematical Modelling of Mental Disorders. Editors: Alan Anticevic and John Murray
  • Behavioural phenomena are central to psychiatric disorders. Computational modelling allows the learning and decision-making processes underlying behaviour to be modelled in great detail. By doing so, specific and possibly highly complex hypotheses about the underlying processes can be directly tested on the data. The first part of this chapter introduces Markov Decision Problems (MDPs) as a formal framework for decision-making. It then describes several solutions to MDPs including reinforcement learning and dynamic programming, and briefly introduces some of their key characteristics. The second part of the chapter provides a tutorial overview over how to use MDPs in a generative modelling framework to test hypotheses about learning and decision-making. The final part of the chapter discusses the methods using a few worked examples from the literature.
  • The neural basis of aversive Pavlovian guidance during planning
  • Lally* N, Huys* QJM, Eshel N, Faulkner P, Dayan P and Roiser JP
  • J. Neurosci. (2017) In Press
  • Important real-world decisions are often arduous as they frequently involve sequences of choices, with initial selections affecting future options. Evaluating every possible combination of choices is computationally intractable, particularly for longer multi-step decisions. Therefore, humans frequently employ heuristics to reduce the complexity of decisions. We recently used a goal-directed planning task to demonstrate the profound behavioral influence and ubiquity of one such shortcut, namely aversive pruning, a reflexive Pavlovian process that involves neglecting parts of the decision space residing beyond salient negative outcomes. However, how the brain implements this important decision heuristic, and what underlies individual differences in its strength have hitherto remained unanswered. Therefore, we administered an adapted version of the same planning task to healthy volunteers undergoing functional magnetic resonance imaging (fMRI) to determine the neural basis of aversive pruning. Through both computational and standard categorical fMRI analyses, we show that when planning was influenced by aversive pruning, the subgenual cingulate cortex was robustly recruited. This neural signature was distinct from those associated with general planning and valuation, two fundamental cognitive components elicited by our task but which are complementary to aversive pruning. Furthermore, we found that individual variation in levels of aversive pruning were associated with the responses of insula and dorsolateral prefrontal cortex to the receipt of large monetary losses, and also with sub-clinical levels of anxiety. In summary, our data reveal the neural signatures of an important reflexive Pavlovian processes that shapes goal-directed evaluations, and thereby determine the outcome of high-level sequential cognitive processes.
  • doi pdf A formal valuation framework for emotions and their control
  • Huys QJM and Renz D
  • Biological Psychiatry (2017) 82:413--420
  • Computational psychiatry attempts to apply mathematical and computational techniques to help improve psychiatric care. Here, we consider formal valuation accounts of emotion. The flexibility of emotional responses and the nature of appraisals suggest the need for a model-based framework for emotions. Resource limitations make plain model-based valuation impossible, and require strategies to apportion cognitive resources adaptively. We argue that emotions can implement such approximations by restricting the range of behaviours and states considered. We consider the processes that guide the deployment of the emotional approximations, discerning between innate, model-free, heuristic and model-based controllers. A focus on complex model-based decisions reveals the necessity for strategies to deal with the complexity of the problems. Emotions may provide such approximations, and this framework may provide a principled approach to examining them.
  • doi pdf Strong seduction - Impulsivity and the impact of contextual cues on instrumental behavior in alcohol dependence
  • Sommer C, Garbusow M, Jünger E, Pooseh S, Bernhardt N, Birkenstock J, Schad D, Jabs B, Glöckler T, Huys QJM, Heinz A, and Smolka M and Zimmermann US
  • Translational Psychiatry (2017) 7, e1183
  • Alcohol-related cues acquire incentive salience through Pavlovian conditioning and then can markedly affect instrumental behavior of alcohol-dependent patients to promote relapse. However, it is unclear whether similar effects occur with alcohol-unrelated cues. We tested 116 early abstinent alcohol-dependent patients and 91 healthy controls who completed a delay discounting task to assess choice impulsivity, and a Pavlovian-to-instrumental transfer (PIT) paradigm employing both alcohol-unrelated and alcohol-related stimuli. To modify instrumental choice behavior, we tiled the background of the computer screen either with conditioned stimuli (CS) previously generated by pairing abstract pictures with pictures indicating monetary gains or losses, or with pictures displaying alcohol or water beverages. CS paired to money gains and losses affected instrumental choices differently. This PIT effect was significantly more pronounced in patients compared to controls, and the group difference was mainly driven by highly impulsive patients. The PIT effect was particularly strong in trials in which the instrumental stimulus required inhibition of instrumental response behavior and the background CS was associated to monetary gains. Under that condition, patients performed inappropriate approach behavior, contrary to their previously formed behavioral intention. Surprisingly, the effect of alcohol and water pictures as background stimuli resembled that of aversive and appetitive CS, respectively. These findings suggest that positively valenced background CS can provoke dysfunctional instrumental approach behavior in impulsive alcohol-dependent patients. Consequently, in real life they might be easily seduced by environmental cues to engage in actions thwarting their long-term goals. Such behaviors may include, but are not limited to, approaching alcohol.
  • doi The impact of traumatic stress on Pavlovian biases
  • Ousdal OT*, Huys QJM*, Milde AM, Craven AR, Ersland L, Endestad T, Melinder A, Hugdahl K, Dolan RJ
  • Psychol Med (2017)
  • doi When habits are dangerous: alcohol expectancies and habitual decision-making in alcohol dependence
  • Sebold M, Nebe S, Garbusow M, Schad DJ, Beck A, Kuitunen-Paul S, Sommer C, Neu P, Zimmermann US, Rapp MA, Smolka MN, Huys QJM, Schlagenhauf F, Heinz A
  • Biological Psychiatry (2017)
  • Addiction is supposed to be characterized by a shift from goal-directed to habitual decision-making, thus facilitating automatic drug intake. The two-step task allows distinguishing between these mechanisms by computationally modelling goal-directed and habitual behavior as model-based and model-free control. In addicted patients, decision-making may also strongly depend upon drug-associated expectations. Therefore, we investigated model-based vs. model-free decision-making and its neural correlates as well as alcohol expectancies in alcohol-dependent patients and healthy controls and assessed treatment outcome in patients. Ninety detoxified, medication-free alcohol-dependent patients and 96 age- and gender-matched control underwent functional magnetic resonance imaging during the two-step task. Alcohol expectancies were measured with the Alcohol Expectancy Questionnaire.. Over a follow-up period of 48 weeks, 37 patients remained abstinent whereas 53 patients. Patients who relapsed displayed reduced medial prefrontal cortex (mPFC) activation during model-based decision-making. Furthermore high alcohol expectancies were associated with low model-based control in relapsers, while the opposite was observed in abstainers and healthy controls. However, reduced model-based control per se was not associated with subsequent relapse. These findings suggest that poor treatment outcome in addicted patients does not simply result from reduced model-based control but is rather dependent on the interaction between high drug expectancies and low model-based decision-making. Reduced model-based mPFC signatures in prospective relapsers point to a neural correlate of relapse risk. These observations suggest that therapeutic interventions should target subjective alcohol expectancies.
  • doi pdf No association of goal-directed and habitual control with alcohol consumption in young adults
  • Nebe S, Kroemer NB, Schad DJ, Bernhardt N, Sebold M, Müller DK, Scholl L, Kuitunen-Paul S, Heinz A, Rapp M, Huys QJ, Smolka MN
  • Addiction Biology (2017)
  • Alcohol dependence is a mental disorder which has been associated with an imbalance in behavioral control favoring model-free habitual over model-based goal-directed strategies. It is as yet unknown, however, whether such an imbalance reflects a predisposing vulnerability or results as a consequence of repeated and/or excessive alcohol exposure. We, therefore, examined the association of alcohol consumption with model-based goal-directed and model-free habitual control in 188 eighteen-year-old social drinkers in a two-step sequential decision-making task while undergoing fMRI before prolonged alcohol misuse could have led to severe neurobiological adaptations. Behaviorally, participants showed a mixture of model-free and model-based decision-making as observed previously. Measures of impulsivity were positively related to alcohol consumption. In contrast, neither model-free nor model-based decision weights nor the tradeoff between them were associated with alcohol consumption. There were also no significant associations between alcohol consumption and neural correlates of model-free or model-based decision quantities in either ventral striatum or ventromedial prefrontal cortex. Exploratory whole-brain fMRI analyses with a lenient threshold revealed early onset of drinking to be associated with an enhanced representation of model-free reward prediction errors in the posterior putamen. These results suggest that an imbalance between model-based goal-directed and model-free habitual control might rather not be a trait marker of alcohol intake per se.
  • doi pdf The anchoring bias reflects rational use of cognitive resources.
  • Lieder F, Griffiths TL, Huys QJM and Goodman ND
  • Psychonomic Bulletin & Review (2017)
  • Cognitive biases, such as the anchoring bias, pose a serious challenge to rational accounts of human cognition. We investigate whether rational theories can meet this challenge by taking into account the mind's bounded cognitive resources. We asked what reasoning under uncertainty would look like if people made rational use of their finite time and limited cognitive resources. To answer this question, we applied a mathematical theory of bounded rationality to the problem of numerical estimation. Our analysis led to a rational process model that can be interpreted in terms of anchoring-and-adjustment. This model provided a unifying explanation for ten anchoring phenomena including the differential effect of accuracy motivation on the bias towards provided versus self-generated anchors. Our results illustrate the potential of resource-rational analysis to provide formal theories that can unify a wide range of empirical results and reconcile the impressive capacities of the human mind with its apparently irrational cognitive biases.
  • doi pdf Empirical evidence for resource-rational anchoring and adjustment
  • Lieder F, Griffiths TL, Huys QJM and Goodman ND
  • Psychonomic Bulletin & Review (2017)
  • People's estimates of numerical quantities are systematically biased towards their initial guess. This anchoring bias is usually interpreted as sign of human irrationality, but it has recently been suggested that the anchoring bias instead results from people's rational use of their finite time and limited cognitive resources. If this were true, then adjustment should decrease with the relative cost of time. To test this hypothesis, we designed a new numerical estimation paradigm that controls people's knowledge and varies the cost of time and error independently while allowing people to invest as much or as little time and effort into refining their estimate as they wish. Two experiments confirmed the prediction that adjustment decreases with time cost but increases with error cost regardless of whether the anchor was self-generated or provided. These results support the hypothesis that people rationally adapt their number of adjustments to achieve a near-optimal speed-accuracy tradeoff. This suggests that the anchoring bias might be a signature of the rational use of finite time and limited cognitive resources rather than a sign of human irrationality.
  • doi pdf Value-based decision-making battery: A Bayesian adaptive approach to assess impulsive and risky behavior
  • Pooseh S, Bernhardt N, Guevara A, Huys QJM and Smolka MN
  • Behav Res Methods (2017)
  • Using simple mathematical models of choice behavior, we present a Bayesian adaptive algorithm to assess measures of impulsive and risky decision making. Practically, these measures are characterized by discounting rates and are used to classify individuals or population groups, to distinguish unhealthy behavior, and to predict developmental courses. However, a constant demand for improved tools to assess these constructs remains unanswered. The algorithm is based on trial-by-trial observations. At each step, a choice is made between immediate (certain) and delayed (risky) options. Then the current parameter estimates are updated by the likelihood of observing the choice, and the next offers are provided from the indifference point, so that they will acquire the most informative data based on the current parameter estimates. The procedure continues for a certain number of trials in order to reach a stable estimation. The algorithm is discussed in detail for the delay discounting case, and results from decision making under risk for gains, losses, and mixed prospects are also provided. Simulated experiments using prescribed parameter values were performed to justify the algorithm in terms of the reproducibility of its parameters for individual assessments, and to test the reliability of the estimation procedure in a group-level analysis. The algorithm was implemented as an experimental battery to measure temporal and probability discounting rates together with loss aversion, and was tested on a healthy participant sample.
  • doi pdf Predicting relapse after antidepressant withdrawal - a systematic review
  • Berwian IM, Walter H, Seifritz E, Huys QJM
  • Psychol. Med. (2017) 47(3):426-437
  • Background: A substantial proportion of the burden of depression arises from its recurrent nature. The risk of relapse after antidepressant (ADM) discontinuation is high but not uniform. Predictors of individual relapse risk after antidepressant discontinuation could help to guide treatment and mitigate the long-term course of depression.

    Methods: We conducted a systematic literature search in Pubmed to identify relapse predictors using the search terms "(depress* OR MDD*) AND (relapse* OR recurren*) AND (predict* OR risk) AND (discontinu* OR withdraw* OR maintenance OR maintain or continu*) AND (antidepress* OR medication OR drug)" for published studies until November 2014. Studies investigating predictors of relapse in patients aged between 18 and 65 with a main diagnosis of Major Depressive Disorder (MDD) who remitted from a depressive episode while treated with antidepressant medication and were followed up for at least 6 months to assess relapse after part of the sample discontinued their ADM, were included in the review.

    Results: Although relevant information is present in many studies, only thirteen studies based on nine separate samples investigated predictors for relapse after ADM discontinuation. There are multiple promising predictors, including markers of true treatment response and the number of prior episodes. However, the existing evidence is weak and there are no established, validated markers of individual relapse risk after antidepressant cessation.

    Conclusion: There is little evidence to guide discontinuation decisions in an individualized manner beyond overall recurrence risk. Thus, there is a pressing need to investigate neurobiological markers of individual relapse risk, focusing on treatment discontinuation.

Key Publications

  • doi pdf A Roadmap for the Development of Applied Computational Psychiatry
  • Paulus MP, Huys QJM and Maia T
  • Biological Psychiatry: Cognitive Neuroscience and Neuroimaging (2016) 1(5):386-392
  • Background: Computational psychiatry is a burgeoning field that utilizes mathematical approaches to investigate psychiatric disorders, derive quantitative predictions, and integrate data across multiple levels of description. Computational psychiatry has already led to many new insights into the neurobehavioral mechanisms that underlie several psychiatric disorders, but its usefulness from a clinical standpoint is only now starting to be considered. Methods: Examples of computational psychiatry are highlighted, and a phase-based pipeline for the development of clinical computational- psychiatry applications is proposed, similar to the phase-based pipeline used in drug development. It is proposed that each phase has unique endpoints and deliverables, which will be important milestones to move tasks, procedures, computational models, and algorithms from the laboratory to clinical practice. Results: Application of computational approaches should be tested on healthy volunteers in Phase I, transitioned to target populations in Phase IB and Phase IIA, and thoroughly evaluated using randomized clinical trials in Phase IIB and Phase III. Successful completion of these phases should be the basis of determining whether computational models are useful tools for prognosis, diagnosis, or treatment of psychiatric patients. Conclusions: A new type of infrastructure will be necessary to implement the proposed pipeline. This infrastructure should consist of groups of investigators with diverse backgrounds collaborating to make computational psychiatry relevant for the clinic.

  • doi pdf Computational psychiatry as a bridge from neuroscience to clinical applications
  • Huys QJM*, Maia T* and Frank MJ
  • Nature Neuroscience (2016) 19(3):404--413
    Commentary in Scientific American
  • Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ----the brain---and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction, and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data-driven and theory-driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes, or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.

  • doi pdf The specificity of Pavlovian regulation is associated with recovery from depression
  • Huys QJM, Gölzer M, Friedel E, Heinz A, Cools R, Dayan P and Dolan RJ
  • Psychological Medicine (2016) 46(5):1027-1035
  • Background: Changes in reflexive emotional responses are hallmarks of depression, but how emotional reflexes impact on adaptive decision-making in depression has not been examined formally. Using a Pavlovian-Instrumental Transfer (PIT) task, we compared the influence of affectively valenced stimuli on decision-making in depression and generalized anxiety disorder compared with healthy controls; and related this to the longitudinal course of the illness.

    Methods: Fourty subjects with a current DSM-IV-TR diagnosis of Major Depressive Disorder, Dysthymia, Generalised Anxiety Disorder or a combination thereof, and 40 matched healthy controls performed a Pavlovian-instrumental transfer (PIT) task that assesses how instrumental approach and withdrawal behaviours are influenced by appetitive and aversive Pavlovian conditioned stimuli. Patients were followed up after 4-6 months. Analyses focussed on patients with depression alone (n=25).

    Results: In healthy controls, Pavlovian conditioned stimuli (CSs) exerted action-specific effects, with appetitive CSs boosting active approach and aversive CSs withdrawal. This action-specificity was absent in currently depressed subjects. Greater action-specificity in patients was associated with better recovery over the follow-up period.

    Conclusions: Depression is associated with abnormal influence of emotional reactions on decision-making in a way that is predictive of recovery.

  • doi pdf Pavlovian-to-instrumental transfer effects in the nucleus accumbens relate to relapse in alcohol dependence
  • Garbusow M*, Schad D*, Sebold M, Friedel E, Bernhardt N, Koch SP, Steinacher B, Kathmann N, Geurts DE, Sommer C, Müller DK, Nebe S, Paul S, Wittchen H-U, Zimmermann US, Walter H, Smolka MN, Sterzer P, Rapp MA, Huys QJM*, Schlagenhauf F* and Heinz A*
  • Addiction Biology (2016) 21(3):719-31
  • In detoxified alcohol-dependent patients, alcohol-related stimuli can promote relapse. However, to date, the mechanisms by which contextual stimuli promote relapse have not been elucidated in detail. One hypothesis is that such contextual stimuli directly stimulate the motivation to drink via associated brain regions like the ventral striatum and thus promote alcohol seeking, intake and relapse. Pavlovian-to-Instrumental-Transfer (PIT) may be one of those behavioral phenomena contributing to relapse, capturing how Pavlovian conditioned (contextual) cues determine instrumental behavior (e.g. alcohol seeking and intake). We used a PIT paradigm during functional magnetic resonance imaging to examine the effects of classically conditioned Pavlovian stimuli on instrumental choices in n=31 detoxified patients diagnosed with alcohol dependence and n=24 healthy controls matched for age and gender. Patients were followed up over a period of 3 months. We observed that (1) there was a significant behavioral PIT effect for all participants, which was significantly more pronounced in alcohol-dependent patients; (2) PIT was significantly associated with blood oxygen level-dependent (BOLD) signals in the nucleus accumbens (NAcc) in subsequent relapsers only; and (3) PIT-related NAcc activation was associated with, and predictive of, critical outcomes (amount of alcohol intake and relapse during a 3 months follow-up period) in alcohol-dependent patients. These observations show for the first time that PIT-related BOLD signals, as a measure of the influence of Pavlovian cues on instrumental behavior, predict alcohol intake and relapse in alcohol dependence.
  • doi pdf Decision-theoretic psychiatry
  • Huys QJM, Guitart-Masip M, Dolan RJ and Dayan P
  • Clin Psychol Sci (2015) 3(3):400-421
  • Psychiatric disorders profoundly impair many aspects of decision-making. Poor choices have negative consequences in the moment, and also make it very hard to navigate complex social environments. Computational neuroscience provides normative, neurobiologically informed, descriptions of the components of decision-making which serve as a platform for a principled exploration of dysfunctions. Here, we identify and discuss three classes of failure-modes arising in these formalisms. They stem from abnormalities in the framing of problems or tasks, from the mechanisms of cognition used to solve the tasks, or from the historical data available from the environment.
  • doi pdf Depression: A Decision-Theoretic Analysis
  • Huys QJM, Daw ND and Dayan P
  • Annu Rev Neurosci (2015), 38:1-23
  • The manifold symptoms of depression are a common and often transient feature of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioural implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.
  • doi pdf The interplay of approximate planning strategies
  • Huys QJM, Lally N, Faulkner P, Eshel N, Seifritz E, Gershman SJ, Dayan P and Roiser JP
  • PNAS (2015), 112(10):3098-3103
    Commentary by Daniel, Schuck & Niv
  • Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."
  • doi pdf The role of learning-related dopamine signals in addiction vulnerability
  • Huys QJM, Tobler PT, Hasler G, Flagel SB
  • Progress in Neurobiology (2014): 211:31-77
  • Dopaminergic signals play a mathematically precise role in reward-related learning, and variations in dopaminergic signalling have been implicated in vulnerability to addiction. Here, we provide a detailed overview of the relationship between theoretical, mathematical and experimental accounts of phasic dopamine signalling, with implications for the role of learning-related dopamine signalling in addiction and related disorders. We describe the theoretical and behavioural characteristics of model-free learning based on errors in the prediction of reward, including step-by-step explanations of the underlying equations. We then use recent insights from an animal model that highlights individual variation in learning during a Pavlovian conditioning paradigm to describe overlapping aspects of incentive salience attribution and model-free learning. We argue that this provides a computationally coherent account of some features of addiction.
  • doi pdf
  • Striatal dysfunction during reversal learning in unmedicated schizophrenia patients
  • Schlagenhauf F*, Huys QJM*, Deserno M, Rapp MA, Beck A and Heinz A
  • Neuroimage (2014) 89:171-180
  • Subjects with schizophrenia are impaired at reinforcement-driven reversal learning from as early as their first episode. The neurobiological basis of this deficit is unknown. We obtained behavioral and fMRI data in 24 unmedicated, primarily first episode, schizophrenia patients and 24 age-, IQ- and gender-matched healthy controls during a reversal learning task. We supplemented our fMRI analysis, focusing on learning from prediction errors, with detailed computational modeling to probe task solving strategy including an ability to deploy an internal goal directed model of the task. Patients displayed reduced functional activation in the ventral striatum (VS) elicited by prediction errors. However, modeling task performance revealed that a subgroup did not adjust their behavior according to an accurate internal model of the task structure, and these were also the more severely psychotic patients. In patients who could adapt their behavior, as well as in controls, task solving was best described by cognitive strategies according to a Hidden Markov Model. When we compared patients and controls who acted according to this strategy, patients still displayed a significant reduction in VS activation elicited by informative errors that precede salient changes of behavior (reversals). Thus, our study shows that VS dysfunction in schizophrenia patients during reward-related reversal learning remains a core deficit even when controlling for task solving strategies. This result highlights VS dysfunction is tightly linked to a reward- related reversal learning deficit in early, unmedicated schizophrenia patients.
  • doi pdf Mapping anhedonia onto reinforcement learning. A behavioural meta-analysis.
  • Huys QJM, Pizzagalli DA, Bogdan R and Dayan P
  • Biology of Mood & Anxiety Disorders (2013) 3:12
  • Background: Depression is characterised partly by blunted reactions to reward. However, tasks probing this deficiency have not distinguished insensitivity to reward from insensitivity to the prediction errors for reward that determine learning and are putatively reported by the phasic activity of dopamine neurons. We attempted to disentangle these factors with respect to anhedonia in the context of stress, Major Depressive Disorder (MDD), Bipolar Disorder (BPD) and a dopaminergic challenge.

    Methods: Six behavioural datasets involving 392 experimental sessions were subjected to a model-based, Bayesian meta-analysis. Participants across all six studies performed a probabilistic reward task that used an asymmetric reinforcement schedule to assess reward learning. Healthy controls were tested under baseline conditions, stress or after receiving the dopamine D2 agonist pramipexole. In addition, participants with current or past MDD or BPD were evaluated. Reinforcement learning models isolated the contributions of variation in reward sensitivity and learning rate.

    Results: MDD and anhedonia reduced reward sensitivity more than they affected the learning rate, while a low dose of the dopamine D2 agonist pramipexole showed the opposite pattern. Stress led to a pattern consistent with a mixed effect on reward sensitivity and learning rate.

    Conclusion: Reward-related learning reflected at least two partially separable contributions. The first related to phasic prediction error signalling, and was preferentially modulated by a low dose of the dopamine agonist pramipexole. The second related directly to reward sensitivity, and was preferentially reduced in MDD and anhedonia. Stress altered both components. Collectively, these findings highlight the contribution of model-based reinforcement learning meta-analysis for dissecting anhedonic behavior.

  • doi pdf Go and nogo learning in reward and punishment: Interactions between affect and effect
  • Guitart-Masip M*, Huys QJM*, Fuentemilla L, Dayan P, Düzel E and Dolan RJ
  • Neuroimage (2012) 62(1):154-66
  • Decision-making invokes two fundamental axes of control: affect or valence, spanning reward and punishment, and effect or action, spanning invigoration and inhibition. We studied the acquisition of instrumental responding in healthy human volunteers in a task in which we orthogonalized action requirements and outcome valence. Subjects were much more successful in learning active choices in rewarded conditions, and passive choices in punished conditions. Using computational reinforcement-learning models, we teased apart contributions from putatively instrumental and Pavlovian components in the generation of the observed asymmetry during learning. Moreover, using model-based fMRI, we showed that BOLD signals in striatum and substantia nigra/ventral tegmental area (SN/VTA) correlated with instrumentally learnt action values, but with opposite signs for go and no-go choices. Finally, we showed that successful instrumental learning depends on engagement of bilateral inferior frontal gyrus. Our behavioral and computational data showed that instrumental learning is contingent on overcoming inherent and plastic Pavlovian biases, while our neuronal data showed this learning is linked to unique patterns of brain activity in regions implicated in action and inhibition respectively.
  • doi pdf Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees
  • Huys QJM*, Eshel N*, O'Lions E, Sheridan L, Dayan P and Roiser JP
  • PLoS Comp Biol (2012) 8(3): e1002410
  • When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal- directed choices in a manner that may be important for theories of decision-making in mood disorders.
  • doi pdf Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding
  • Huys QJM, Cools R, Gölzer M, Friedel E, Heinz A, Dolan RJ and Dayan P
  • PLoS Comp Biol (2011) 7(4): e1002028
  • Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.
  • doi pdf A Bayesian formulation of behavioral control
  • Huys QJM and Dayan P
  • Cognition (2009) Special Issue
  • Helplessness, a belief that the world is not subject to behavioral control, has long been central to our understanding of depression, and has influenced cognitive theories, animal models and behavioral treatments. However, despite its importance, there is no fully accepted definition of helplessness or behavioral control in psychology or psychiatry, and the formal treatments in engineering appear to capture only limited aspects of the intuitive concepts. Here, we formalize controllability in terms of characteristics of prior distributions over affectively charged environments. We explore the relevance of this notion of control to reinforcement learning methods of optimising behavior in such environments and consider how apparently maladaptive beliefs can result from normative inference processes. These results are discussed with reference to depression and animal models thereof.