Research
Copyright
The documents distributed here have been provided as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.
* indicates shared first / senior authorship
Publications
Computational Psychiatry & Decision-making
Papers, reviews & chapters
- psyArXiv pdf Selective outcome reinstatement during evaluation drives heuristics in risky choice
- psyArXiv kb6ew
- A ubiquitous feature of human decision making under risk is that individuals differ from each other, as well as from normativity, in how they incorporate reward and probability information. One possible explanation for these deviations is a desire to reduce the number of potential outcomes considered during choice evaluation. Although multiple behavioral models can be invoked involving selective consideration of choice outcomes, whether differences in these tendencies underlie behavioral differences in sensitivity to reward and probability information is unknown. Here we consider neural evidence where we exploit magnetoencephalography (MEG) to decode the actual choice outcomes participants consider when they decide between a gamble and a safe outcome. We show that variability in tendencies of individual participants to reinstate neural outcome representations, based on either their probability or reward, explains variability in the extent to which their choices reflect consideration of probability and reward information. In keeping with this we also show that participants who are higher in behavioral impulsivity fail to preferentially reinstate outcomes with higher probability. Our results suggest that neural differences in the degree to which outcomes are considered shape risk taking strategy, both in decision making tasks, as well as in real life.
- preprint bioArxiv A Hierarchical Reinforcement Learning Model Explains Individual Differences in Attentional Set Shifting
- bioArxiv
-
Attentional set shifting refers to the ease with which the focus of attention is directed and switched. Cognitive tasks such as CANTAB IED reveal great variation in set shifting ability in the general population, with notable impairments in those with psychiatric diagnoses. The attentional and learning processes underlying this cognitive ability, and how they lead to the observed variation remain unknown. To directly test this, we used a modelling approach on two independent large-scale online general-population samples performing CANTAB IED and psychiatric symptom assessment. We found a hierarchical model that learnt both feature values and dimension attention best explained the data, and that compulsive symptoms were associated with slower learning and higher attentional bias to the first relevant stimulus dimension. This data showcase a new methodology to analyse data from the CANTAB IED task, and suggest a possible mechanistic explanation for the variation in set shifting performance, and its relationship to compulsive symptoms.
- PsyArXiv Disengaging punishment avoidance is difficult for humans
- PsyArxiv
-
Managing multiple goals is essential to wellbeing, yet we are only beginning to understand the computations by which we navigate this resource-demanding balancing act. Here, we sought to elucidate algorithms humans use to balance reward seeking and punishment avoidance goals, and to examine how these algorithms are affected in anxious individuals. To do so, we developed a novel multigoal pursuit task that includes trial-specific instructed goals to either pursue reward (without risk of punishment) or avoid punishment (without the opportunity for reward). Participants (n=192) in general were less flexible in avoiding punishment than in pursuing reward. Thus, when instructed to pursue reward, they often persisted in avoiding features that had previously been associated with punishment, even though at decision time these features were unambiguously benign. Participants also showed no significant downregulation of punishment avoidance when punishment avoidance goals became less abundant in the task. Importantly, individuals with chronic worry had particular difficulty disengaging punishment avoidance during instructed reward seeking. Taken together, the findings demonstrate that people avoid punishment less flexibly than they pursue reward, and this difference is pronounced in individuals with chronic worry.
- preprint psyArxiv A Computational View on the Nature of Reward and Value in Anhedonia
- psyArxiv
-
Anhedonia -- a common feature of depression -- encompasses a reduction in the subjective experience and anticipation of rewarding events, and a reduction in the motivation to seek out such events. The presence of anhedonia often predicts or accompanies treatment resistance, and as such better interventions and treatments are important. Yet the mechanisms giving rise to anhedonia are not well-understood. In this chapter, we briefly review existing computational conceptualisations of anhedonia. We argue that they are mostly descriptive and fail to provide an explanatory account of why anhedonia may occur. Working within the framework of reinforcement learning, we examine two potential computational mechanisms that could give rise to anhedonic phenomena. First, we show how anhedonia can arise in multidimensional drive reduction settings through a trade-off between different rewards or needs. We then generalize this in terms of model-based value inference and identify a key role for associational belief structure. We close with a brief discussion of treatment implications of both of these conceptualisations. In summary, computational accounts of anhedonia have provided a useful descriptive framework. Recent advances in reinforcement learning suggest promising avenues by which the mechanisms underlying anhedonia may be teased apart, potentially motivating novel approaches to treatment.
- doi pdf Explaining distortions in metacognition with an attractor network model of decision uncertainty
- PLoS Comp Biol 17(7): e1009201
-
Metacognition is the ability to reflect on, and evaluate, our cognition and behaviour. Distortions in metacognition are common in mental health disorders, though the neural underpinnings of such dysfunction are unknown. One reason for this is that models of key components of metacognition, such as decision confidence, are generally specified at an algorithmic or process level. While such models can be used to relate brain function to psychopathology, they are difficult to map to a neurobiological mechanism. Here, we develop a biologically-plausible model of decision uncertainty in an attempt to bridge this gap. We first relate the model's uncertainty in perceptual decisions to standard metrics of metacognition, namely mean confidence level (bias) and the accuracy of metacognitive judgments (sensitivity). We show that dissociable shifts in metacognition are associated with isolated disturbances at higher-order levels of a circuit associated with self-monitoring, akin to neuropsychological findings that highlight the detrimental effect of prefrontal brain lesions on metacognitive performance. In contrast to existing theoretical work, we account for empirical confidence judgements by fitting our biophysical model solely to first-order performance data, specifically choice and response times. Lastly, in a reanalysis of existing data we show that self-reported mental health symptoms relate to disturbances in an uncertainty-monitoring component of the network. By bridging a gap between a biologically-plausible model of confidence formation and observed disturbances of metacognition in mental health disorders we provide a first step towards mapping theoretical constructs of metacognition onto dynamical models of decision uncertainty. In doing so, we provide a computational framework for modelling metacognitive performance in settings where access to explicit confidence reports is not possible.
- doi pdf How representative are neuroimaging samples? Large-scale evidence for trait anxiety differences between MRI and behaviour-only research participants.
- Soc Cog Aff Neurosci nsab057
- Over the past three decades, MRI has become a key tool to study how cognitive processes are implemented in the human brain. However, the question of whether participants recruited into MRI studies differ from participants recruited into other study contexts has received little to no attention. This is particularly pertinent when effects fail to generalize across study contexts: for example, if a behavioural effect discovered in a non-imaging context does not replicate in a neuroimaging environment. Here, we tested the hypothesis, motivated by preliminary findings (n=272), that MRI study participants differ from behaviour-only study participants on one fundamental individual difference variable: trait anxiety. Analysing a large-scale dataset drawn from multiple institutions (n=3317) and controlling for possible confounding variables, we found robust support for lower trait anxiety in MRI study participants, consistent with a sampling bias. Distributions of trait anxiety scores differed most markedly when psychiatric screening was minimal. Our findings highlight the need for surveying trait anxiety at recruitment and for appropriate screening procedures, in an attempt to mitigate this bias.
- doi pdf Neurocomputational mediators in psychotherapy
- Curr Op Behav Sci (2021) 38: 103-109
-
A classic definition of intrusive thinking is "any distinct, identifiable cognitive event that is unwanted, unintended, and recurrent. It interrupts the flow of thought, interferes in task performance, is associated with negative affect, and is difficult to control" (Clark 2005:4). While easy to understand and applicable to many cases, this definition does not seem to encompass the entire spectrum of intrusions. For example, intrusive thoughts may not always be experienced as unpleasant or unwanted, and may in some situations even be adaptive. This chapter revisits the definition of intrusive thinking, by systematically considering all the circumstances in which intrusions might occur, their manifestations across health and disorders, and develops an alternative, more inclusive definition of intrusions as being "interruptive, salient, experienced mental events." It proposes that clinical intrusive thinking differs from its nonclinical form with regard to frequency, intensity, and maladaptive reappraisal. Further, it discusses the neurocognitive processes underlying intrusive thinking and its control, including memory pro- cesses involved in action control, working memory and long-term memory encoding, retrieval, and suppression. As part of this, current methodologies used to study intrusive thinking are evaluated and areas are highlighted where more research and/or technical innovation is needed. It concludes with a discussion of the theoretical, therapeutic, and sociocultural implications of intrusive thinking and its control.
- doi pdf Model-based and model-free control predicts alcohol consumption developmental trajectory in young adults - a three-year prospective study
- Biological Psychiatry (2021) 89(10): 980-989
- Background: A shift from goal-directed toward habitual control has been associated with alcohol dependence. Whether such a shift predisposes pathological drinking is not yet clear. We investigated how goal-directed and habitual control at age 18 predict alcohol use trajectories over the course of three years. Methods: Goal-directed and habitual control, as informed by model-based and model-free learning, were assessed with a two-step sequential decision-making task during fMRI in 146 healthy 18-year-old male adults. Two key drinking variables were used to model the three-year alcohol use developmental trajectory: a consumption score from the Alcohol Use Disorders Identification Test (AUDIT-C; assessed every six months) and a binge drinking score (gram alcohol/occasion; assessed every year). We applied a latent growth curve model to examine how model-based and model-free control predicted the drinking trajectory. Results: The drinking behavior was best characterized by a linear trajectory. The model-based behavioral control was negatively associated with the development of the binge drinking score; the model-free reward prediction error (RPE) BOLD signals in the ventromedial prefrontal cortex and the ventral striatum predicted a higher starting point and steeper increase of the consumption score over time, respectively. Conclusions: We found that model-based behavioral control was associated with the binge drinking trajectory, while the model-free RPE signal was closely linked to the consumption score development. These findings support the idea that a shift from model-based to model-free control might be an important individual vulnerability in predisposing hazardous drinking behavior.
- doi pdf A Comparison of 'Pruning' During Multi-Step Planning in Depressed and Healthy Individuals
- Psychological Medicine 1-9.
- Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."
- doi pdf The relationship between resting-state functional connectivity, antidepressant discontinuation and depression relapse
- Scientific reports (2020) 10:22346
- The risk of relapsing into depression after stopping antidepressants is high, but no established predictors exist. Resting-state functional magnetic resonance imaging (rsfMRI) measures may help predict relapse and identify the mechanisms by which relapses occur. rsfMRI data were acquired from healthy controls and from patients with remitted major depressive disorder on antidepressants. Patients were assessed a second time either before or after discontinuation of the antidepressant, and followed up for six months to assess relapse. A seed-based functional connectivity analysis was conducted focusing on the left subgenual anterior cingulate cortex and left posterior cingulate cortex. Seeds in the amygdala and dorsolateral prefrontal cortex were explored. 44 healthy controls (age: 33.8 (10.5), 73% female) and 84 patients (age: 34.23 (10.8), 80% female) were included in the analysis. 29 patients went on to relapse and 38 remained well. The seed-based analysis showed that discontinuation resulted in an increased functional connectivity between the right dorsolateral prefrontal cortex and the parietal cortex in non-relapsers. In an exploratory analysis, this functional connectivity predicted relapse risk with a balanced accuracy of 0.86. Further seed-based analyses, however, failed to reveal differences in functional connectivity between patients and controls, between relapsers and non-relapsers before discontinuation and changes due to discontinuation independent of relapse. In conclusion, changes in the connectivity between the dorsolateral prefrontal cortex and the posterior default mode network were associated with and predictive of relapse after open-label antidepressant discontinuation. This finding requires replication in a larger dataset.
- pdf The association of the OPRM1 A118G polymorphism and Pavlovian-to-instrumental transfer: clinical relevance for alcohol dependence
- J Psychopharmacology (2020) in press
- Background: Pavlovian-to-instrumental transfer (PIT) quantifies the extent to which a stimulus that has been associated with reward or punishment alters operant behavior. In alcohol dependence (AD), the PIT effect serves as a paradigmatic model of cue-induced relapse. Preclinical studies have suggested a critical role of the opioid system in modulating Pavlovian-instrumental interactions. The A118G polymorphism of the OPRM1 gene affects opioid receptor availability and function. Furthermore, this polymorphism interacts with cue-induced approach behavior and is a potential biomarker for pharmacological treatment response in AD. Here, we test whether the OPRM1 polymorphism is associated with the PIT effect and relapse in AD. Methods: Using a Pavlovian-to-instrumental transfer task, we examined three independent samples including young healthy subjects (n=161), detoxified alcohol-dependent patients (n=186) and age-matched healthy controls (n=105). We used data of a larger study designed to assess the role of learning mechanisms in the development and maintenance of AD. Subjects were genotyped for the A118G (rs1799971) polymorphism of the OPRM1 gene. Relapse was assessed after three months. Results: In all three samples, participants with the minor OPRM1 G-Allele (G+-carriers) showed increased expression of the PIT effect in the absence of learning differences. Relapse was not associated with the OPRM1 polymorphism. Instead, G+ carriers displaying increased PIT effects were particularly prone to relapse. Conclusion: These results support a role for the opioid system in incentive salience motivation. Furthermore, they inform a mechanistic model of aberrant salience processing and are in line with the pharmacological potential of opioid receptor targets in the treatment of AD.
- doi pdf Susceptibility to interference between Pavlovian and instrumental control is associated with early hazardous alcohol use
- Addiction Biology (2020) e12983
- Pavlovian-to-instrumental transfer (PIT) tasks examine the influence of Pavlovian stimuli on ongoing instrumental behaviour. Previous studies reported associations between a strong PIT effect, high-risk drinking, and alcohol use disorder. This study investigated whether susceptibility to interference between Pavlovian and instrumental control is linked to risky alcohol use in a community sample of 18-year-old male adults. Participants (N=191) were instructed to "collect good shells" and "leave bad shells" during the presentation of appetitive (monetary reward), aversive (monetary loss), or neutral Pavlovian stimuli. We compared instrumental error rates (ER) and fMRI brain responses between the congruent and incongruent conditions, as well as among high-risk and low-risk drinking groups. On average, individuals showed a substantial PIT effect, i.e. increased ER when Pavlovian cues and instrumental stimuli were in conflict compared to congruent trials. Neural PIT correlates were found in the ventral striatum and the dorsomedial and lateral prefrontal cortices (lPFC). Importantly, high-risk drinking was associated with a stronger behavioural PIT effect, a decreased lPFC response, and an increased neural response in the ventral striatum on the trend level. Moreover, high-risk drinkers showed weaker connectivity from the ventral striatum to the lPFC during incongruent trials. Our study links interference during PIT to drinking behaviour in healthy, young adults. High-risk drinkers showed higher susceptibility to Pavlovian cues, especially when they conflicted with instrumental behaviour, indicating lower interference control abilities. Increased activity in the ventral striatum (bottom-up), decreased lPFC response (top-down), along with their altered interplay may contribute to poor interference control in the high-risk drinkers.
- pdf Neuropsychological Mechanisms of Intrusive Thinking
- In: Intrusive Thinking: From Molecules to Free Will, edited by Kalivas, P and Paulus, M. Strüngmann Forum Reports, vol. 30, J. Lupp, series editor. Cambridge, MA: MIT Press
-
A classic definition of intrusive thinking is "any distinct, identifiable cognitive event that is unwanted, unintended, and recurrent. It interrupts the flow of thought, interferes in task performance, is associated with negative affect, and is difficult to control" (Clark 2005:4). While easy to understand and applicable to many cases, this definition does not seem to encompass the entire spectrum of intrusions. For example, intrusive thoughts may not always be experienced as unpleasant or unwanted, and may in some situations even be adaptive. This chapter revisits the definition of intrusive thinking, by systematically considering all the circumstances in which intrusions might occur, their manifestations across health and disorders, and develops an alternative, more inclusive definition of intrusions as being "interruptive, salient, experienced mental events." It proposes that clinical intrusive thinking differs from its nonclinical form with regard to frequency, intensity, and maladaptive reappraisal. Further, it discusses the neurocognitive processes underlying intrusive thinking and its control, including memory pro- cesses involved in action control, working memory and long-term memory encoding, retrieval, and suppression. As part of this, current methodologies used to study intrusive thinking are evaluated and areas are highlighted where more research and/or technical innovation is needed. It concludes with a discussion of the theoretical, therapeutic, and sociocultural implications of intrusive thinking and its control.
- doi preprint pdf Advances in the computational understanding of mental illness
- Neuropsychopharmacology In Press
-
Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.
- doi pdf Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task
- European Neuropsychopharmacology (2020) 35:17-29
-
When facing decisions to approach rewards or to avoid punishments, we often figuratively go with our gut, and the impact of metabolic states such as hunger on motivation are well documented. However, whether and how vagal feedback signals from the gut influence instrumental actions is unknown. Here, we investigated the effect of non-invasive transcutaneous vagus nerve stimulation (tVNS) vs. sham (randomized cross-over design) on approach and avoidance behavior using an established go/no-go reinforcement learning paradigm (Guitart-Masip et al., 2012) in 39 healthy, participants after an overnight fast. First, mixed-effects logistic regression analysis of choice accuracy showed that tVNS acutely impaired decision-making, p = .045. Computational reinforcement learning models identified the cause of this as a reduction in the learning rate through tVNS (Delta = -0.092, pboot = .002), particularly after punishment (Delta Pun = -0.081, pboot = .012 vs. Delta Rew = -0.031, p = .22). However, tVNS had no effect on go biases, Pavlovian response biases or response time. Hence, tVNS appeared to influence learning rather than action execution. These results highlight a novel role of vagal afferent input in modulating reinforcement learning by tuning the learning rate according to homeostatic needs.
- doi Opportunities for emotion and mental health research in the resource-rationality framework
- Behav Brain Sci (2020) 43:e21
- We discuss opportunities in applying the resource-rationality framework toward answering questions in emotion and mental health research. These opportunities rely on characterization of individual differences in cognitive strategies; an endeavor that may be at odds with the normative approach outlined in the target article. We consider ways individual differences might enter the framework and the translational opportunities offered by each.
- doi Canonical correlation analysis for identifying biotypes of depression
- Biological Psychiatry CNNI (2020) 5(5): 478-480
- n old question is challenging novel methods: do diagnoses such as schizophrenia and depression contain subgroups of patients? And might such subgroups be identifiable by neurobiological means, and have differential responses to therapies? If so, then long-awaited biomarkers for psychiatric diagnosis and therapeutics might be found.
- doi pdf Computational mechanisms of effort and reward decisions in depression and their relationship to relapse after antidepressant discontinuation
- JAMA Psychiatry (2020) 77(5):513-522
- IMPORTANCE Nearly 1 in 3 patients with major depressive disorder who respond to antidepressants relapse within 6 months of treatment discontinuation. No predictors of relapse exist to guide clinical decision-making in this scenario. OBJECTIVES To establish whether the decision to invest effort for rewards represents a persistent depression process after remission, predicts relapse after remission, and is affected by antidepressant discontinuation. DESIGN, SETTING, AND PARTICIPANTS This longitudinal randomized observational prognostic study in a Swiss and German university setting collected data from July 1, 2015, to January 31, 2019, from 66 healthy controls and 123 patients in remission from major depressive disorder in response to antidepressants prior to and after discontinuation. Study recruitment took place until January 2018. EXPOSURE Discontinuation of antidepressants. MAIN OUTCOMES AND MEASURES Relapse during the 6 months after discontinuation. Choice and decision times on a task requiring participants to choose how much effort to exert for various amounts of reward and the mechanisms identified through parameters of a computational model. RESULTS A total of 123 patients (mean [SD] age, 34.5 [11.2] years; 94 women [76%]) and 66 healthy controls (mean [SD] age, 34.6 [11.0] years; 49 women [74%]) were recruited. In the main subsample, mean (SD) decision times were slower for patients (n = 74) compared with controls (n = 34) (1.77 [0.38] seconds vs 1.61 [0.37] seconds; Cohen d = 0.52; P = .02), particularly for those who later relapsed after discontinuation of antidepressants (n = 21) compared with those who did not relapse (n = 39) (1.95 [0.40] seconds vs 1.67 [0.34] seconds; Cohen d = 0.77; P < .001). This slower decision time predicted relapse (accuracy = 0.66; P = .007). Patients invested less effort than healthy controls for rewards (F1,98 = 33.970; P < .001). Computational modeling identified a mean (SD) deviation from standard drift-diffusion models that was more prominent for patients than controls (patients, 0.67 [1.56]; controls, -0.71 [1.93]; Cohen d = 0.82; P < .001). Patients also showed higher mean (SD) effort sensitivity than controls (patients, 0.31 [0.92]; controls, -0.08 [1.03]; Cohen d = 0.51; P = .05). Relapsers differed from nonrelapsers in terms of the evidence required to make a decision for the low-effort choice (mean [SD]: relapsers, 1.36 [0.35]; nonrelapsers, 1.17 [0.26]; Cohen d = 0.65; P = .02). Group differences generally did not reach significance in the smaller replication sample (27 patients and 21 controls), but decision time prediction models from the main sample generalized to the replication sample (validation accuracy = 0.71; P = .03). CONCLUSIONS AND RELEVANCE This study found that the decision to invest effort was associated with prospective relapse risk after antidepressant discontinuation and may represent a persistent disease process in asymptomatic remitted major depressive disorder. Markers based on effort-related decision-making could potentially inform clinical decisions associated with antidepressant discontinuation.
- doi pdf Low predictive power of clinical features for relapse prediction after antidepressant discontinuation in a naturalistic setting
- bioRxiv
- Background: The risk of relapse after antidepressant medication (ADM) discontinuation is high. Predictors of relapse could guide clinical decision-making, but are yet to be established. Method: We assessed demographic and clinical variables in a longitudinal observational study before antidepressant discontinuation. State-dependent variables were re-assessed either after discontinuation or before discontinuation after a waiting period. Relapse was assessed during six months after discontinuation. We applied logistic general linear models in combination with least absolute shrinkage and selection operator and elastic nets to avoid overfitting in order to identify predictors of relapse and estimated their generalisability using cross-validation. Results: The final sample included 104 patients (age: 34.86 (11.1), 77% female) and 57 healthy controls (age: 34.12 (10.6), 70% female). 36% of the patients experienced a relapse. Treatment by a general practitioner increased the risk of relapse. Although within-sample statistical analyses suggested reasonable sensitivity and specificity, out-of-sample prediction of relapse was at chance level. Residual symptoms increased with discontinuation, but did not relate to relapse. Conclusion: and Relevance Demographic and standard clinical variables appear to carry little predictive power and therefore are of limited use for patients and clinicians in guiding clinical decision-making.
- Abnormal Reward Valuation and Event-Related Connectivity in Unmedicated Major Depressive Disorder
- Psychological Medicine (2020)
- Background. Experience of emotion is closely linked to valuation. Mood can be viewed as a bias to experience positive or negative emotions and abnormally biased subjective reward valuation and cognitions are core characteristics of major depression. Methods. Thirty-four unmedicated subjects with major depressive disorder and controls estimated the probability that fractal stimuli were associated with reward, based on passive observations, so they could subsequently choose the higher of either their estimated fractal value or an explicitly presented reward probability. Using model-based fMRI, we estimated each subject's internal value estimation, with psychophysiological interaction analysis used to examine event-related connectivity, testing hypotheses of abnormal reward valuation and cingulate connectivity in depression. Results. Reward value encoding in the hippocampus and rostral anterior cingulate was abnormal in depression. In addition, abnormal decision-making in depression was associated with increased anterior mid-cingulate activity and a signal in this region encoded the difference between the values of the two options. This localised decision-making and its impairment to the anterior mid-cingulate cortex consistent with theories of cognitive control. Notably, subjects with depression had significantly decreased event-related connectivity between the anterior mid-cingulate cortex and rostral cingulate regions during decision-making, implying impaired communication between the neural substrates of expected value estimation and decision-making in depression. Conclusions. Our findings support the theory that abnormal neural reward valuation plays a central role in MDD. To the extent that emotion reflects valuation, abnormal valuation could explain abnormal emotional experience in MDD, reflect a core pathophysiological process and be a target of treatment.
- doi Psychiatric Illnesses as Disorders of Network Dynamics
- Biological Psychiatry: Cognitive Neuroscience and Neuroimaging. In Press.
-
This review provides a dynamical systems perspective on psychiatric symptoms and disease, and discusses its potential implications for diagnosis, prognosis, and treatment. After a brief introduction into the theory of dynamical systems, we will focus on the idea that cognitive and emotional functions are implemented in terms of dynamical systems phenomena in the brain, a common assumption in theoretical and computational neuroscience. Specific computational models, anchored in biophysics, for generating different types of network dynamics, and with a relation to psychiatric symptoms, will be briefly reviewed, as well as methodological approaches for reconstructing the system dynamics from observed time series (like fMRI or EEG recordings). We then attempt to outline how psychiatric phenomena, associated with schizophrenia, depression, PTSD, ADHD, phantom pain, and others, could be understood in dynamical systems terms. Most importantly, we will try to convey that the dynamical systems level may provide a central, hub-like level of convergence which unifies and links multiple biophysical and behavioral phenomena, in the sense that diverse biophysical changes can give rise to the same dynamical phenomena and, vice versa, similar changes in dynamics may yield different behavioral symptoms depending on the brain area where these changes manifest. If this assessment is correct, it may have profound implications for the diagnosis, prognosis, and treatment of psychiatric conditions, as it puts the focus on dynamics. We therefore argue that consideration of dynamics should play an important role in the choice and target of interventions.
- Individual differences in dopamine function underlying the balance between model- based and model-free control
- bioRxiv
- Reinforcement learning involves a balance between model-free (MF) and model-based (MB) systems. Recent studies suggest that individuals with either pharmacologically enhanced levels of dopamine (DA) or higher baseline levels of DA exhibit more MB control. However, it remains unknown whether such pharmacological effects depend on baseline DA. Here, we investigated whether effects of L-DOPA on the balance of MB/MF control depend on ventral striatal baseline DA. Sixty participants had two functional magnetic resonance imaging (fMRI) scans while performing a two-stage sequential decision-making task under 150 mg L-DOPA or placebo (counterbalanced), followed by a 4-hour 18F-DOPA positron emission tomography (PET) scan (on a separate occasion). We found an interaction between baseline DA levels and L-DOPA induced changes in MB control. Individuals with higher baseline DA levels showed a greater L-DOPA induced enhancement in MB control. Surprisingly, we found a corresponding drug-by-baseline DA interaction on MF, but not MB learning signals in the ventromedial prefrontal cortex. We did not find a significant interaction between baseline DA levels and L-DOPA effects on MF control or MB/MF balance. In sum, our findings point to a baseline dependency of L-DOPA effects on differential aspects of MB and MF control. Individual differences in DA washout may be an important moderator of L-DOPA effects. Overall, our findings complement the general notion where higher DA levels is related to a greater reliance on MB control. Although the relationship between phasic DA firing and MF learning is clear in the animal literature, the relationship between DA and MF control is not as straightforward and requires further clarification.
- blog doi pdf Dissociating neural learning signals in human sign- and goal-trackers
- Nat. Hum. Behav. (2020) 4:201-214
- Individuals differ in how they learn from experience. In Pavlovian conditioning paradigms, where cues predict reinforcer delivery at a different goal location, some animals--so-called sign-trackers--come to approach the cue, whereas others, called `goal-trackers', approach the goal. In sign-trackers, model-free phasic dopaminergic reward prediction errors underlie learning, which renders stimuli `wanted'. `Goal-trackers' do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in male humans using eye-tracking, pupillometry and fMRI informed by computational models. We show that only sign- trackers exhibit a neural reward prediction error signal. Only for them is gaze and pupil dilation guided by model-free value. Goal-trackers exhibit a stronger model-based neural state prediction error signal. Only for them do model-based constructs determine gaze and pupil dilation. As sign-tracking may be a vulnerability factor for impulsive and addictive behavior, these findings have implications for mental health.
- doi pdf No substantial change in the balance between model-free and model-based control via training on the two-step task
- PLos Comp Biol (2019) 15(11): e1007443
- Human decisions can be habitual or goal-directed, also known as model-free (MF) or model-based (MB) control. Previous work suggests that the balance between the two decision systems is impaired in psychiatric disorders such as compulsion and addiction, via overreliance on MF control. However, little is known whether the balance can be altered through task training. Here, 20 healthy participants performed a well-established two-step task that differentiates MB from MF control, across five training sessions. We used computational modelling and functional near-infrared spectroscopy to assess changes in decision-making and brain activity over time. Mixed-effects modeling revealed overall no substantial changes in MF and MB behavior across training. Although our findings show task-induced changes in learning rates, these parameters have no direct relation to either MF or MB control or the balance between the two systems, and thus do not support the assertion that training affects MF or MB strategies. The missing training effect in behavior correlated well with activity in the inferior lateral prefrontal cortex, which has been suggested to arbitrate between MF and MB control. Our findings indicate that simple repeated training on the two-step paradigm does not lead to a shift in the balance between MF and MB control. We discuss these results with respect to implications for restoring the balance between MF and MB control in psychiatric conditions.
- doi Realizing the Clinical Potential of Computational Psychiatry: Report from the Banbury Center Meeting, February 2019
- Biological Psychiatry (2020)
- Computational psychiatry is an emerging field that examines phenomena in mental illness using formal techniques from computational neuroscience, mathematical psychology and machine learning. These techniques can be used in a theory-driven manner to gain algorithmic insight into neural or cognitive processes and in a data-driven way to identify predictive and explanatory relationships in complex datasets. The approaches complement each other: theory-driven models can be used to infer mechanisms, and the resulting measurements can be used in data-driven approaches for prediction. Data-driven algorithms can be used to answer theory-driven questions and pose quantitative problems requiring theoretical analysis. Recent computational studies have successfully described and measured novel mechanisms in a range of disorders, and have identified novel predictors of treatment response. These methods hold the potential to improve identification of relevant clinical variables, and could be superior to classification based on traditional behavioral or neural data alone. However, these promising results have been slow to influence clinical practice or to improve patient outcomes.
- psyarxiv The importance of standards for sharing of computational models and data
- Computational Brain & Behavior (2019) 2(3-4): 229--232
- The Target Article by Lee et al. (2019) highlights the ways in which ongoing concerns about research reproducibility extend to model-based approaches in cognitive science. Whereas Lee et al. focus primarily on the importance of research practices to improve model robustness, we propose that the transparent sharing of model specifications, including their inputs and outputs, is also essential to improving the reproducibility of model-based analyses. We outline an ongoing effort (within the context of the Brain Imaging Data Structure community) to develop standards for the sharing of the structure of computational models and their outputs.
- doi pdf Reward-Based Learning, Model-Based and Model-Free
- In Jaeger, D and Jung, R (Ed): Encyclopædia of Computational Neuroscience. SpringerReference. Springer Verlag Berlin Heidelberg, 2019.
- Reinforcement learning (RL) techniques are a set of solutions for optimal long-term action choice such that actions take into account both immediate and delayed consequences. They fall into two broad classes: model-based and model-free approaches. Model-based approaches assume an explicit model of the environment and the agent. The model describes the consequences of actions and the associated returns. From this, optimal policies can be inferred. Psychologically, model-based descriptions apply to goal-directed decisions, in which choices reflect current preferences over outcomes. Model-free approaches forget any explicit knowledge of the dynamics of the environment or the consequences of actions and evaluate how good actions are through trial-and-error learning. Model-free values underlie habitual and Pavlovian conditioned responses that are emitted reflexively when faced with certain stimuli. While model-based techniques have substantial computational demands, model-free techniques require extensive experience.o
- doi pdf Pavlovian-To-Instrumental Transfer and Alcohol Consumption in Young Male Social Drinkers: Behavioral, Neural and Polygenic Correlates
- J. Clin. Med. 2019, 8(8), 1188
- In animals and humans, behavior can be influenced by irrelevant stimuli, a phenomenon called Pavlovian-to-instrumental transfer (PIT). In subjects with substance use disorder, PIT is even enhanced with functional activation in the nucleus accumbens (NAcc) and amygdala. While we observed enhanced behavioral and neural PIT effects in alcohol-dependent subjects, we here aimed to determine whether behavioral PIT is enhanced in young men with high-risk compared to low-risk drinking and subsequently related functional activation in an a-priori region of interest encompassing the NAcc and amygdala and related to polygenic risk for alcohol consumption. A representative sample of 18-year old men (n = 1937) was contacted: 445 were screened, 209 assessed: resulting in 191 valid behavioral, 139 imaging and 157 genetic datasets. None of the subjects fulfilled criteria for alcohol dependence according to the Diagnostic and Statistical Manual of Mental Disorders-IV-TextRevision (DSM-IV-TR). We measured how instrumental responding for rewards was influenced by background Pavlovian conditioned stimuli predicting action-independent rewards and losses. Behavioral PIT was enhanced in high-compared to low-risk drinkers (b = 0.09, SE = 0.03, z = 2.7, p < 0.009). Across all subjects, we observed PIT-related neural blood oxygen level-dependent (BOLD) signal in the right amygdala (t = 3.25, pSVC = 0.04, x = 26, y = -6, z = -12), but not in NAcc. The strength of the behavioral PIT effect was positively correlated with polygenic risk for alcohol consumption (rs = 0.17, p = 0.032). We conclude that behavioral PIT and polygenic risk for alcohol consumption might be a biomarker for a subclinical phenotype of risky alcohol consumption, even if no drug-related stimulus is present. The association between behavioral PIT effects and the amygdala might point to habitual processes related to out PIT task. In non-dependent young social drinkers, the amygdala rather than the NAcc is activated during PIT; possible different involvement in association with disease trajectory should be investigated in future studies.
- doi pdf Machine learning and big data in psychiatry: toward clinical applications
- Current Opinion in Neurobiology (2019) 55:152-159
- Psychiatry is a medical field concerned with the treatment of mental illness. Psychiatric disorders broadly relate to higher functions of the brain, and as such are richly intertwined with social, cultural, and experiential factors. This makes them exquisitely complex phenomena that depend on and interact with a large number of variables. Computational psychiatry provides two ways of approaching this complexity. Theory-driven computational approaches employ mechanistic models to make explicit hypotheses at multiple levels of analysis. Data-driven machine-learning approaches can make predictions from high-dimensional data and are generally agnostic as to the underlying mechanisms. Here, we review recent advances in the use of big data and machine-learning approaches toward the aim of alleviating the suffering that arises from psychiatric disorders.
- doi pdf Dysfunctional approach behavior triggered by alcohol-unrelated Pavlovian cues predicts long-term relapse in alcohol dependence
- Addiction Biology (2018)
- Pavlovian conditioned cues (CSs) can drive instrumental behavior in alcohol-dependent patients. However, it remains unclear if the influence of Pavlovian CSs might also promote maladaptive decisions that can increase the risk of relapse. We studied 109 abstinent alcohol-dependent patients and 93 controls who completed a Pavlovian-to-instrumental transfer (PIT) paradigm, and assessed patients' subsequent relapse status during a 1-year follow-up. In our PIT task, participants had to collect "good shells" (instrumental approach) or leave "bad shells" (instrumental inhibition) during the presence of money-related Pavlovian CSs or drink-related pictures in the background. Pavlovian CSs indicated either a monetary gain (i.e. 1#, 2#), a monetary loss (i.e. -1#, -2#) or a neutral stimulus (0#). Drink-related background pictures were either pictures of participants' favorite alcoholic drink or pictures of water. We found that the influence of money-related Pavlovian CSs on instrumental behavior (i.e., the PIT effect) was more pronounced in future relapsers compared to abstainers and controls. Relapsers particularly failed to correctly perform in trials where the instrumental stimulus required inhibition while a Pavlovian CS in the background indicated a monetary gain. Under that condition, future relapsers falsely approached the instrumental stimulus, independent of the expected punishment. In contrast, we found no difference in PIT between future relapsers and abstainers when pictures of alcohol or water were presented in the background. The failure of inhibiting an aversive stimulus in favor of approaching an appetitive contextual cue might reflect a general susceptibility in relapsers to the influence of positive valued cues that are incongruent to previous learned behavior. A possible relation to maladaptive decision making that can lead to high-risk situations for relapse is discussed.
- arxiv pdf Psychiatric Illnesses as Disorders of Network Dynamics
- arXiv (2018) 1809.06303
-
This review provides a dynamical systems perspective on psychiatric symptoms and disease, and discusses its potential implications for diagnosis, prognosis, and treatment. After a brief introduction into the theory of dynamical systems, we will focus on the idea that cognitive and emotional functions are implemented in terms of dynamical systems phenomena in the brain, a common assumption in theoretical and computational neuroscience. Specific computational models, anchored in biophysics, for generating different types of network dynamics, and with a relation to psychiatric symptoms, will be briefly reviewed, as well as methodological approaches for reconstructing the system dynamics from observed time series (like fMRI or EEG recordings). We then attempt to outline how psychiatric phenomena, associated with schizophrenia, depression, PTSD, ADHD, phantom pain, and others, could be understood in dynamical systems terms. Most importantly, we will try to convey that the dynamical systems level may provide a central, hub-like level of convergence which unifies and links multiple biophysical and behavioral phenomena, in the sense that diverse biophysical changes can give rise to the same dynamical phenomena and, vice versa, similar changes in dynamics may yield different behavioral symptoms depending on the brain area where these changes manifest. If this assessment is correct, it may have profound implications for the diagnosis, prognosis, and treatment of psychiatric conditions, as it puts the focus on dynamics. We therefore argue that consideration of dynamics should play an important role in the choice and target of interventions.
- doi pdf Generalization and Search in Risky Environments
- Cognitive Science (2018) 42(8):2592-2620
-
How do people pursue rewards in risky environments, where some outcomes should be avoided at all costs? We investigate how participant search for spatially correlated rewards in scenarios where one must avoid sampling rewards below a given threshold. This requires not only the bal- ancing of exploration and exploitation, but also reasoning about how to avoid potentially risky areas of the search space. Within risky versions of the spatially correlated multi-armed bandit task, we show that participants' behavior is aligned well with a Gaussian process function learning algorithm, which chooses points based on a safe optimization routine. Moreover, using leave-one-block-out cross-validation, we find that participants adapt their sampling behavior to the riskiness of the task, although the underlying function learning mechanism remains relatively unchanged. These results show that participants can adapt their search behavior to the adversity of the envi- ronment and enrich our understanding of adaptive behavior in the face of risk and uncertainty.
- doi pdf Self-regulation of the dopaminergic reward circuit in cocaine users with mental imagery and neurofeedback
-
EBioMedicine (2018) 37:489--498
Commentary by Kathryn Dickerson -
Background:Enhanced drug-related reward sensitivity accompanied by impaired sensitivity to non-drug related rewards in the mesolimbic dopamine system are thought to underlie the broad motivational deficits and dysfunctional decision-making frequently observed in cocaine use disorder (CUD). Effective approaches to modify this imbalance and reinstate non-drug reward responsiveness are urgently needed. Here, we examined whether cocaine users (CU) can use mental imagery of non-drug rewards to self-regulate the ventral tegmental area and substantia nigra (VTA/SN). We expected that obsessive and compulsive thoughts about cocaine con- sumption would hamper the ability to self-regulate the VTA/SN activity and tested if real-time fMRI (rtfMRI) neurofeedback (NFB) can improve self-regulation of the VTA/SN.
Methods: Twenty-two CU and 28 healthy controls (HC) were asked to voluntarily up-regulate VTA/SN activity with non-drug reward imagery alone, or combined with rtfMRI NFB.
Results: On a group level, HC and CU were able to activate the dopaminergic midbrain and other reward regions with reward imagery. In CU, the individual ability to self-regulate the VTA/SN was reduced in those with more severe obsessive-compulsive drug use. NFB enhanced the effect of reward imagery but did not result in transfer effects at the end of the session.
Conclusion: CU can voluntary activate their reward system with non-drug reward imagery and improve this abil- ity with rtfMRI NFB. Combining mental imagery and rtFMRI NFB has great potential for modifying the mal- adapted reward sensitivity and reinstating non-drug reward responsiveness. This motivates further work to examine the use of rtfMRI NFB in the treatment of CUD.
- doi pdf Prefrontal glutamate levels predict altered amygdala-prefrontal connectivity in traumatized youths
- Psychol Med (2018) Epub ahead of print.
-
Background: Neurobiological models of stress and stress-related mental illness, including post-traumatic stress disorder (PTSD), converge on the amygdala and the prefrontal cortex (PFC). While a surge of research has reported altered structural and functional connectivity between amygdala and the medial PFC following severe stress, few have addressed the underlying neurochemistry. Methods: We combined resting-state functional magnetic resonance imaging (rsfMRI) measures of amygdala connectivity with in vivo MR-spectroscopy (1H-MRS) measurements of glutamate in 26 survivors from the 2011 Norwegian terror attack and 34 control subjects. Results: Traumatized youths showed altered amygdala - anterior midcingulate cortex (aMCC) and amygdala - ventromedial prefrontal cortex (vmPFC) connectivity. Moreover, the trauma survivors exhibited reduced levels of glutamate in the vmPFC which fits with previous findings of reduced levels of Glx (glutamate + glutamine) in the aMCC (Ousdal et al. 2017) and together suggest long-term impact of a traumatic experience on glutamatergic pathways. Importantly, local glutamatergic metabolite levels predicted the individual amygdala - aMCC and amygdala - vmPFC functional connectivity, and also mediated the observed group difference in amygdala - aMCC connectivity. Conclusions: Our findings suggest that traumatic stress may influence amygdala-prefrontal neuronal connectivity through an effect on prefrontal glutamate and its compounds. Understanding the neurochemical underpinning of altered amygdala connectivity after trauma may ultimately lead to the discovery of new pharmacological agents which can prevent or treat stress-related mental illness.
- doi pdf Major Depression Impairs the Use of Reward Values for Decision-Making
- Scientific Reports (2018) 8(1):13798
- Depression is a debilitating condition with a high prevalence. Depressed patients have been shown to be diminished in their ability to integrate their reinforcement history to adjust future behaviour during instrumental reward learning tasks. Here, we tested whether such impairments could also be observed in a Pavlovian conditioning task. We recruited and analysed 32 subjects, 15 with depression and 17 healthy controls, to study behavioural group differences in learning and decision-making. Participants had to estimate the probability of some fractal stimuli to be associated with a binary reward, based on a few passive observations. They then had to make a choice between one of the observed fractals and another target for which the reward probability was explicitly given. Computational modelling was used to succinctly describe participants' behaviour. Patients performed worse than controls at the task. Computational modelling revealed that this was caused by behavioural impairments during both learning and decision phases. Depressed subjects showed lower memory of observed rewards and had an impaired ability to use internal value estimations to guide decision-making in our task.
- doi pdf Depression is associated with enhanced aversive Pavlovian control over instrumental behaviour
- Scientific Reports (2018) 8(1):12582
- The dynamic modulation of instrumental behaviour by conditioned Pavlovian cues is an important process in decision-making. Patients with major depressive disorder (MDD) are known to exhibit mood-congruent biases in information processing, which may occur due to Pavlovian influences, but this hypothesis has never been tested directly in an unmedicated sample. To address this we tested unmedicated MDD patients and healthy volunteers on a computerized Pavlovian-Instrumental Transfer (PIT) task designed to separately examine instrumental approach and withdrawal actions in the context of Pavlovian appetitive and aversive cues. This design allowed us to directly measure the degree to which Pavlovian cues influence instrumental responding. Depressed patients were profoundly influenced by aversive Pavlovian stimuli, to a significantly greater degree than healthy volunteers. This was the case for instrumental behaviour both in the approach condition (in which aversive Pavlovian cues inhibited `go' responses), and in the withdrawal condition (in which aversive Pavlovian cues facilitated `go' responses). Exaggerated aversive PIT provides a potential cognitive mechanism for biased emotion processing in major depression. This finding also has wider significance for the understanding of disrupted motivational processing in neuropsychiatric disorders.
- doi pdf Deficits in context-dependent adaptive coding in early psychosis and healthy individuals with schizotypal personality traits
- Brain (2018) 141:2806 - 2819
-
Objective:Adaptive coding of information is a fundamental principle of brain functioning. It allows for efficient representation over a large range of inputs and thereby alleviates the limited coding range of neurons. In the present study, we investigated for the first time potential alterations in context-dependent reward adaptation and its association with symptom dimensions in the schizophrenia spectrum.
Method: We studied 27 patients with first-episode psychosis, 26 individuals with schizotypal personality traits and 25 healthy controls. We used functional magnetic resonance imaging in combination with a variant of the monetary incentive delay task and assessed adaptive reward coding in two reward conditions with different reward ranges.
Results: Compared to healthy controls, patients with first-episode psychosis and healthy individuals with schizotypal personality traits showed a deficit in increasing the blood-oxygen-level dependent response slope in the right caudate for the low reward range compared to the high reward range. In other words, the two groups showed inefficient neural adaptation to the current reward context. In addition, we found impaired adaptive coding of reward in the caudate nucleus and putamen to be associated with total symptom severity across the schizophrenia spectrum. Symptom severity was more strongly associated with neural deficits in adaptive coding than with the neural coding of absolute reward outcomes.
Conclusions: Deficits in adaptive coding were prominent across the schizophrenia spectrum and even detectable in unmedicated (healthy) individuals with schizotypal personality traits. Furthermore, the association between total symptom severity and impaired adaptive coding in the right caudate and putamen suggests a dimensional mechanism underlying imprecise neural adaptation. Our findings support the idea that impaired adaptive coding may be a general information-processing deficit explaining disturbances within the schizophrenia spectrum over and above a simple model of blunted absolute reward signals.
- doi pdf Personalized prediction of antidepressant versus placebo response: Evidence from the EMBARC study.
- Psychological Medicine (2018). Epub ahead of print.
- Background: Major Depressive Disorder (MDD) is a highly heterogeneous condition in terms of symptom presentation and, likely, underlying pathophysiology. Accordingly, it is possible that only certain individuals with MDD are well-suited to antidepressants. A potentially fruitful approach to parsing this heterogeneity is to focus on promising endophenotypes of depression, such as neuroticism, anhedonia and cognitive control deficits. Methods: Within an eight-week multisite trial of sertraline vs. placebo for depressed adults (n =216), we examined whether the combination of machine learning with a Personalized Advantage Index (PAI) can generate individualized treatment recommendations on the basis of endophenotype profiles coupled with clinical and demographic characteristics. Results: Five pre-treatment variables moderated treatment response. Higher depression severity and neuroticism, older age, less impairment in cognitive control and being employed were each associated with better outcomes to sertraline than placebo. Across 1000 iterations of a 10-fold cross-validation, the PAI model predicted that 31% of the sample would exhibit a clinically meaningful advantage (post-treatment Hamilton Rating Scale for Depression [HRSD] difference # 3) with sertraline relative to placebo. Although there were no overall outcome differences between treatment groups (d =.15), those identified as optimally suited to sertraline at pre- treatment had better week 8 HRSD scores if randomized to sertraline (10.7) than placebo (14.7)(d =.58). Conclusions: A subset of MDD patients optimally suited to sertraline can be identified on the basis of pre-treatment characteristics. This model must be tested prospectively before it can be used to inform treatment selection. However, findings demonstrate the potential to improve individual outcomes through algorithm-guided treatment recommendations.
- doi pdf Drunk decisions: Alcohol shifts choice from habitual towards goal-directed control in adolescent intermediate-risk drinkers
- J Psychopharmacol (2018) 32(8):855-866
- Background: Studies in humans and animals suggest a shift from goal-directed to habitual decision-making in addiction. We therefore tested whether acute alcohol administration reduces goal-directed and promotes habitual decision-making, and whether these effects are moderated by self-reported drinking problems. Methods: Fifty-three socially drinking males completed the two-step task in a randomised crossover design while receiving an intravenous infusion of ethanol (blood alcohol level=80 mg%), or placebo. To minimise potential bias by long-standing heavy drinking and subsequent neuropsychological impairment, we tested 18- to 19-year-old adolescents. Results: Alcohol administration consistently reduced habitual, model-free decisions, while its effects on goal-directed, model-based behaviour varied as a function of drinking problems measured with the Alcohol Use Disorders Identification Test. While adolescents with low risk for drinking problems (scoring <8) exhibited an alcohol-induced numerical reduction in goal-directed choices, intermediate-risk drinkers showed a shift away from habitual towards goal-directed decision-making, such that alcohol possibly even improved their performance. Conclusions: We assume that alcohol disrupted basic cognitive functions underlying habitual and goal-directed decisions in low-risk drinkers, thereby enhancing hasty choices. Further, we speculate that intermediate-risk drinkers benefited from alcohol as a negative reinforcer that reduced unpleasant emotional states, possibly displaying a novel risk factor for drinking in adolescence.
- doi pdf Advancing Clinical Improvements for Patients Using the Theory-Driven and Data-Driven Branches of Computational Psychiatry
- JAMA Psychiatry (2018). 75(3):225-226
- This viewpoint is part of a series on pragmatic evidence-based psychiatry that arises from the tangible disillusionment with the speed at which explanatory advances in our understanding of the brain have been translated into clinical improvements for patients. Here, I argue that both theory-driven and data-driven branches of computational psychiatry can advance this pragmatic agenda.
- Neural correlates of instrumental responding in the context of alcohol-related cues index disorder severity and risk
- Eur. Arch. Psychiatry Clin. Neurosci. (2017). In Press.
- The influence of Pavlovian conditioned stimuli on ongoing behavior may contribute to explaining how alcohol cues stimulate drug seeking and intake. Using a Pavlovian-instrumental transfer (PIT) task, we investigated the effects of alcohol-related cues on approach behavior (i.e., instrumental response behavior) and its neural correlates, and related both to relapse after detoxification in alcohol-dependent patients. Thirty-one recently detoxified alcohol-dependent patients and 24 healthy controls underwent instrumental training where approach or non-approach towards initially neutral stimuli was reinforced by monetary incentives. Approach behavior was tested during extinction with either alcohol-related or neutral stimuli (as Pavlovian cues) presented in the background during functional magnetic resonance imaging (fMRI). Patients were subsequently followed up for six months. We observed that alcohol-related background stimuli inhibited approach behavior in detoxified alcohol-dependent patients (t=-3.86, p<.001) but not in healthy controls (t=-0.92, p=.36). This behavioral inhibition was associated with neural activation in the nucleus accumbens (NAcc) (t(30)=2.06, p<.05). Interestingly, both effects were only present in subsequent abstainers but not relapsers and in those with mild but not severe dependence. Our data show that alcohol-related cues can acquire inhibitory behavioral features typical of aversive stimuli despite being accompanied by a stronger NAcc activation, suggesting salience attribution. The fact that these findings are restricted to abstinence and milder illness suggests that they may be potential resilience factors. Clinical trial: LeAD study, www.lead-studie.de, NCT01679145
- doi pdf The neural basis of aversive Pavlovian guidance during planning
- J. Neurosci. (2017) 37(42):10215-10229
- Important real-world decisions are often arduous as they frequently involve sequences of choices, with initial selections affecting future options. Evaluating every possible combination of choices is computationally intractable, particularly for longer multi-step decisions. Therefore, humans frequently employ heuristics to reduce the complexity of decisions. We recently used a goal-directed planning task to demonstrate the profound behavioral influence and ubiquity of one such shortcut, namely aversive pruning, a reflexive Pavlovian process that involves neglecting parts of the decision space residing beyond salient negative outcomes. However, how the brain implements this important decision heuristic, and what underlies individual differences in its strength have hitherto remained unanswered. Therefore, we administered an adapted version of the same planning task to healthy volunteers undergoing functional magnetic resonance imaging (fMRI) to determine the neural basis of aversive pruning. Through both computational and standard categorical fMRI analyses, we show that when planning was influenced by aversive pruning, the subgenual cingulate cortex was robustly recruited. This neural signature was distinct from those associated with general planning and valuation, two fundamental cognitive components elicited by our task but which are complementary to aversive pruning. Furthermore, we found that individual variation in levels of aversive pruning were associated with the responses of insula and dorsolateral prefrontal cortex to the receipt of large monetary losses, and also with sub-clinical levels of anxiety. In summary, our data reveal the neural signatures of an important reflexive Pavlovian processes that shapes goal-directed evaluations, and thereby determine the outcome of high-level sequential cognitive processes.
- doi pdf Theory-based computational psychiatry
- Biological Psychiatry (2017) 82: 382--384
- Editorial for special issue on computational psychiatry.
- pdf Bayesian approaches to Learning and Decision Making
- in: Computational Psychiatry: Mathematical Modelling of Mental Disorders. Editors: Alan Anticevic and John Murray
- Behavioural phenomena are central to psychiatric disorders. Computational modelling allows the learning and decision-making processes underlying behaviour to be modelled in great detail. By doing so, specific and possibly highly complex hypotheses about the underlying processes can be directly tested on the data. The first part of this chapter introduces Markov Decision Problems (MDPs) as a formal framework for decision-making. It then describes several solutions to MDPs including reinforcement learning and dynamic programming, and briefly introduces some of their key characteristics. The second part of the chapter provides a tutorial overview over how to use MDPs in a generative modelling framework to test hypotheses about learning and decision-making. The final part of the chapter discusses the methods using a few worked examples from the literature.
- doi pdf A formal valuation framework for emotions and their control
- Biological Psychiatry (2017) 82:413--420
- Computational psychiatry attempts to apply mathematical and computational techniques to help improve psychiatric care. Here, we consider formal valuation accounts of emotion. The flexibility of emotional responses and the nature of appraisals suggest the need for a model-based framework for emotions. Resource limitations make plain model-based valuation impossible, and require strategies to apportion cognitive resources adaptively. We argue that emotions can implement such approximations by restricting the range of behaviours and states considered. We consider the processes that guide the deployment of the emotional approximations, discerning between innate, model-free, heuristic and model-based controllers. A focus on complex model-based decisions reveals the necessity for strategies to deal with the complexity of the problems. Emotions may provide such approximations, and this framework may provide a principled approach to examining them.
- doi pdf Strong seduction - Impulsivity and the impact of contextual cues on instrumental behavior in alcohol dependence
- Translational Psychiatry (2017) 7, e1183
- Alcohol-related cues acquire incentive salience through Pavlovian conditioning and then can markedly affect instrumental behavior of alcohol-dependent patients to promote relapse. However, it is unclear whether similar effects occur with alcohol-unrelated cues. We tested 116 early abstinent alcohol-dependent patients and 91 healthy controls who completed a delay discounting task to assess choice impulsivity, and a Pavlovian-to-instrumental transfer (PIT) paradigm employing both alcohol-unrelated and alcohol-related stimuli. To modify instrumental choice behavior, we tiled the background of the computer screen either with conditioned stimuli (CS) previously generated by pairing abstract pictures with pictures indicating monetary gains or losses, or with pictures displaying alcohol or water beverages. CS paired to money gains and losses affected instrumental choices differently. This PIT effect was significantly more pronounced in patients compared to controls, and the group difference was mainly driven by highly impulsive patients. The PIT effect was particularly strong in trials in which the instrumental stimulus required inhibition of instrumental response behavior and the background CS was associated to monetary gains. Under that condition, patients performed inappropriate approach behavior, contrary to their previously formed behavioral intention. Surprisingly, the effect of alcohol and water pictures as background stimuli resembled that of aversive and appetitive CS, respectively. These findings suggest that positively valenced background CS can provoke dysfunctional instrumental approach behavior in impulsive alcohol-dependent patients. Consequently, in real life they might be easily seduced by environmental cues to engage in actions thwarting their long-term goals. Such behaviors may include, but are not limited to, approaching alcohol.
- doi pdf The impact of traumatic stress on Pavlovian biases
- Psychol Med (2017)
-
Disturbances in Pavlovian valuation systems are reported to follow traumatic stress exposure. However, motivated decisions are also guided by instrumental mechanisms, but to date the effect of traumatic stress on these instrumental systems remain poorly investigated. Here, we examine whether a single episode of severe traumatic stress influences flexible instrumental decisions through an impact on a Pavlovian system. Twenty-six survivors of the 2011 Norwegian terror attack and 30 matched control subjects performed an instrumental learning task in which Pavlovian and instrumental associations promoted congruent or conflicting responses. We used reinforcement learning models to infer how traumatic stress affected learning and decision-making. Based on the importance of dorsal anterior cingulate cortex (dACC) for cognitive control, we also investigated if individual concentrations of Glx (=glutamate + glutamine) in dACC predicted the Pavlovian bias of choice. Survivors of traumatic stress expressed a greater Pavlovian interference with instrumental action selection and had significantly lower levels of Glx in the dACC. Across subjects, the degree of Pavlovian interference was negatively associated with dACC Glx concentrations. Experiencing traumatic stress appears to render instrumental decisions less flexible by increasing the susceptibility to Pavlovian influences. An observed association between prefrontal glutamatergic levels and this Pavlovian bias provides novel insight into the neurochemical basis of decision-making, and suggests a mechanism by which traumatic stress can impair flexible instrumental behaviours.
- doi pdf When habits are dangerous: alcohol expectancies and habitual decision-making in alcohol dependence
- Biological Psychiatry (2017) 82(11):847-856 Commentary by T. Duka
- Addiction is supposed to be characterized by a shift from goal-directed to habitual decision-making, thus facilitating automatic drug intake. The two-step task allows distinguishing between these mechanisms by computationally modelling goal-directed and habitual behavior as model-based and model-free control. In addicted patients, decision-making may also strongly depend upon drug-associated expectations. Therefore, we investigated model-based vs. model-free decision-making and its neural correlates as well as alcohol expectancies in alcohol-dependent patients and healthy controls and assessed treatment outcome in patients. Ninety detoxified, medication-free alcohol-dependent patients and 96 age- and gender-matched control underwent functional magnetic resonance imaging during the two-step task. Alcohol expectancies were measured with the Alcohol Expectancy Questionnaire.. Over a follow-up period of 48 weeks, 37 patients remained abstinent whereas 53 patients. Patients who relapsed displayed reduced medial prefrontal cortex (mPFC) activation during model-based decision-making. Furthermore high alcohol expectancies were associated with low model-based control in relapsers, while the opposite was observed in abstainers and healthy controls. However, reduced model-based control per se was not associated with subsequent relapse. These findings suggest that poor treatment outcome in addicted patients does not simply result from reduced model-based control but is rather dependent on the interaction between high drug expectancies and low model-based decision-making. Reduced model-based mPFC signatures in prospective relapsers point to a neural correlate of relapse risk. These observations suggest that therapeutic interventions should target subjective alcohol expectancies.
- doi pdf No association of goal-directed and habitual control with alcohol consumption in young adults
- Addiction Biology (2017)
- Alcohol dependence is a mental disorder which has been associated with an imbalance in behavioral control favoring model-free habitual over model-based goal-directed strategies. It is as yet unknown, however, whether such an imbalance reflects a predisposing vulnerability or results as a consequence of repeated and/or excessive alcohol exposure. We, therefore, examined the association of alcohol consumption with model-based goal-directed and model-free habitual control in 188 eighteen-year-old social drinkers in a two-step sequential decision-making task while undergoing fMRI before prolonged alcohol misuse could have led to severe neurobiological adaptations. Behaviorally, participants showed a mixture of model-free and model-based decision-making as observed previously. Measures of impulsivity were positively related to alcohol consumption. In contrast, neither model-free nor model-based decision weights nor the tradeoff between them were associated with alcohol consumption. There were also no significant associations between alcohol consumption and neural correlates of model-free or model-based decision quantities in either ventral striatum or ventromedial prefrontal cortex. Exploratory whole-brain fMRI analyses with a lenient threshold revealed early onset of drinking to be associated with an enhanced representation of model-free reward prediction errors in the posterior putamen. These results suggest that an imbalance between model-based goal-directed and model-free habitual control might rather not be a trait marker of alcohol intake per se.
- doi pdf The anchoring bias reflects rational use of cognitive resources.
- Psychonomic Bulletin & Review (2017)
- Cognitive biases, such as the anchoring bias, pose a serious challenge to rational accounts of human cognition. We investigate whether rational theories can meet this challenge by taking into account the mind's bounded cognitive resources. We asked what reasoning under uncertainty would look like if people made rational use of their finite time and limited cognitive resources. To answer this question, we applied a mathematical theory of bounded rationality to the problem of numerical estimation. Our analysis led to a rational process model that can be interpreted in terms of anchoring-and-adjustment. This model provided a unifying explanation for ten anchoring phenomena including the differential effect of accuracy motivation on the bias towards provided versus self-generated anchors. Our results illustrate the potential of resource-rational analysis to provide formal theories that can unify a wide range of empirical results and reconcile the impressive capacities of the human mind with its apparently irrational cognitive biases.
- doi pdf Empirical evidence for resource-rational anchoring and adjustment
- Psychonomic Bulletin & Review (2017)
- People's estimates of numerical quantities are systematically biased towards their initial guess. This anchoring bias is usually interpreted as sign of human irrationality, but it has recently been suggested that the anchoring bias instead results from people's rational use of their finite time and limited cognitive resources. If this were true, then adjustment should decrease with the relative cost of time. To test this hypothesis, we designed a new numerical estimation paradigm that controls people's knowledge and varies the cost of time and error independently while allowing people to invest as much or as little time and effort into refining their estimate as they wish. Two experiments confirmed the prediction that adjustment decreases with time cost but increases with error cost regardless of whether the anchor was self-generated or provided. These results support the hypothesis that people rationally adapt their number of adjustments to achieve a near-optimal speed-accuracy tradeoff. This suggests that the anchoring bias might be a signature of the rational use of finite time and limited cognitive resources rather than a sign of human irrationality.
- doi pdf Value-based decision-making battery: A Bayesian adaptive approach to assess impulsive and risky behavior
- Behav Res Methods (2017)
- Using simple mathematical models of choice behavior, we present a Bayesian adaptive algorithm to assess measures of impulsive and risky decision making. Practically, these measures are characterized by discounting rates and are used to classify individuals or population groups, to distinguish unhealthy behavior, and to predict developmental courses. However, a constant demand for improved tools to assess these constructs remains unanswered. The algorithm is based on trial-by-trial observations. At each step, a choice is made between immediate (certain) and delayed (risky) options. Then the current parameter estimates are updated by the likelihood of observing the choice, and the next offers are provided from the indifference point, so that they will acquire the most informative data based on the current parameter estimates. The procedure continues for a certain number of trials in order to reach a stable estimation. The algorithm is discussed in detail for the delay discounting case, and results from decision making under risk for gains, losses, and mixed prospects are also provided. Simulated experiments using prescribed parameter values were performed to justify the algorithm in terms of the reproducibility of its parameters for individual assessments, and to test the reliability of the estimation procedure in a group-level analysis. The algorithm was implemented as an experimental battery to measure temporal and probability discounting rates together with loss aversion, and was tested on a healthy participant sample.
- doi pdf Predicting relapse after antidepressant withdrawal - a systematic review
- Psychol. Med. (2017) 47(3):426-437
-
Background: A substantial proportion of the burden of depression arises from its recurrent nature. The risk of relapse after antidepressant (ADM) discontinuation is high but not uniform. Predictors of individual relapse risk after antidepressant discontinuation could help to guide treatment and mitigate the long-term course of depression.
Methods: We conducted a systematic literature search in Pubmed to identify relapse predictors using the search terms "(depress* OR MDD*) AND (relapse* OR recurren*) AND (predict* OR risk) AND (discontinu* OR withdraw* OR maintenance OR maintain or continu*) AND (antidepress* OR medication OR drug)" for published studies until November 2014. Studies investigating predictors of relapse in patients aged between 18 and 65 with a main diagnosis of Major Depressive Disorder (MDD) who remitted from a depressive episode while treated with antidepressant medication and were followed up for at least 6 months to assess relapse after part of the sample discontinued their ADM, were included in the review.
Results: Although relevant information is present in many studies, only thirteen studies based on nine separate samples investigated predictors for relapse after ADM discontinuation. There are multiple promising predictors, including markers of true treatment response and the number of prior episodes. However, the existing evidence is weak and there are no established, validated markers of individual relapse risk after antidepressant cessation.
Conclusion: There is little evidence to guide discontinuation decisions in an individualized manner beyond overall recurrence risk. Thus, there is a pressing need to investigate neurobiological markers of individual relapse risk, focusing on treatment discontinuation.
- pdf A valuation framework for emotions applied to depression and recurrence
- In: Computational Psychiatry: New Perspectives on Mental Illness, edited by A. D. Redish and J. A. Gordon. Strüngmann Forum Reports, vol. 20, J. Lupp, series editor. Cambridge, MA: MIT Press
-
The burden of depression is substantially aggravated by relapses and recurrences, and these become more inevitable with every episode of depression. This chapter first describes how computational psychiatry can provide a normative framework for emotions that might provide an integrative approach to core cognitive components of depression and relapse. At the heart of this account is the notion that emotions effectively imply a valuation, and that they are therefore amenable to description and dissection by reinforcement-learning methods. It is argued that cognitive accounts of emotion can be viewed in terms of model-based valuation, and that automatic emotional responses relate to model-free valuation and the innate recruitment of fixed behavioural patterns. The model-based view captures phenomena such as helplessness, hopelessness, attributions and stress sensitization. Considering it in more atomic algorithmic detail opens up the possibility of viewing rumination and emotion regulation in this same normative framework, too. The chapter then briefly outlines the problem of treatment selection for relapse and recurrence prevention, and then suggests ways in which the computational framework of emotions might help in improving this. The discussion closes with a very brief general overview over what we can hope to gain from computational psychiatry.
- pdf Complexity and Heterogeneity in psychiatric disorders.
- In: Computational Psychiatry: New Perspectives on Mental Illness, edited by A. D. Redish and J. A. Gordon. Strüngmann Forum Reports, vol. 20, J. Lupp, series editor. Cambridge, MA: MIT Press
-
Psychiatry faces a number of challenges, among them are the reconceptualization of symptoms and diagnoses, disease prevention, treatment development and monitoring of its effects, and the provision of individualized, precision medicine. Achieving these goals will require an increase in the biological, quantitative, and theoretical grounding of psychiatry. To address these challenges, psychiatry must confront the complexity and heterogeneity intrinsic to the nature of brain disorders. This chapter seeks to identify the sources of complexity and heterogeneity as a means of confronting the challenges facing the field. These sources include the interplay between genetic and epigenetic factors with the environment and their impact on neural circuits. Moreover, these interactions are expressed dynamically over the course of development and continue to play out during the disease process and treatment.
We propose that computational approaches provide a framework for addressing the complexity and heterogeneity that underlie the challenges facing psychiatry. Central to our argument is the idea that these characteristics are not noise to be eliminated from diagnosis and treatment of disorders. Instead, such complexity and heterogeneity arises from intrinsic features of brain function and, therefore, represent opportunities for computational models to provide a more accurate biological foundation for diagnosis and treatment of psychiatric disorders. The challenges to be addressed by a computational framework include the following. First, it must improve the search for risk factors and biomarkers, which can be used toward primary prevention of disease. Second, it must help to represent the biological ground truth of psychiatric disorders, which will improve the accuracy of diagnostic categories, assist in discovering new treatments, and aid in precision medicine. Third, to be useful for secondary prevention, it must represent how risk factors, biomarkers, and the underlying biology change through the course of development, disease progression, and treatment process.
- preprint How do emotion and cognition interact?
- In Fox AS et al., (eds), The nature of emotion, 2nd edition (2016) In Press
- pdf Computational Psychiatry
- Zeitschrift für Psychiatrie, Psychologie und Psychotherapie. 65(1):21--26.
-
"Computational Psychiatry" ist eine neue Forschungsrichtung, die Fortschritte aus den theoretischen und experimentellen Neurowissenschaften in klinische Anwendungen für die Psychiatrie umzusetzen will. Der mögliche Nutzen mathematischer Modelle für psychiatrische Anwendungen ergibt sich vor allem aus der Komplexität psychiatrischer Phänomene, deren Beherrschung neue analytische Herangehensweisen erfordert. Konkret können mithilfe solcher Modelle erstens innerpsychische und ansonsten nicht direkt messbare Prozesse erfasst werden. Ein Beispiel hierfür sind Lernprozesse. Zweitens können Phänomene auf verschiedenen Ebenen quantitativ miteinander in Verbindung gebracht werden, z.B. der Effekt von Ionenkanalstörungen auf das Kurzzeitgedächtnis. Drittens können Methoden aus dem maschinellen Lernen mit diesen Modellen verbunden werden, um grosse Datensätze zu analysieren. Obwohl erste Ansätze aus dieser Forschung schon möglichen klinischen Nutzen erwiesen haben, ist das Feld noch jung. Der Artikel schliesst mit dem Vorschlag, Prozeduren aus der Entwicklung pharmazeutischer Produkte für die Validierung theoretischer Anwendungen herbeizuziehen.
- doi pdf Computational psychiatry: from mechanistic insights to the development of new treatments
- Biological Psychiatry: Cognitive Neuroscience and Neuroimaging (2016) 1(5):382-385
-
Computational psychiatry is a young field that aims to further our understanding of mental illness and its treatment with the use of novel computational techniques. The present issue provides an overview over the breadth of the field. On the one hand, computational techniques can be used to provide mechanistic insight into illnesses. This is exemplified with contributions using Bayesian and reinforcement-learning techniques into schizophrenia, methamphetamine and alcohol use disorders. On the other hand, mechanistically agnostic techniques can directly infer information relevant to treatment. Examples in the issue include prediction of depression treatment responses with EEG and response prediction with fMRI. The issue concludes with a novel way to address heterogeneity, and finally with a proposal to adopt a developmental pathway akin to that in drug development to ensure computational psychiatry fulfills its promise to improve patient outcomes.
- doi pdf Model-free temporal-difference learning and dopamine in alcohol dependence: examining concepts from theory and animals in human imaging
- Biological Psychiatry: Cognitive Neuroscience and Neuroimaging (2016) 1(5):401-410
-
Dopamine potentially unites two important roles: one in addiction, being involved in most substances of abuse including alcohol, and a second one in a specific type of learning, namely model-free temporal-difference reinforce- ment learning. Theories of addiction have long suggested that drugs of abuse may usurp dopamine's role in learning. We here briefly review the preclinical literature to motivate specific hypotheses about model-free temporal-difference learning, and then review the imaging evidence in the drug of abuse with the most substantial societal consequences: alcohol. Despite the breadth of the literature, only very few studies have examined the predictions directly, and these provide at best inconclusive evidence for the involvement of temporal-difference learning alterations in alcohol de- pendence. We discuss the difficulties of testing the theory, make specific suggestions and close with a focus on the interaction with other learning mechanisms.
- doi pdf A Roadmap for the Development of Applied Computational Psychiatry
- Biological Psychiatry: Cognitive Neuroscience and Neuroimaging (2016) 1(5):386-392
-
Background: Computational psychiatry is a burgeoning field that utilizes mathematical approaches to investigate psychiatric disorders, derive quantitative predictions, and integrate data across multiple levels of description. Computational psychiatry has already led to many new insights into the neurobehavioral mechanisms that underlie several psychiatric disorders, but its usefulness from a clinical standpoint is only now starting to be considered. Methods: Examples of computational psychiatry are highlighted, and a phase-based pipeline for the development of clinical computational- psychiatry applications is proposed, similar to the phase-based pipeline used in drug development. It is proposed that each phase has unique endpoints and deliverables, which will be important milestones to move tasks, procedures, computational models, and algorithms from the laboratory to clinical practice. Results: Application of computational approaches should be tested on healthy volunteers in Phase I, transitioned to target populations in Phase IB and Phase IIA, and thoroughly evaluated using randomized clinical trials in Phase IIB and Phase III. Successful completion of these phases should be the basis of determining whether computational models are useful tools for prognosis, diagnosis, or treatment of psychiatric patients. Conclusions: A new type of infrastructure will be necessary to implement the proposed pipeline. This infrastructure should consist of groups of investigators with diverse backgrounds collaborating to make computational psychiatry relevant for the clinic.
- doi pdf Computational neuroimaging strategies for single patient predictions
- Neuroimage (2017) 145(Pt B):180-199
- Neuroimaging increasingly exploits machine learning techniques in an attempt to achieve clinically relevant single-subject predictions. An alternative to machine learning, which tries to establish predictive links between features of the observed data and clinical variables, is the deployment of computational models for inferring on the (patho)physiological and cognitive mechanisms that generate behavioural and neuroimaging responses. This paper discusses the rationale behind a computational approach to neuroimaging-based single-subject inference, focusing on its potential for characterising disease mechanisms in individual subjects and mapping these characterisations to clinical predictions. Following an overview of two main approaches - Bayesian model selection and generative embedding - which can link computational models to individual predictions, we review how these methods accommodate heterogeneity in psychiatric and neurological spectrum disorders, help avoid erroneous interpretations of neuroimaging data, and establish a link between a mechanistic, model-based approach and the statistical perspectives afforded by machine learning.
- doi pdf Don't think, just feel the music: Individuals With Strong Pavlovian-to-instrumental Transfer Effects Rely Less on Model-based Reinforcement Learning
- J Cogn Neurosci (2016) 28(7):985--995
- Behavioral choice can be characterized along two axes. One axis distinguishes reflexive, model-free systems that slowly accumulate values through experience and a model-based system that uses knowledge to reason prospectively. The second axis distinguishes Pavlovian valuation of stimuli from instrumental valuation of actions or stimulus-action pairs. This results in a quartet of values and many possible interactions between them, with important consequences for accounts of individual variation. We here explored whether individual variation along one axis was related to individual variation along the other. Specifically, we asked whether individual's balance between model-based and model-free learning was related to their tendency to show Pavlovian interferences with instrumental decisions. In two independent samples with a total of 243 subjects, Pavlovian-instrumental transfer effects were negatively correlated with the strength of model-based reasoning in a two-step task. This suggests a potential common underlying substrate predisposing individuals to both have strong Pavlovian interference and be less model-based, and provides a framework within which to interpret the observation of both effects in addiction.
- arxiv pdf Better safe than sorry: Risky function exploitation through safe optimization
- Proc Cog Sci 2016
-
Exploration-exploitation of functions, that is learning and optimizing a mapping between inputs and expected outputs, is ubiquitous to many real world situations. These situations sometimes require us to avoid certain outcomes at all cost, for example because they are poisonous, harmful, or otherwise dangerous. We test participants' behavior in scenarios in which they have to find the optimum of a function while at the same time avoid outputs below a certain threshold. In two experiments, we find that Safe-Optimization, a Gaussian Process-based exploration-exploitation algorithm, describes participants' behavior well and that participants seem to care firstly whether a point is safe and then try to pick the optimal point from all such safe points. This means that their trade-off between exploration and exploitation can be seen as an intelligent, approximate, and homeostasis-driven strategy.
- data / code doi pdf German translation and validation of the Cognitive Style Questionnaire short form (CSQ-SF-D)
- PLoS One (2016) 11(3):e0149530
-
Background: The Cognitive Style Questionnaire is a valuable tool for the assessment of hopeless cognitive styles in depression research, with predictive power in longitudinal studies. Even the short form is still long, and neither this nor the original version exist in validated German translations.
Methods: The questionnaire was translated from English to German, back-translated and commented on by clinicians. The reliability, factor structure and external validity of an online form of the questionnaire were examined on 214 participants. External validity was measured on a subset of 90 subjects.
Results: The resulting CSQ-SF-D had good to excellent reliability, both across items and subscales, and similar external validity to the original English version. The internality subscale appeared less robust than other subscales. A detailed analysis of individual item performance suggests that stable results could be achieved with a very short form (CSQ-VSF-D) including only 27 items.
Conclusions: The CSQ-SF-D is a validated and freely distributed translation of the CSQ-SF into German. This should make efficient assessment of cognitive style in German samples more accessible to researchers.
- Rückfallvorhersage bei Depressionen: welche Rolle spielen Schuldgefühle?
- Info Neurologie und Psychiatrie (2016) In Press
- pdf Rückfallvorhersage nach Absetzen von Antidepressiva
- Info Neurologie und Psychiatrie (2016) 14(2):16-19
-
- Nicht alle depressiven Patienten in Remission profitieren von Antidepressiva in Bezug auf R¨ckfallprophylaxe.
- Viele Patienten wollen ihr Antidepressivum absetzen, unter anderem wegen unerwünschten Nebenwirkungen.
- Zurzeit gibt es keine Prädiktoren, die vorhersagen, welcher Patient einen Rückfall bekommt und welcher nicht.
- Möglicherweise können neurobiologische Biomarker als Prädiktoren dienen.
- Die Antidepressiva-Absetzstudie (AIDA) untersucht Prädiktoren, die ein sicheres Absetzen vorhersagen; dazu werden weiterhin Patienten gesucht, die in Remission sind und ihr Antidepressivum absetzen wollen.
- doi pdf Computational psychiatry as a bridge from neuroscience to clinical applications
- Nature Neuroscience (2016) 19(3):404--413 Commentary in Scientific American
-
Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ----the brain---and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction, and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data-driven and theory-driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes, or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.
- doi pdf The specificity of Pavlovian regulation is associated with recovery from depression
- Psychological Medicine (2016) 46(5):1027-1035
-
Background: Changes in reflexive emotional responses are hallmarks of depression, but how emotional reflexes impact on adaptive decision-making in depression has not been examined formally. Using a Pavlovian-Instrumental Transfer (PIT) task, we compared the influence of affectively valenced stimuli on decision-making in depression and generalized anxiety disorder compared with healthy controls; and related this to the longitudinal course of the illness.
Methods: Fourty subjects with a current DSM-IV-TR diagnosis of Major Depressive Disorder, Dysthymia, Generalised Anxiety Disorder or a combination thereof, and 40 matched healthy controls performed a Pavlovian-instrumental transfer (PIT) task that assesses how instrumental approach and withdrawal behaviours are influenced by appetitive and aversive Pavlovian conditioned stimuli. Patients were followed up after 4-6 months. Analyses focussed on patients with depression alone (n=25).
Results: In healthy controls, Pavlovian conditioned stimuli (CSs) exerted action-specific effects, with appetitive CSs boosting active approach and aversive CSs withdrawal. This action-specificity was absent in currently depressed subjects. Greater action-specificity in patients was associated with better recovery over the follow-up period.
Conclusions: Depression is associated with abnormal influence of emotional reactions on decision-making in a way that is predictive of recovery.
- doi pdf Charting the Landscape of Priority Problems in Psychiatry. Part 1: Nosology and Diagnosis
- Lancet Psychiatry (2015)
- Contemporary psychiatry faces major challenges. Its phenomenological nosology continues to lack mechanistic interpretability and predictive guidance; treatment largely depends on trial and error; drug development is impeded through ignorance of potential beneficiary subgroups; neuroscientific and genetics research has yet to impact disease definitions or to contribute to clinical decision-making. In this parlous setting, what should psychiatric research focus on? In two companion papers, we present a list of concrete problems nominated by clinicians and researchers from different disciplines as candidates for future scientific investigation of mental disorders. These problems are loosely grouped into challenges concerning nosology and diagnosis (this article) and those addressing pathogenesis and aetiology (in the companion article). Motivated by successful examples in other disciplines (particularly the famous list of "Hilbert's problems" in mathematics), this subjective and eclectic list of priority problems is intended to provide inspiration for the field, helping to refocus existing research and providing perspectives for future psychiatric science.
- doi pdf Charting the Landscape of Priority Problems in Psychiatry. Part 2: Pathogenesis and Aetiology
- Lancet Psychiatry (2015)
- This is the second of two companion papers proposing priority problems for research on mental disorders. Whereas the first article focuses on questions of nosology and diagnosis, the challenges articulated in this paper concern pathogenesis and aetiology of psychiatric diseases. We hope that this (non-exhaustive and subjective) list of problems, nominated by scientists and clinicians from different fields and institutions, provides guidance and perspectives for choosing future directions in psychiatric science.
- doi pdf Computational Psychiatry: Towards a mathematically informed understanding of mental illness
- J Neurol Neurosurg Psych (2016): 87(1):53-63
- Computational Psychiatry aims to describe the relationship between the brain's neurobiology, its environment and mental symptoms in computational terms. In so doing, it may improve psychiatric classification and the diagnosis and treatment of mental illness. It can unite many levels of description in a mechanistic and rigorous fashion, whilst avoiding biological reductionism and artificial categorization. We describe how computational models of cognition can infer the current state of the environment and weigh up future actions, and how these models provide new perspectives on two example disorders, depression and schizophrenia. Reinforcement learning describes how the brain can choose and value courses of actions according to their long-term future value. Some depressive symptoms may result from aberrant valuations, which could arise from prior beliefs about the loss of agency ("helplessness"), or from an inability to inhibit the mental exploration of aversive events. Predictive coding explains how the brain might perform Bayesian inference about the state of its environment by combining sensory data with prior beliefs, each weighted according to their certainty (or precision). Several cortical abnormalities in schizophrenia might reduce precision at higher levels of the inferential hierarchy, biasing inference towards sensory data and away from prior beliefs. We discuss whether striatal hyperdopaminergia might have an adaptive function in this context, and also how reinforcement learning and incentive salience models may shed light on the disorder. Finally, we review some of Computational Psychiatry's applications to neurological disorders such as Parkinson's disease, and some pitfalls to avoid when applying its methods.
- doi pdf Neural Correlates of Three Promising Endophenotypes of Depression: Evidence from the EMBARC Study
- Neuropsychopharmacology (2016): 41(2):454-63
- Major Depressive Disorder (MDD) is clinically, and likely pathophysiologically, heterogeneous. A potentially fruitful approach to parsing this heterogeneity is to focus on promising endophenotypes. Guided by the NIMH Research Domain Criteria (RDoC) initiative, we used source localization of scalp-recorded EEG resting data to examine the neural correlates of three emerging endophenotypes of depression: neuroticism, blunted reward learning and cognitive control deficits. Data were drawn from the ongoing multi-site EMBARC study. We estimated intracranial current density for standard EEG frequency bands in 82 unmedicated adults with MDD, using Low-Resolution Brain Electromagnetic Tomography (LORETA). Region-of- interest and whole-brain analyses tested associations between resting state EEG current density and endophenotypes of interest. Neuroticism was associated with increased resting gamma (36.544 Hz) current density in the ventral (subgenual) anterior cingulate cortex (ACC) and orbitofrontal cortex (OFC). In contrast, reduced cognitive control correlated with decreased gamma activity in the left dorsolateral prefrontal cortex (dlPFC), decreased theta (6.58 Hz) and alpha2 (10.512 Hz) activity in the dorsal ACC, and increased alpha2 activity in the right dlPFC. Finally, blunted reward learning correlated with lower OFC and left dlPFC gamma activity. Computational modeling of trial-by-trial reinforcement learning further indicated that lower OFC gamma activity was linked to reduced reward sensitivity. Three putative endophenotypes of depression were found to have partially dissociable resting intracranial EEG correlates, reflecting different underlying neural dysfunctions. Overall, these findings highlight the need to parse the heterogeneity of MDD by focusing on promising endophenotypes linked to specific pathophysiological abnormalities.
- doi pdf Serotonin's many meanings elude simple theories
- eLife (2015) 4:e07390
- doi pdf Pavlovian-to-instrumental transfer effects in the nucleus accumbens relate to relapse in alcohol dependence
- Addiction Biology (2016) 21(3):719-31
- In detoxified alcohol-dependent patients, alcohol-related stimuli can promote relapse. However, to date, the mechanisms by which contextual stimuli promote relapse have not been elucidated in detail. One hypothesis is that such contextual stimuli directly stimulate the motivation to drink via associated brain regions like the ventral striatum and thus promote alcohol seeking, intake and relapse. Pavlovian-to-Instrumental-Transfer (PIT) may be one of those behavioral phenomena contributing to relapse, capturing how Pavlovian conditioned (contextual) cues determine instrumental behavior (e.g. alcohol seeking and intake). We used a PIT paradigm during functional magnetic resonance imaging to examine the effects of classically conditioned Pavlovian stimuli on instrumental choices in n=31 detoxified patients diagnosed with alcohol dependence and n=24 healthy controls matched for age and gender. Patients were followed up over a period of 3 months. We observed that (1) there was a significant behavioral PIT effect for all participants, which was significantly more pronounced in alcohol-dependent patients; (2) PIT was significantly associated with blood oxygen level-dependent (BOLD) signals in the nucleus accumbens (NAcc) in subsequent relapsers only; and (3) PIT-related NAcc activation was associated with, and predictive of, critical outcomes (amount of alcohol intake and relapse during a 3 months follow-up period) in alcohol-dependent patients. These observations show for the first time that PIT-related BOLD signals, as a measure of the influence of Pavlovian cues on instrumental behavior, predict alcohol intake and relapse in alcohol dependence.
- doi pdf Decision-theoretic psychiatry
- Clin Psychol Sci (2015) 3(3):400-421
- Psychiatric disorders profoundly impair many aspects of decision-making. Poor choices have negative consequences in the moment, and also make it very hard to navigate complex social environments. Computational neuroscience provides normative, neurobiologically informed, descriptions of the components of decision-making which serve as a platform for a principled exploration of dysfunctions. Here, we identify and discuss three classes of failure-modes arising in these formalisms. They stem from abnormalities in the framing of problems or tasks, from the mechanisms of cognition used to solve the tasks, or from the historical data available from the environment.
- doi pdf Failure modes of the will: From goals to habits to compulsions?
- Am J Psychiatry (2015), 172(3):1-3
-
Good decisions reflect the past and improve the future. Trouble ensues when the rewards of the past conflict with the goals of the future. New goals often fail to override behaviors reinforced in the past, for instance, when patients with obsessive-compulsive disorder (OCD) try to stop the counting behavior that has been their main strategy to avoid aversive intrusions. [...]
In this issue of the Journal, Gillan et al. (5) add to an impressive catalog of studies on the imbalance between habitual and goal-directed decisions in OCD.
- doi pdf The interplay of approximate planning strategies
- PNAS (2015), 112(10):3098-3103 Commentary by Daniel, Schuck & Niv
- Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."
- doi pdf Ventral Striatal Dopamine Reflects Behavioral and Neural Signatures of Model-Based Control during Sequential Decision Making
- PNAS (2015), 112(5):1595-600
- Dual system theories suggest that behavioral control is parsed between a deliberative `model-based' and a more reflexive `model-free' system. A balance of control exerted by these systems is thought to be related to dopamine neurotransmission. However, in the absence of direct measures of human dopamine, it remains unknown whether this reflects a quantitative relation with dopamine either in the striatum or other brain areas. Using a sequential decision task performed during fMRI, combined with striatal measures of dopamine using [18F]DOPA PET, we show that higher presynaptic ventral striatal dopamine levels were associated with a behavioral bias towards more model-based control. Higher presynaptic dopamine in ventral striatum was associated with greater coding of model-based information in lateral prefrontal cortex and diminished coding of model-free prediction errors in ventral striatum. Thus, inter- individual variability in ventral striatal presynaptic dopamine reflects a balance in the behavioral expression and the neural signatures of model-free and model-based control. Our data provide a novel perspective on how alterations in presynaptic dopamine levels might be accompanied by a disruption of behavioral control as observed in aging or neuropsychiatric diseases such as schizophrenia and addiction.
- doi pdf Depression: A Decision-Theoretic Analysis
- Annu Rev Neurosci (2015), 38:1-23
- The manifold symptoms of depression are a common and often transient feature of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioural implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.
- pdf Dualdiagnose Schizophrenie und Sucht
- Pro Mente Sana (2014) 4:28-29
- Menschen mit Schizophrenie weisen ein deutlich höheres Risiko auf als gesunde Menschen, im Laufe ihres Lebens eine Abhängigkeitserkrankung zu entwickeln. Bei PatientInnen, die beide Erkrankungen aufweisen, treten häufiger Rückfälle, motorische Medikamentennebenwirkungen und soziale Probleme auf.
- doi pdf Processing speed enhances model-based over model-free reinforcement learning in the presence of high working memory functioning
- Front. Psychol. (2014) 5:1450
- Theories of decision-making and its neural substrates have long assumed the existence of two distinct and competing valuation systems, variously described as goal-directed versus habitual, or, more recently and based on statistical arguments, as model-free versus model-based reinforcement-learning. Though both have been shown to control choices, the cognitive abilities associated with these systems are under ongoing investigation. Here we examine the link to cognitive abilities, and find that individual differences in processing speed covary with a shift from model-free to model-based choice control in the presence of above-average working memory function. This suggests shared cognitive and neural processes; provides a bridge between literatures on intelligence and valuation; and may guide the development of process models of different valuation components. Furthermore, it provides a rationale for individual differences in the tendency to deploy valuation systems, which may be important for understanding the manifold neuropsychiatric diseases associated with malfunctions of valuation.
- doi pdf Chronic alcohol intake abolishes the relationship between dopamine synthesis capacity and learning signals in ventral striatum
- Eur J Neurosci (2014), 41(4):477-86
- Drugs of abuse elicit dopamine release in ventral striatum, possibly biasing dopamine-driven reinforcement learning towards drug-related reward at the expense of non-drug-related reward. Indeed, reactivity in dopaminergic target areas of patients with alcohol dependence is shifted from non-drug-related stimuli towards drug-related stimuli. Such `hijacked' dopamine signals may impair flexible learning from non-drug-related rewards and thus promote craving for the drug of abuse. Here, we used fMRI to measure ventral striatal activation by reward prediction errors (RPEs) during a probabilistic reversal learning task in recently detoxified alcohol- dependent patients and healthy controls (N=27). The same subjects also underwent FDOPA PET to assess ventral striatal dopamine synthesis capacity. Neither ventral striatal activation by RPEs, nor striatal dopamine synthesis capacity differed between groups. However, ventral striatal coding of RPEs was negatively correlated with craving in patients. Furthermore, we found a negative correlation between ventral striatal coding of RPEs and dopamine synthesis capacity in healthy controls, but not in alcohol-dependent patients. Moderator analyses showed that the magnitude of the association between RPE coding and dopamine synthesis capacity depended on the amount of chronic-habitual alcohol intake. Given the relatively small sample size, a power analysis showed that it is rather unlikely to obtain these results by chance. Using a multimodal imaging approach, this study suggests that dopaminergic modulation of neural learning signals is disrupted in alcohol dependence and this is linked to long-term alcohol intake of patients. Alcohol intake may perpetuate itself by interfering with dopaminergic modulation of neural learning signals in ventral striatum, thus increasing craving for habitual drug intake.
- doi pdf The effects of life stress and neural learning signals on fluid intelligence
- Eur Arch Psychiatry Clin Neurosci (2014), 70(2):122-31
- Fluid intelligence (fluid IQ), defined as the capacity for rapid problem solving and behavioral adaptation, is known to be modulated by learning and experience. Both stressful life events (SLES) and neural correlates of learning [specifically, a key mediator of adaptive learning in the brain, namely the ventral striatal representation of prediction errors (PE)] have been shown to be associated with individual differences in fluid IQ. Here, we examine the interaction between adaptive learning signals (using a well-characterized probabilistic reversal learning task in combination with fMRI) and SLES on fluid IQ measures. We find that the correlation between ventral striatal BOLD PE and fluid IQ, which we have previously reported, is quantitatively modulated by the amount of reported SLES. Thus, after experiencing adversity, basic neuronal learning signatures appear to align more closely with a general measure of flexible learning (fluid IQ), a finding complementing studies on the effects of acute stress on learning. The results suggest that an understanding of the neurobiological correlates of trait variables like fluid IQ needs to take socioemotional influences such as chronic stress into account.
- doi pdf Individual differences in bodily freezing predict emotional biases in decision making
- Front. Behav. Neurosci., (2014), 8:237
- Instrumental decision making has long been argued to be vulnerable to emotional responses. Literature on multiple decision making systems suggests that this emotional biasing might reflect effects of a system that regulates innately specified, evolutionarily preprogrammed responses. To test this hypothesis directly, we investigated whether effects of emotional faces on instrumental action can be predicted by effects of emotional faces on bodily freezing, an innately specified response to aversive relative to appetitive cues. We tested 43 women using a novel emotional decision making task combined with posturography, which involves a force platform to detect small oscillations of the body to accurately quantify postural control in upright stance. On the platform, participants learned whole body approach-avoidance actions based on monetary feedback, while being primed by emotional faces (angry/happy). Our data evidence an emotional biasing of instrumental action. Thus, angry relative to happy faces slowed instrumental approach relative to avoidance responses. Critically, individual differences in this emotional biasing effect were predicted by individual differences in bodily freezing. This result suggests that emotional biasing of instrumental action involves interaction with a system that controls innately specified responses. Furthermore, our findings help bridge (animal and human) decision making and emotion research to advance our mechanistic understanding of decision making anomalies in daily encounters as well as in a wide range of psychopathology.
- doi pdf Pavlovian-to-Instrumental Transfer in alcohol dependence - a pilot study
- Neuropsychobiology (2014) 70(2):111-21
- Background: Pavlovian processes are thought to play an important role in the development, maintenance and relapse of alcohol dependence, possibly by influencing and usurping on- going thought and behavior. The influence of Pavlovian stimuli on on-going behavior is paradigmatically measured by Pavlovian-to-instrumental-transfer (PIT) tasks. These involve multiple stages and are complex. Whether increased PIT is involved in human alcohol dependence is uncertain. We therefore aimed to establish and validate a modified PIT paradigm that would be robust, consistent, and tolerated by healthy controls as well as by patients suffering from alcohol dependence, and to explore whether alcohol dependence is associated with enhanced Pavlovian-Instrumental transfer. Methods: 32 recently detoxified alcohol-dependent patients and 32 age and gender matched healthy controls performed a PIT task with instrumental go/no-go approach behaviours. The task involved both Pavlovian stimuli associated with monetary rewards and losses, and images of drinks. Results: Both patients and healthy controls showed a robust and temporally stable PIT effect. Strengths of PIT effects to drug-related and monetary conditioned stimuli were highly correlated. Patients more frequently showed a PIT effect and the effect was stronger in response to aversively conditioned CSs (conditioned suppression), but there was no group difference in response to appetitive CSs. Conclusion: The implementation of PIT has favorably robust properties in chronic alcohol- dependent patients and in healthy controls. It shows internal consistency between monetary and drug-related cues. The findings support an association of alcohol dependence with an increased propensity towards PIT.
- doi pdf Model-based and model-free decisions in alcohol dependence
- Neuropsychobiology (2014) 70(2):122-31
- Background: Human and animal work suggests a shift from goal-directed to habitual decision-making in addiction. However, the evidence for this in human alcohol dependence is as yet inconclusive. Methods: Twenty-six healthy controls and twenty-six recently detoxified alcohol-dependent patients underwent behavioral testing with a two-step task designed to disentangle goal- directed and habitual response patterns. Results: Alcohol-dependent patients showed less evidence of goal-directed choices than healthy controls, particularly after losses. There was no difference in the strength of the habitual component. The group differences did not survive controlling for performance on the digit symbol substitution task. Conclusion: Chronic alcohol use appears to selectively impair goal-directed function, rather than promoting habitual responding. It appears to do so particularly after non-rewards, and this may be mediated by the effects of alcohol on more general cognitive functions subserved by the prefrontal cortex.
- doi pdf Optimism as a prior belief about the probability of future reward
- PLoS Comp Biol (2014) 10(5): e1003605
- Optimists hold positive a priori beliefs about the future. In Bayesian statistical theory, a priori beliefs can be overcome by experience. However, optimistic beliefs can at times appear surprisingly resistant to evidence, suggesting that optimism might also influence how new information is selected and learned. Here, we use a novel Pavlovian conditioning task, embedded in a normative framework, to directly assess how trait optimism, as classically measured using self-report questionnaires, influences choices between visual targets, as learning about their association with reward progresses. We find that trait optimism relates to an a priori belief about the likelihood of rewards, but not losses, in our task. Critically, this positive belief behaves like a probabilistic prior, i.e. its influence reduces with increasing experience. Contrary to findings in the literature related to unrealistic optimism and self-beliefs, it does not appear to influence the iterative learning process directly.
- doi pdf Model-based and model-free decision-making
- In Jaeger, D and Jung, R (Ed): Encyclopædia of Computational Neuroscience. SpringerReference. Springer Verlag Berlin Heidelberg, 2014.
- Reinforcement learning (RL) techniques are a set of solutions for optimal long-term action choice such that actions take into account both immediate and delayed consequences. They fall into two broad classes. Model-based approaches assume an explicit model of the environment and the agent. The model describes the consequences of actions and the associated returns. From this, optimal policies can be inferred. Psychologically, model-based descriptions apply to goal-directed decisions, in which choices reflect current preferences over outcomes. Model-free approaches forgo any explicit knowledge of the dynamics of the environment or the consequences of actions and evaluate how good actions are through trial-and-error learning. Model-free values underlie habitual and Pavlovian conditioned responses that are emitted reflexively when faced with certain stimuli. While model-based techniques have substantial computational demands, model-free techniques require extensive experience.
- preprint Neurobiology and computational structure of decision-making in addiction
- Mishara et al. (ed.): Phenomenological Neuropsychiatry: Bridging the clinic and clinical neuroscience
- An increasing wealth of experimental detail is becoming available about the development and nature of addiction. Critical issues such as the varying vulnerabilities of individuals who develop addiction are being illuminated across levels of phenomenological, psychological and neurobiological detail. Furthermore, a rich theoretical understanding is emerging in the field of neural reinforcement learning, with glimmers as to how this might be related to the subjective experience of those individuals affected. In this chapter, we consider some particularly pressing current issues in the interface between experiment and theory, notably the so-called "compulsive" phase of drug taking.
- doi pdf The role of learning-related dopamine signals in addiction vulnerability
- Progress in Neurobiology (2014): 211:31-77
- Dopaminergic signals play a mathematically precise role in reward-related learning, and variations in dopaminergic signalling have been implicated in vulnerability to addiction. Here, we provide a detailed overview of the relationship between theoretical, mathematical and experimental accounts of phasic dopamine signalling, with implications for the role of learning-related dopamine signalling in addiction and related disorders. We describe the theoretical and behavioural characteristics of model-free learning based on errors in the prediction of reward, including step-by-step explanations of the underlying equations. We then use recent insights from an animal model that highlights individual variation in learning during a Pavlovian conditioning paradigm to describe overlapping aspects of incentive salience attribution and model-free learning. We argue that this provides a computationally coherent account of some features of addiction.
- doi pdf
- Striatal dysfunction during reversal learning in unmedicated schizophrenia patients
- Neuroimage (2014) 89:171-180
- Subjects with schizophrenia are impaired at reinforcement-driven reversal learning from as early as their first episode. The neurobiological basis of this deficit is unknown. We obtained behavioral and fMRI data in 24 unmedicated, primarily first episode, schizophrenia patients and 24 age-, IQ- and gender-matched healthy controls during a reversal learning task. We supplemented our fMRI analysis, focusing on learning from prediction errors, with detailed computational modeling to probe task solving strategy including an ability to deploy an internal goal directed model of the task. Patients displayed reduced functional activation in the ventral striatum (VS) elicited by prediction errors. However, modeling task performance revealed that a subgroup did not adjust their behavior according to an accurate internal model of the task structure, and these were also the more severely psychotic patients. In patients who could adapt their behavior, as well as in controls, task solving was best described by cognitive strategies according to a Hidden Markov Model. When we compared patients and controls who acted according to this strategy, patients still displayed a significant reduction in VS activation elicited by informative errors that precede salient changes of behavior (reversals). Thus, our study shows that VS dysfunction in schizophrenia patients during reward-related reversal learning remains a core deficit even when controlling for task solving strategies. This result highlights VS dysfunction is tightly linked to a reward- related reversal learning deficit in early, unmedicated schizophrenia patients.
- doi pdf
- Serotonin and aversive Pavlovian control of instrumental behavior in humans
- J. Neuroscience (2013), 33(48): 18932-18939
- Adaptive decision-making involves interaction between systems regulating Pavlovian and instrumental control of behavior. Here we investigate in humans the role of serotonin in such Pavlovian-instrumental transfer in both the aversive and the appetitive domain using acute tryptophan depletion, known to lower central serotonin levels. Acute tryptophan depletion attenuated the inhibiting effect of aversive Pavlovian cues on instrumental behavior, while leaving unaltered the activating effect of appetitive Pavlovian cues. These data suggest that serotonin is selectively involved in Pavlovian inhibition due to aversive expectations and have implications for our understanding of the mechanisms underlying a range of affective, impulsive, and aggressive neuropsychiatric disorders.
- doi pdf Differential, but not opponent, effects of L-DOPA and citalopram on action learning with reward and punishment
- Psychopharmacology (2013): 231:955-966.
- Background: Decision-making involves two fundamental axes of control namely valence, spanning reward and punishment, and action, spanning invigoration and inhibition. We recently used a task whose contingencies explicitly decouple valence and action to show that these axes are inextricably coupled during learning. This results in a disadvantage in acquiring active choices in punished conditions and passive choices in rewarded conditions. The neuromodulators dopamine and serotonin are likely to play a role in these asymmetries. Dopamine signals anticipation of future rewards and is involved in an invigoration of motor responses leading to reward. Serotonin is associated with motor inhibition and punishment processing. Methods: Here we combined computational modelling with a pharmacological manipulation of dopamine and serotonin to examine acquisition of instrumental responding in a task that crosses action (go/no go) with valence (reward/punishment) in healthy human volunteers. Results: Contrary to expectation we found that levodopa decreased the coupling of action and valence that was evident in the placebo and citalopram groups. Citalopram had distinct effects and increased participants tendency to perform active responses independent of outcome valence, consistent with decreased motor inhibition. Conclusion: The current data highlights the importance of orthogonally manipulating action requirements and outcome valence if one wants to reveal the full complexity of the roles played by dopamine and serotonin in instrumental learning.
- doi pdf Computational Psychiatry
- Encyclopædia of Computational Neuroscience (2014) 1-10
-
Computational Psychiatry is a heterogeneous field at the intersection of computational neuroscience and psychiatry. Incorporating methods from psychiatry, psychology, neuroscience, behavioural economics and machine learning, computational psychiatry focuses on building mathematical models of neural or cognitive phenomena relevant to psychiatric diseases. The models span a wide range - from biologically detailed models of neurons or networks to abstract models describing high-level cognitive abilities of an organism. Psychiatric diseases are conceptualized as either an extreme of normal function, or as a consequence of alterations in parts of the model.
As in computational neuroscience more generally, the building of models forces key concepts to be made concrete and hidden assumptions to be made explicit. One critical functions of these models in the setting of psychiatry are their ability to bridge between low-level biological and high-level cognitive features. While many neurobiological alterations are known, the exclusively atheoretical focus of standard psychiatric nosology on high-level symptoms has as yet prevented an integration of these bodies of knowledge. David Marr pointed out that models at different levels may be independent (Marr, 1982). Nevertheless, implementational details may constrain functions at the computational level. The models used in computational psychiatry make these constraints explicit, and thereby aim to provide normative conduits between the different levels at which neural systems are analysed (Stephan et al., 2006; Huys et al., 2011; Hasler, 2012; Montague et al., 2012). This in turn allows for a principled approach to study dysfunctions, and indeed may allow the dysfunctions observed in psychiatry to inform neuroscience in general.
Practically, it underpins hopes that computational techniques may facilitate the development of a psychiatric nomenclature based on an understanding of the underlying neuroscience. Computational models enhance experimental designs by allowing more intricate neural and/or cognitive processes to be inferred from complex features of the data, often via Bayesian inference. These aspects motivate hopes that it may facilitate the development of clinical treatment decision tools informed by advances in neuroscience.
- doi pdf Pavlovian-Instrumental Transfer Effects in Alcohol Dependence
- Sucht (2013) 59(4):215-223
-
Hintergrund: Alkoholabhängigkeit ist eine Verkettung ungünstiger Entscheidungen in Bezug auf Alkoholkonsum. Dieses Entscheidungsmuster scheint sich u. a. wegen pawlowsch-instrumentellen Transfereffekten (PIT-Effekten) immer wieder zu reproduzieren. Ziel dieser Literaturzusammenschau ist, wichtige Befunde zum Zusammenhang zwischen PIT-Effekten und Suchterkrankungen zusammenzutragen und offene Fragen im Hinblick auf PIT bei Alkoholabhängigkeit aufzuzeigen. Methoden: Die Literaturzusammenschau nutzte keine systematische Literaturrecherche, sondern basierte auf den Recherchen im Rahmen der DFG Forschergruppe 1617 (Learning and Habitization in Alcohol Dependence, LeAD). Ergebnisse: PIT-Effekte könnten im Zusammenhang mit Alkoholabhängigkeit möglicherweise zu einem Teufelskreis führen. Dieser besteht aus der Verstärkung von PIT-Effekten durch Alkoholkonsum und verstärktem Alkoholkonsum aufgrund von verstärkten PIT-Effekten. Diskussion: PIT-Effekte bei Alkoholabhängigkeit sind bisher vorwiegend aus Tierstudien bekannt. Das PIT-Paradigma kann uns allerdings auch in der humanen Suchtforschung Aufschluss darüber geben, wie bestimmte Reizmuster Alkoholabhängige zum wiederholten Alkoholkonsum motivieren. Demnach können PIT-Experimente womöglich auch helfen Alkoholrückfälle vorherzusagen.
Background: Disadvantageous decisions with respect to alcohol consumption play a central role in alcohol dependency (AD). This decision making pattern seems to be in part a result of Pavlovian-to-instrumental-transfer effects (PIT effects). The aim of this review is to summarize important findings on PIT within the scope of addiction disorders. Building on this, open questions in the field of human AD are discussed. Methods: This review is not based on a systematic and standardized literature research. Instead the review was based on the literature search conducted in DFG research group 1617 (Learning and Habitization in Alcohol Dependence, LeAD). Selection of research articles was based on expert opinion. Results: PIT effects in AD might possibly lead to a vicious cycle consisting of enhanced PIT effects through alcohol consumption and enhanced alcohol consumption through enhanced PIT effects. Discussion: PIT effects in alcohol addiction are mainly known from animal studies because there are but few human AD PIT studies. In human AD research the PIT paradigm may be able to reveal how particular cues disproportionally motivate AD patients to drink alcohol. PIT experiments thus have potential uses in the prediction of relapse and the measurement of addiction severity.
- doi pdf
- Aversive Pavlovian control of instrumental behaviour in humans
- J. Cogn. Neurosci. (2013) 25(9):1428-41
- Adaptive behaviour involves interactions between systems regulating Pavlovian and instrumental control of actions. Here, we present the first investigation of the neural mechanisms underlying aversive Pavlovian-instrumental transfer using fMRI in humans. Recent evidence indicates that these Pavlovian influences on instrumental actions are action-specific: Instrumental approach is invigorated by appetitive Pavlovian cues, but inhibited by aversive Pavlovian cues. Conversely, instrumental withdrawal is inhibited by appetitive Pavlovian cues, but invigorated by aversive Pavlovian cues. We show that BOLD responses in the amygdala and the nucleus accumbens were associated with behavioural inhibition by aversive Pavlovian cues, irrespective of action context. Furthermore, BOLD responses in the ventromedial prefrontal cortex differed between approach and withdrawal actions. Aversive Pavlovian conditioned stimuli modulated connectivity between the ventromedial prefrontal cortex and the caudate nucleus. These results show that action-specific aversive control of instrumental behaviour involves the modulation of fronto-striatal interactions by Pavlovian conditioned stimuli.
- doi pdf Mapping anhedonia onto reinforcement learning. A behavioural meta-analysis.
- Biology of Mood & Anxiety Disorders (2013) 3:12
-
Background: Depression is characterised partly by blunted reactions to reward. However, tasks probing this deficiency have not distinguished insensitivity to reward from insensitivity to the prediction errors for reward that determine learning and are putatively reported by the phasic activity of dopamine neurons. We attempted to disentangle these factors with respect to anhedonia in the context of stress, Major Depressive Disorder (MDD), Bipolar Disorder (BPD) and a dopaminergic challenge.
Methods: Six behavioural datasets involving 392 experimental sessions were subjected to a model-based, Bayesian meta-analysis. Participants across all six studies performed a probabilistic reward task that used an asymmetric reinforcement schedule to assess reward learning. Healthy controls were tested under baseline conditions, stress or after receiving the dopamine D2 agonist pramipexole. In addition, participants with current or past MDD or BPD were evaluated. Reinforcement learning models isolated the contributions of variation in reward sensitivity and learning rate.
Results: MDD and anhedonia reduced reward sensitivity more than they affected the learning rate, while a low dose of the dopamine D2 agonist pramipexole showed the opposite pattern. Stress led to a pattern consistent with a mixed effect on reward sensitivity and learning rate.
Conclusion: Reward-related learning reflected at least two partially separable contributions. The first related to phasic prediction error signalling, and was preferentially modulated by a low dose of the dopamine agonist pramipexole. The second related directly to reward sensitivity, and was preferentially reduced in MDD and anhedonia. Stress altered both components. Collectively, these findings highlight the contribution of model-based reinforcement learning meta-analysis for dissecting anhedonic behavior.
- doi pdf Frontal theta overrides Pavlovian learning biases
- J. Neurosci. (2013) 33(19):8541-8548
- Pavlovian biases influence learning and decision making by intricately coupling reward seeking with action invigoration and punishment avoidance with action suppression. This bias is not always adaptive; it can oftentimes interfere with instrumental requirements. The prefrontal cortex is thought to help resolve such conflict between motivational systems, but the nature of this control process remains unknown. EEG recordings of mid-frontal theta band power are sensitive to conflict and predictive of adaptive control over behavior, but it is not clear whether this signal would reflect control over conflict between motivational systems. Here we utilized a task that orthogonalized action requirements and outcome valence while recording concurrent EEG in human participants. By applying a computational model of task performance, we derived parameters reflective of the latent influence of Pavlovian bias and how it was modulated by mid- frontal theta power during motivational conflict. Between subjects, individuals who performed better under Pavlovian conflict exhibited higher mid-frontal theta power. Within subjects, trial- to-trial variance in theta power was predictive of ability to overcome the influence of the Pavlovian bias, and this effect was most pronounced in individuals with higher mid-frontal theta to conflict. These findings demonstrate that mid-frontal theta is not only a sensitive index of prefrontal control, but it can also reflect the application of top-down control over instrumental processes.
- pdf Learned helplessness and generalization
- Proceedings of the 35th Annual Conference of the Cognitive Science Society (2013)
- In learned helplessness experiments, subjects first experience a lack of control in one situation, and then show learning deficits when performing or learning another task in another situation. Generalization, thus, is at the core of the learned helplessness phenomenon. Substantial experimental and theoretical effort has been invested into establishing that a state- and task-independent belief about controllability is necessary. However, to what extent generalization is also sufficient to explain the transfer has not been examined. Here, we show qualitatively and quantitatively that Bayesian learning of action-outcome contingencies at three levels of abstraction is sufficient to account for the key features of learned helplessness, including escape deficits and impairment of appetitive learning after inescapable shocks.
- doi pdf Dopamine restores reward prediction errors in older age
- Nature Neuroscience (2013) 16, 648-653
- Senescence affects the ability to utilize information about the likelihood of rewards for optimal decision-making. Using functional magnetic resonance imaging in humans, we found that healthy older adults had an abnormal signature of expected value, resulting in an incomplete reward prediction error (RPE) signal in the nucleus accumbens, a brain region that receives rich input projections from substantia nigra/ventral tegmental area (SN/VTA) dopaminergic neurons. Structural connectivity between SN/VTA and striatum, measured by diffusion tensor imaging, was tightly coupled to inter-individual differences in the expression of this expected reward value signal. The dopamine precursor levodopa (L-DOPA) increased the task-based learning rate and task performance in some older adults to the level of young adults. This drug effect was linked to restoration of a canonical neural RPE. Our results identify a neurochemical signature underlying abnormal reward processing in older adults and indicate that this can be modulated by L-DOPA.
- doi pdf Go and nogo learning in reward and punishment: Interactions between affect and effect
- Neuroimage (2012) 62(1):154-66
- Decision-making invokes two fundamental axes of control: affect or valence, spanning reward and punishment, and effect or action, spanning invigoration and inhibition. We studied the acquisition of instrumental responding in healthy human volunteers in a task in which we orthogonalized action requirements and outcome valence. Subjects were much more successful in learning active choices in rewarded conditions, and passive choices in punished conditions. Using computational reinforcement-learning models, we teased apart contributions from putatively instrumental and Pavlovian components in the generation of the observed asymmetry during learning. Moreover, using model-based fMRI, we showed that BOLD signals in striatum and substantia nigra/ventral tegmental area (SN/VTA) correlated with instrumentally learnt action values, but with opposite signs for go and no-go choices. Finally, we showed that successful instrumental learning depends on engagement of bilateral inferior frontal gyrus. Our behavioral and computational data showed that instrumental learning is contingent on overcoming inherent and plastic Pavlovian biases, while our neuronal data showed this learning is linked to unique patterns of brain activity in regions implicated in action and inhibition respectively.
- doi pdf correction Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees
- PLoS Comp Biol (2012) 8(3): e1002410
- When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal- directed choices in a manner that may be important for theories of decision-making in mood disorders.
- doi pdf Ventral striatal prediction error signalling is associated with dopamine synthesis capacity and fluid intelligence
- Human Brain Mapping (2013) 34(6):1490-9
- Fluid intelligence represents the capacity for flexible problem solving and rapid behavioral adaptation. Rewards drive flexible behavioral adaptation, in part via a teaching signal expressed as reward prediction errors in the ventral striatum, which has been associated with phasic dopamine release in animal studies. We examined a sample of 28 healthy male adults using multimodal imaging and biological parametric mapping with (1) functional magnetic resonance imaging during a reversal learning task and (2) in a subsample of 17 subjects also with positron emission tomography using 6-[(18) F]fluoro-L-DOPA to assess dopamine synthesis capacity. Fluid intelligence was measured using a battery of nine standard neuropsychological tests. Ventral striatal BOLD correlates of reward prediction errors were positively correlated with fluid intelligence and, in the right ventral striatum, also inversely correlated with dopamine synthesis capacity (FDOPA K#inapp). When exploring aspects of fluid intelligence, we observed that prediction error signaling correlates with complex attention and reasoning. These findings indicate that individual differences in the capacity for flexible problem solving relate to ventral striatal activation during reward-related learning, which in turn proved to be inversely associated with ventral striatal dopamine synthesis capacity.
- doi pdf Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain
- J. Neurosci (2011) 31(21):7867-75
- The acquisition of reward and the avoidance of punishment could logically be contingent on either emitting or withholding particular actions. However, the separate pathways in the striatum for go and no-go appear to violate this independence, instead coupling affect and effect. Respect for this interdependence has biased many studies of reward and punishment, so potential action-outcome valence interactions during anticipatory phases remain unexplored. In a functional magnetic resonance imaging study with healthy human volunteers, we manipulated subjects' requirement to emit or withhold an action independent from subsequent receipt of reward or avoidance of punishment. During anticipation, in the striatum and a lateral region within the substantia nigra/ventral tegmental area (SN/VTA), action representations dominated over valence representations. Moreover, we did not observe any representation associated with different state values through accumulation of outcomes, challenging a conventional and dominant association between these areas and state value representations. In contrast, a more medial sector of the SN/VTA responded preferentially to valence, with opposite signs depending on whether action was anticipated to be emitted or withheld. This dominant influence of action requires an enriched notion of opponency between reward and punishment.
- doi pdf Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding
- PLoS Comp Biol (2011) 7(4): e1002028
- Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.
- doi pdf Are computational models useful for psychiatry?
- Neural Networks (2011) 24(6):544-551
- Mathematically rigorous descriptions of key hypotheses and theories are becoming more common in neuroscience and are beginning to be applied to psychiatry. In this article two fictional characters, Dr. Strong and Mr. Micawber, debate the use of such computational models (CMs) in psychiatry. We present four fundamental challenges to the use of CMs in psychiatry: (a) the applicability of mathematical approaches to core concepts in psychiatry such as subjective experiences, conflict and suffering; (b) whether psychiatry is mature enough to allow informative modelling; (c) whether theoretical techniques are powerful enough to approach psychiatric problems; and (d) the issue of communicating clinical concepts to theoreticians and vice versa. We argue that CMs have yet to influence psychiatric practice, but that they help psychiatric research in two fundamental ways: (a) to build better theories integrating psychiatry with neuroscience; and (b) to enforce explicit, global and efficient testing of hypotheses through more powerful analytical methods. CMs allow the complexity of a hypothesis to be rigorously weighed against the complexity of the data. The paper concludes with a discussion of the path ahead. It points to stumbling blocks, like the poor communication between theoretical and medical communities. But it also identifies areas in which the contributions of CMs will likely be pivotal, like an understanding of social influences in psychiatry, and of the co-morbidity structure of psychiatric diseases.
- pdf Computational unhappiness: Modelling depression
- Frontiers in Neurosciences (2009) 3(2):263
- doi pdf Serotonin in affective control
- Annu Rev Neurosci (2009) :: 32: 95-126
- Serotonin is a neuromodulator that is extensively entangled in fundamental aspects of brain function and behavior. We present a computational view of its involvement in the control of appetitively and aversively motivated actions. We first describe a range of its effects in invertebrates, endowing specific structurally fixed networks with plasticity at multiple spatial and temporal scales. We then consider its rather widespread distribution in the mammalian brain. We argue that this is associated with a more unified representational and functional role in aversive processing that is amenable to computational analyses with the kinds of reinforcement learning techniques that have helped elucidate dopamine's role in appetitive behavior. Finally, we suggest that it is only a partial reflection of dopamine because of essential asymmetries between the natural statistics of rewards and punishments.
- doi pdf A Bayesian formulation of behavioral control
- Cognition (2009) Special Issue
- Helplessness, a belief that the world is not subject to behavioral control, has long been central to our understanding of depression, and has influenced cognitive theories, animal models and behavioral treatments. However, despite its importance, there is no fully accepted definition of helplessness or behavioral control in psychology or psychiatry, and the formal treatments in engineering appear to capture only limited aspects of the intuitive concepts. Here, we formalize controllability in terms of characteristics of prior distributions over affectively charged environments. We explore the relevance of this notion of control to reinforcement learning methods of optimising behavior in such environments and consider how apparently maladaptive beliefs can result from normative inference processes. These results are discussed with reference to depression and animal models thereof.
- pdf Psychiatry: insights into depression through normative decision-making models
- NIPS 2008
- Decision making lies at the very heart of many psychiatric diseases. It is also a central theoretical concern in a wide variety of fields and has undergone detailed, in-depth, analyses. We take as an example Major Depressive Disorder (MDD), applying insights from a Bayesian reinforcement learning framework. We focus on anhedonia and helplessness. Helplessness-a core element in the conceptual- izations of MDD that has lead to major advances in its treatment, pharmacolog- ical and neurobiological understanding-is formalized as a simple prior over the outcome entropy of actions in uncertain environments. Anhedonia, which is an equally fundamental aspect of the disease, is related to the effective reward size. These formulations allow for the design of specific tasks to measure anhedonia and helplessness behaviorally. We show that these behavioral measures capture explicit, questionnaire-based cognitions. We also provide evidence that these tasks may allow classification of subjects into healthy and MDD groups based purely on a behavioural measure and avoiding any verbal reports.
- doi pdf suppl Serotonin, behavioral inhibition and negative mood
- PLoS Computational Biology (2008) 4(2): e4
- Pavlovian predictions of future aversive outcomes lead to behavioral inhibition, suppression, and withdrawal. There is considerable evidence for the involvement of serotonin in both the learning of these predictions and the inhibitory consequences that ensue, although less for a causal relationship between the two. In the context of a highly simplified model of chains of affectively charged thoughts, we interpret the combined effects of serotonin in terms of pruning a tree of possible decisions, (i.e., eliminating those choices that have low or negative expected outcomes). We show how a drop in behavioral inhibition, putatively resulting from an experimentally or psychiatrically influenced drop in serotonin, could result in unexpectedly large negative prediction errors and a significant aversive shift in reinforcement statistics. We suggest an interpretation of this finding that helps dissolve the apparent contradiction between the fact that inhibition of serotonin reuptake is the first-line treatment of depression, although serotonin itself is most strongly linked with aversive rather than appetitive outcomes and predictions.
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2009
2008
PhD thesis
- pdf Reinforcers and control: Towards a computational ætiology of depression
- Gatsby Computational Neuroscience Unit, UCL (2007)
- Depression, like many psychiatric disorders, is a disorder of affect. Over the past decades, a large number of affective issues in depression have been characterised, both in human experiments and animal models of the disorder. Over the same period, experimental neuroscience, helped by computational theories such as reinforcement learning, has provided detailed descriptions of the psychology and neurobiology of affective decision making. Here, we attempt to harvest the advances in the understanding of the brain's normal dealings with rewards and punishments to dissect out and define more clearly the components that make up depression. We start by exploring changes to primary reinforcer sensitivity in the learned helplessness animal models of depression. Then, a detailed formalisation of control in a goal-directed decision making framework is presented and related to animal and human data. Finally, we show how serotonin's joint involvement in reporting negative values and inhibiting actions may explain some aspects of its involvement in depression. Throughout, aspects of depression are seen as emerging from normal affective function and reinforcement learning, and we thus conclude that computational descriptions of normal affective function provide one possible avenue by which to define an ætiology of depression.
Abstracts / Unrefereed
- Habitual and goal-directed learning in alcohol dependence
- Dopamine (2013)
- Representations for planning
- Einstein Symposium Berlin (2013)
- poster Controllability and resource-rational planning
- Cosyne (2013)
- Optimism can function as a normative prior belief on the probability of rewards
- Mathematical Biosciences Institute, Workshop on Cognitive Neuroscience, Columbus Ohio, USA, December 2012
- poster The neural bases of reversal learning deficits in unmedicated schizophrenia patients
- Einstein meeting Berlin (2011)
- Efficient decision-making is dependent on the efficient pruning of decision trees.
- Summer Meeting of the British Association for Psychopharmacology (2011)
- poster Psychopathy: a dysfunction in Pavlovian-to-instrumental transfer
- Biological Psychiatry (2011)
- doi Approaching avoidance: instrumental and Pavlovian asymmetries in the procesing of rewards and punishments
- Cosyne 2010
- doi Bonsai trees: how the Pavlovian system sculpts sequential decisions
- Cosyne 2010
- Reward and helplessness in depression
- Cosyne 2008
- Normative learning: a route to depression?
- CNS 2007
- Depression as normal learning
- Nature medicine-roche conference on translational psychiatry
- poster Optimal helplessness: a normative framework for depression
- CNS 2006
- poster Malignant evaluation: reinforcement learning, neuromodulation and depression
- Cosyne 2006
2013
2012
2011
2010
2008
2007
2006
Population coding
Peer reviewed
- doi pdf Encoding and decoding spikes for dynamic stimuli
- Neural Computation (2008) 20(9):2325-60
Naturally occurring sensory stimuli are dynamic. In this letter, we consider how spiking neural populations might transmit information about continuous dynamic stimulus variables. The combination of simple encoders and temporal stimulus correlations leads to a code in which information is not readily available to downstream neurons. Here, we explore a complex encoder that is paired with a simple decoder that allows representation and manipulation of the dynamic information in neural systems. The encoder we present takes the form of a biologically plausible recurrent spiking neural network where the output population recodes its inputs to produce spikes that are independently decodeable. We show that this network can be learned in a supervised manner by a simple local learning rule.
- doi pdf code Fast population coding
- Neural Computation (2007) 19(2):460-97
- Uncertainty coming from the noise in its neurons and the ill-posed nature of many tasks plagues neural computations. Maybe surprisingly, many studies show that the brain manipulates these forms of uncertainty in a probabilistically consistent and normative manner, and there is now a rich theoretical literature on the capabilities of populations of neurons to implement computations in the face of uncertainty. However, one major facet of uncertainty has received comparatively little attention: time. In a dynamic, rapidly changing world, data are only temporarily relevant. Here, we analyze the computational consequences of encoding stimulus trajectories in populations of neurons. For the most obvious, simple, instantaneous encoder, the correlations induced by natural, smooth stimuli engender a decoder that requires access to information that is nonlocal both in time and across neurons. This formally amounts to a ruinous representation. We show that there is an alternative encoder that is computationally and representationally powerful in which each spike con- tributes independent information; it is independently decodable, in other words. We suggest this as an appropriate foundation for understanding time-varying population codes. Furthermore, we show how adaptation to temporal stimulus statistics emerges directly from the demands of simple decoding.
- pdf Probabilistic computation in spiking populations
- NIPS 2004
- As animals interact with their environments, they must constantly update estimates about their states. Bayesian models combine prior probabilities, a dynamical model and sensory evidence to update estimates optimally. These models are consistent with the results of many diverse psychophysical studies. However, little is known about the neural representation and manipulation of such Bayesian information, particularly in populations of spiking neurons. We consider this issue, suggesting a model based on standard neural architecture and activations. We illustrate the approach on a simple random walk example, and apply it to a sensorimotor integration task that provides a particularly compelling example of dynamic probabilistic computation.
- abstract talk Fast population coding
- CNS 2006
- Dynamic population codes for sensorimotor processing
- Cosyne 2006
- poster Population coding in a fast-changing world
- Cosyne 2005
- Representational pursuit: population codes for dynamic environments
- Cosyne 2005
- poster A simple population code in a fast-changing world
- Soc. Neurosci. Abstr. 2005
- doi pdf code movie Smoothing of, and parameter estimation from, noisy biophysical recordings
- PLoS Computational Biology (2009) 5(5): e1000379
- Biophysically detailed models of single cells are difficult to fit to real data. Recent advances in imaging techniques allow simultaneous access to various intracellular variables, and these data can be used to significantly facilitate the modelling task. These data, however, are noisy, and current approaches to building biophysically detailed models are not designed to deal with this. We extend previous techniques to take the noisy nature of the measurements into account. Sequential Monte Carlo (``particle filtering'') methods, in combination with a detailed biophysical description of a cell, are used for principled, model-based smoothing of noisy recording data. We also provide an alternative formulation of smoothing where the neural nonlinearities are estimated in a non-parametric manner. Biophysically important parameters of detailed models (such as channel densities, intercompartmental conductances, input resistances, and observation noise) are inferred automatically from noisy data via expectation-maximisation. Overall, we find that model-based smoothing is a powerful, robust technique for smoothing of noisy biophysical data and for inference of biophysical parameters in the face of recording noise.
- doi pdf code Efficient estimation of detailed single-neuron models
- J. Neurophysiol. (2006) 96: 872-890
- Biophysically accurate multicompart-mental models of individual neurons have significantly advanced our understanding of the input output function of single cells. These models depend on a large number of parameters that are difficult to estimate. In practice, they are often hand-tuned to match measured physiological behaviors, thus raising questions of identifiability and interpretability. We propose a statistical approach to the automatic estimation of various biologically relevant parameters, including 1) the distribution of channel densities, 2) the spatiotemporal pattern of synaptic input, and 3) axial resistances across extended dendrites. Recent experimental advances, notably in voltage-sensitive imaging, motivate us to assume access to: i) the spatiotemporal voltage signal in the dendrite and ii) an approximate description of the channel kinetics of interest. We show here that, given i and ii, parameters 13 can be inferred simultaneously by nonnegative linear regression; that this optimization problem possesses a unique solution and is guaranteed to converge despite the large number of parameters and their complex nonlinear interaction; and that standard optimization algorithms efficiently reach this optimum with modest computational and data requirements. We demonstrate that the method leads to accurate estimations on a wide variety of challenging model data sets that include up to about 104 parameters (roughly two orders of magnitude more than previously feasible) and describe how the method gives insights into the functional interaction of groups of channels.
- pdf Large-scale biophysical parameter estimation in single neurons via constrained linear regression
- NIPS 2005
- Our understanding of the input-output function of single cells has been substantially advanced by biophysically accurate multi-compartmental models. The large number of parameters needing hand tuning in these models has, however, somewhat hampered their applicability and interpretability. Here we propose a simple and well-founded method for auto- matic estimation of many of these key parameters: 1) the spatial distribution of channel densities on the cell's membrane; 2) the spatiotemporal pattern of synaptic input; 3) the channels' reversal potentials; 4) the intercompartmental conductances; and 5) the noise level in each compart- ment. We assume experimental access to: a) the spatiotemporal voltage signal in the dendrite (or some contiguous subpart thereof, e.g. via voltage sensitive imaging techniques), b) an approximate kinetic description of the channels and synapses present in each compartment, and c) the morphology of the part of the neuron under investigation. The key observation is that, given data a)-c), all of the parameters 1)-4) may be si- multaneously inferred by a version of constrained linear regression; this regression, in turn, is efficiently solved using standard algorithms, without any "local minima" problems despite the large number of parameters and complex dynamics. The noise level 5) may also be estimated by standard techniques. We demonstrate the method's accuracy on several model datasets, and describe techniques for quantifying the uncertainty in our estimates.
- Model-based smoothing of noisy, intermittent biophysical samples
- Cosyne 2007
- abstract poster talk Model-based optimal interpolation and filtering for noisy, intermittent biophysical recordings
- Cns 2006
- poster Estimating non-homogeneous channel densities and synaptic activity from spatiotemporal dendritic voltage recordings
- Cosyne 2005
- poster Estimating non-homogeneous channel densities and synaptic activity from spatiotemporal dendritic voltage recordings
- SfN 2005
- doi pdf Screening patients with sensorineural hearing loss for a vestibular Schwannoma using a Bayesian classifier
- J. Clin. Otolaryngology (2007) 32(4):248-54
-
Objectives: Selecting patients with asymmetrical sensorineural hearing loss for further investigation continues to pose clinical and medicolegal challenges, given the disparity between the number of symptomatic patients, and the low incidence of vestibular schwannoma as the underlying cause. We developed and validated a diagnostic model using a generalisation of neural networks, for detecting vestibular schwannomas from clinical and audiological data, and compared its performance with six previously published clinical and audiological decision- support screening protocols.
Design: Probabilistic complex data classification using a neural network generalization.
Settings: Tertiary referral lateral skull base and a computational neuroscience unit.
Participants: Clinical and audiometric details of 129 patients with, and as many age and sex-matched patients without vestibular schwannomas, as determined with magnetic resonance imaging.
Main outcome measures: The ability to diagnose a patient as having or not having vestibular schwannoma.
Results: A Gaussian Process Ordinal Regression Classifier was trained and cross-validated to classify cases as `with' or `without' vestibular schwannoma, and its diagnostic performance was assessed using receiver operator characteristic plots. It proved possible to pre-select sensitivity and specificity, with an area under the curve of 0.8025. At 95% sensitivity, the trained system had a specificity of 56%, 30% better than audiological protocols with closest sensitivities. The sensitivities of previously-published audiological protocols ranged between 82-97%, and their specificities ranged between 15-61%.
Discussion: The Gaussian Process Ordinal Regression Classifier increased the flexibility and specificity of the screening process for vestibular schwannoma when applied to a sample of matched patients with and without this condition. If applied prospectively, it could reduce the number of `normal' magnetic resonance (MR) scans by as much as 30% without reducing detection sensitivity. Performance can be further improved through incorporating additional data domains. Current findings need to be reproduced using a larger dataset.
- Multiple endocrinopathies presenting simultaneously in the post-partum phase
- SfE National Clinical Cases 2012
- Screening of patients with sensorineural hearing loss for a vestibular Schwannoma
- R. Soc. Med. Section of Otology Abs. 2006