Knowing how much you don't know; a neural organization of uncertainty estimates

“Uncertainty and expectation are the joys of life. Security is an insipid thing.” William Congreve, 1670–1729, playwright. Nothing is perfect and everything we sense, think and do is imbued with imprecision. Imprecision impedes us reaching our goals, but if we can estimate this impreci- sion we can use it to improve the success of our actions: we open our arms wider to pick up a faltering toddler than one who is sitting still, we look more carefully for other cars when driving in fog, and when secur- ing our pensions we avoid shares that have a history of excessively large gains and losses. Another term for imprecision is uncertainty. We can be uncertain about many different things and, as a consequence, uncertainty has been studied from many perspectives in the neurosciences, often utilizing diver- gent theoretical assumptions and empirical approaches. This Review unravels various strands of thinking relat- ing to the concept of uncertainty and integrates them within an organized terminological and theoretical framework, in which our aim is to highlight possible common mechanisms of neural encoding and reveal areas that are in need of greater theoretical or empirical refinement. We discuss studies of uncertainty in a framework that proposes four different processing levels that per- tain to decision making and action planning (FIG. 1): sensory processing, state evaluation, rule identification and outcome prediction. This framework reflects dif- ferent types of variables about which we can be uncer- tain (BOX 1) rather than different sources of uncertainty (for example, environmental or internal). This provides a simplifying heuristic, although we acknowledge that these four levels do not necessarily form exclusive or coherent processing levels. To add to the complexity, within each of these levels uncertainty is often concep- tualized in several distinct ways. For each of these levels, we address three questions. The first is whether uncertainty guides behaviour. In many areas of behaviour and decision making, nor- mative models prescribe the utilization of uncertainty estimates1–6. But behaviour of biological agents is not always optimal, and we can construe several situations in which optimal behaviour might mandate that uncer- tainty be ignored7. Thus, although there is good reason to expect that uncertainty influences behaviour, this needs to be empirically demonstrated in detail. Second, if uncertainty guides behaviour, is there a distinct neural encoding of uncertainty? Again, the answer is not obvious from theoretical considerations alone. For example, behavioural phenomena that seem- ingly relate to uncertainty (for example, risk prefer- ences) can be explained by divergent mechanisms8,9. Any investigation into neural representations of uncer- tainty has to take into account that noise is inherent in complex systems and can be greatly amplified by suboptimal inference10. Noise is not necessarily used 1Wellcome Trust Centre for Neuroimaging, University College London, 12 Queen Square, London WC1N 3BG, UK. 2Berlin School of Mind and Brain, Humboldt-Universität zu Berlin, Luisenstraße 56, 10099 Berlin, Germany. Correspondence to D.R.B.  e-mail: d.bach@ucl.ac.uk doi:10.1038/nrn3289 Published online 11 July 2012 Knowing how much you don’t know: a neural organization of uncertainty estimates Dominik R. Bach1,2, Raymond J. Dolan1,2 Abstract | How we estimate uncertainty is important in decision neuroscience and has wide-ranging implications in basic and clinical neuroscience, from computational models of optimality to ideas on psychopathological disorders including anxiety, depression and schizophrenia. Empirical research in neuroscience, which has been based on divergent theoretical assumptions, has focused on the fundamental question of how uncertainty is encoded in the brain and how it influences behaviour. Here, we integrate several theoretical concepts about uncertainty into a decision-making framework. We conclude that the currently available evidence indicates that distinct neural encoding (including summary statistic-type representations) of uncertainty occurs in distinct neural systems. R E V I E W S NATURE REVIEWS | NEUROSCIENCE ADVANCE ONLINE PUBLICATION | 1 Nature Reviews Neuroscience | AOP, published online 11 July 2012; doi:10.1038/3289 © 2012 Macmillan Publishers Limited. All rights reserved Nature Reviews | Neuroscience Sensory information Perception Transition rules Future states Sensory uncertainty State uncertainty Rule uncertainty Inference of current state Outcome uncertainty Environmental uncertainty Internal noise Current state Probability of transition to state B Probability of transition to state A Possible future state A Possible future state B Rule identification If I cross when it is already red — what is the chance of an accident? Outcome evaluation 1% chance of an accident — will I make it? State evaluation How far is it to reach the crossing? A CAR Sensory processing Is this a yellow light or just a reflection of the sun? to guide behaviour and hence a key aim is to identify neural representations of uncertainty over and above incidental noise that influence further neural com- putations and, indeed, behaviour. Although there are several theoretical proposals on how uncertainty might be encoded at a neuronal level2,11, empirical investiga- tions often assume that uncertainty is encoded in the firing rate of specific neuronal populations. We call this a summary-statistic encoding of uncertainty, as it sum- marizes uncertainty into a single number. The focus on summary-statistic encoding is not driven by theoretical considerations alone (indeed, it is perhaps not the most plausible option from a theoretical viewpoint), it also reflects an assumption that such an encoding is prob- ably not due to incidental noise. The third question we address concerns the topo- graphical distribution of uncertainty representations in the brain. For example, is there a distinct uncertainty representation about line orientation in the visual sys- tem and a distinct uncertainty representation about Summary statistic A concise way of describing a set of observations without having to refer to each individual observation. Hence, the set of observations can be described with just a few values. For example, one for the location (for example, mean) and another for the dispersion, that is, uncertainty, (for example, variance). Figure 1 | Processing levels in decision making and action planning. A real-life situation is used here to illustrate four processing levels in an action episode: a driver sees a yellow light while approaching a crossing. The first level concerns sensory processing: incoming information needs to be quantified or categorized. In this example, it needs to be determined whether the yellow light comes from the traffic light or is just a reflection of the sun. Sensory uncertainty refers to the fact that the driver knows only his noisy sensory input and not the actual status of the traffic light. The second level concerns state evaluation. In this example, the state of the environment (that is, the distance from the crossing) determines the chance of making it over the crossing before the light turns red. State uncertainty refers to the fact that inferring the distance to the crossing from a moving car is imprecise. The third level concerns rule identification. In this case, the driver will not make it over the crossing in time and plans to accelerate anyway, and the chances of having an accident constitute an example of a rule. Rule uncertainty in this example could arise from the driver having little experience of such a situation and receiving conflicting advice from two passengers (‘don’t worry’ versus ‘you’ll kill all of us’). The fourth level concerns outcome prediction. Even if the driver knows the precise odds of having an accident, it is uncertain whether this possible accident will happen or not. R E V I E W S 2 | ADVANCE ONLINE PUBLICATION www.nature.com/reviews/neuro © 2012 Macmillan Publishers Limited. All rights reserved Pr ob ab ili ty Pr ob ab ili ty Pr ob ab ili ty Pr ob ab ili ty Outcome (£) 0 0.0 0.2 0.4 0.6 0.8 1.0 10 20 30 40 50 60 Outcome (£) 0 0.0 0.2 0.4 0.6 0.8 1.0 10 20 30 40 50 60 Outcome (£) 0 0.0 0.2 0.4 0.6 0.8 1.0 10 20 30 40 50 60 Outcome (£) 0 0.0 0.2 0.4 0.6 0.8 1.0a b c d 10 20 30 40 50 60 Nature Reviews | Neuroscience decision outcomes in a decision-making network? Or is there a unified representation for different forms of uncertainty — a de facto canonical uncertainty repre- sentation? It has been suggested that uncertainty on a number of distinct decision-making variables is pro- cessed by a single brain area and is employed to elicit a common behavioural response12. Indeed, within com- putational models of decision making one can envis- age collapsing different sources of uncertainty into a single quantity, and this might in principle occur in an ‘uncertainty area’. However, theories that posit a hierar- chically organized brain1,2,6 usually assume that uncer- tainty about the value of a particular variable is bound to a representation of the value of that variable, thereby arguing against such a canonical uncertainty represen- tation. A less strong version of a canonical uncertainty proposition is that common principles underlie the neural encoding of different forms of uncertainty, albeit organized in different locations within a neural hierarchy11. Box 1 | Measures of uncertainty Uncertainty about a variable means that its true state or, for a quantitative variable, its magnitude is unknown. In other words, the variable in question can express one of several possible values. For each of these possible values, we can assign a probability to our degree of belief that it is the true value (see the figure). Loosely speaking, the most likely value of this variable forms our expectation of the value (for example, a value of 0 in the figure, panel b), and the dispersion in the distribution of possible values is one way of expressing our uncertainty on this expectation. Probabilistic computations can be performed using the complete distribution of possibilities, without an explicit measure of uncertainty. However, given a situation in which there are many possible values, a measure of uncertainty can greatly simplify such computations. How can we quantify, or measure, uncertainty? In neuroscience, two principal approaches are used. The first approach is used when the variable in question takes several discrete states that are treated as nominal. That is, there is no scalar quantity associated with the states (for example, the colour of an object that can be red, blue or green) or when the scalar quantity is disregarded. Here, one can quantify uncertainty as the Shannon entropy113 of a discrete probability distribution (that is, probability mass function), which quantifies how much information is gained if the true state of the variable is revealed (that is, the more uncertain the true value of the variable is, the more information is gained by knowing it). The other principal approach, which is mainly used in economic theory, is to take a scalar quantity into account. For example, when we want to express that our uncertainty about winning either £40 or £60 from a toss of a coin is much lower than winning £0 or £100, even though the Shannon entropies of these lotteries are the same. In these cases, we can define uncertainty as dispersion of the probability distribution; for example, as variance or coefficient of variation114. This approach extends to continuous random variables that are governed by probability density functions. A third, and more principled, approach is to measure uncertainty in continuous variables as their differential or relative entropy (Kullback–Leibler divergence) with respect to a reference distribution. Similar to dispersion measures, these entropy measures naturally take the scalar value of the variable into account. However, they have rarely been used in experimental research on the neural representation of uncertainty. The figure illustrates several examples of probability functions and how to measure their associated uncertainty. The first example (a) shows a flip of a fair coin with probabilities of 0.5 and a win of £5 or £15 from heads or tails, respectively (Shannon entropy: 1 bit; variance 25 £2). The second example (b) shows a toss of a fair dice with a win of £60 from a six and nothing for the numbers one to five (Shannon entropy: 0.65 bit; variance 500 £2). Shannon entropy, which does not take into account outcome magnitude, is larger in panel a, but the variance of possible outcomes is larger in panel b. The third example (c) shows a toss of a fair dice, winning the amount of money shown by the number on the dice in £, plus £6.50 (Shannon entropy: 2.59 bit; variance 2.9 £2). This example has high Shannon entropy because the actual outcome is more unpredictable than in panels a and b, but there is little variance because possible outcomes are more similar than in the other examples. The final example (d) shows a probability density function for an unspecified example of continuous outcome possibilities. Here, uncertainty can, for example, be quantified as dispersion of the distribution. A measure of uncertainty provides a basis for probing its behavioural and neural correlates. This is different from using a categorical approach, in which uncertainty is contrasted with certainty, to investigate behavioural and neural correlates of uncertainty. Such a categorical approach is related to theories which posit that uncertainty and certainty induce particular states of mind115–117. Hence, contrasting uncertainty with (near) certainty might highlight neural activity changes that are unrelated to uncertainty coding per se118. Attempts to disambiguate this confounding feature from a quantifiable encoding of uncertainty encoding are rare119. This Review therefore focuses on studies that continuously varied uncertainty on some variable. R E V I E W S NATURE REVIEWS | NEUROSCIENCE ADVANCE ONLINE PUBLICATION | 3 © 2012 Macmillan Publishers Limited. All rights reserved Uncertainty about sensory information Imagine trying to negotiate an unfamiliar street in dark and misty weather. Incoming visual information needs to be quantitatively measured (for example, to gauge dis- tances) and categorized (for example, to decipher letters on a street sign). Such sensory information is imbued with imprecision, and access to this imprecision might be beneficial (for example, equipped with this knowl- edge one might drive more carefully than usual). Studies have investigated sensory uncertainty mainly in three dominant, partially overlapping, domains: multisensory integration, sensorimotor control and unimodal sen- sory decision-making. To quantify sensory uncertainty, one approach is to infer the overall uncertainty due to stimulus uncertainty and internal noise from overt behav- iour. Another is to use a measure of stimulus uncertainty alone; this is often implemented as a semi-quantitative measure termed task difficulty. Sensory uncertainty guides behaviour. Multisensory integration experiments build on the idea that when we combine conflicting information (or cues), more uncertain information deserves less weight13,14. Indeed, humans and monkeys weight information sources according to their individual overall uncertainty in a near-optimal manner (that is, close to minimizing the error of the combined estimate). This has been demon- strated for a wide range of cue combinations from vari- ous modalities and under different response conditions, both when cue uncertainty is stationary15–21 or experi- mentally varied22–29. Previous experience is an additional source of information (often termed prior information) that can be weighted and integrated with current sen- sory cues according to Bayes’ theorem. Indeed, this is what humans appear to do, both under stationary condi- tions30, and when sensory uncertainty31,32 or prior uncer- tainty33,34 are experimentally varied. Note that in these and other experiments35–37, information integration is sometimes suboptimal, but, even in these cases, most of the time uncertainty does influence behaviour. Behavioural sensitivity to sensory uncertainty is also evident in studies of sensorimotor control13. Optimal motor planning takes into account uncertainty in sen- sory information. For example, grip aperture should be wider when we are more uncertain about the position of an object to grip. This has been shown to occur for objects imbued with visual uncertainty38 and for objects for which its position is uncertain owing to an impre- cise coordinate transform between body and eye refer- ence39. Furthermore, when humans are asked to point to a target, they are quicker to adjust their movement upon target change when the initial target has a higher position uncertainty, and they are also quicker when the final target has a lower position uncertainty40. In addi- tion, a motor task such as catching a ball involves setting the optimal time point for starting the movement: if we do not observe the ball for long enough, sensory infor- mation is more uncertain; but if we start moving too late, the movement will be imbued with greater motor uncertainty (see below). Across different tasks, humans integrate visual and motor uncertainty in a near-optimal manner, therefore showing that both visual and motor uncertainty guide behaviour under stationary condi- tions41,42. In another paradigm, human subjects com- bine estimates of their hand position based on previous proprioceptive and visual feedback with current visual feedback to adapt their hand movements. When cur- rent visual feedback is more uncertain, adaptation takes longer43, and if previous feedback is more uncertain, adaptation is faster43, as predicted by optimal integration. A third strand of evidence for an effect of sensory uncertainty on behaviour comes from experiments on unimodal sensory decisions. Take an experiment in which monkeys perform a random-dot task (BOX 2) and saccade to indicate the direction of net dot motion, and they receive a reward for a correct response44. For half the trials, the animals can opt out of the ‘bet’ and instead saccade to a third target that yields a smaller, fixed reward. Opting out is advantageous when the fixed reward exceeds the expected reward from the random-dot task. This enables one to measure a mon- key’s estimate of expected reward — an estimate cor- responding to the monkey’s confidence that a decision would be correct. Crucially, monkeys learn to opt out more often when there is more uncertainty about net dot motion, indicating that the level of uncertainty guides sensory decision-making. Note that the stimu- lus configuration depends on the number of consist- ently moving dots, and the monkey brain might simply associate this configuration with a level of reward, rather than encode the uncertainty. To shed light on this point, recording of neural activity can be utilized (see below). Similar to findings from this study44, when humans make sensory decisions regarding two noisy stimuli — one with high and one with low uncertainty — the difference in uncertainty between these two stimuli correlates with the probability of choosing the less noisy stimulus45. In summary, there is compelling evidence from a range of experimental contexts and tasks to suggest that sensory uncertainty, which is inherent in the environ- ment or due to internal noise, influences behaviour, thus often lea