Expected free energy is a quantity minimized by active inference agents when selecting actions. It scores each possible policy (sequence of actions) by balancing two imperatives: attaining preferred outcomes (exploitation) and resolving uncertainty about the environment (exploration). Under specific parameter settings, expected free energy reduces to either Bayesian decision theory (maximizing expected utility) or optimal Bayesian design (maximizing information gain). The T-maze foraging simulation demonstrates how an active inference agent that minimizes expected free energy naturally exhibits a mix of information-seeking and goal-directed behavior, while agents that maximize only expected utility or only information gain fail to achieve high performance.
Key Points
Expected free energy combines Bayesian decision theory and optimal Bayesian design under a single functional.
Removing prior outcome preferences reduces active inference to optimal Bayesian design (pure information gain maximization).
Removing ambiguity and posterior predictive entropy reduces active inference to Bayesian decision theory (pure expected utility maximization).
Expected free energy can be decomposed as risk plus ambiguity, or equivalently as intrinsic value plus extrinsic value.
Intrinsic value corresponds to expected information gain about latent states (and optionally model parameters), driving curiosity and exploration.
Extrinsic value corresponds to log prior preferences over outcomes, analogous to reward or cost in reinforcement learning.
In a T-maze simulation, the active inference agent consistently selects a policy that first resolves context uncertainty then exploits the known reward location, outperforming agents that optimise either expected utility or expected information gain alone.
Concepts
Active inference: A framework for understanding behaviour in self-organising systems (e.g., humans) that maintain a steady state by minimising free energy. It comprises two functionals: variational free energy (fitting the generative model to observations) and expected free energy (selecting actions that lead to preferred states).
Variational free energy: The evidence lower bound used to fit a generative model to observed outcomes, minimised through inference (over states and policies) and learning (over parameters).
Generative model: A probabilistic model of the environment, typically formalised as a partially observable Markov decision process (POMDP) with states (s), observations (o), policies (\pi), and discrete time (\tau). Its joint distribution is (P(o_{1:T}, s_{1:T}, \pi) = P(\pi) P(s_1) \prod_t P(o_t|s_t) \prod_{t>1} P(s_t|s_{t-1}, \pi)).
Risk: The Kullback–Leibler (KL) divergence between predicted beliefs about future latent states and prior (preferred) beliefs. Minimising risk aligns behaviour with prior preferences, a form of KL control.
Ambiguity: The expected conditional entropy of outcomes given beliefs about latent states. Minimising ambiguity selects policies that yield unambiguous, salient observations.
Intrinsic value (epistemic value): Expected information gain about latent states (and optionally model parameters). Promotes curiosity, artificial curiosity, and exploration.
Extrinsic value (pragmatic value): Expected log prior preference over outcomes, analogous to a reward or utility function. Drives exploitative behaviour to attain preferred outcomes.
Details
Expected free energy is defined for a policy (\pi) at a future time (\tau \ge t) as
where the predictive distribution is (Q(o_\tau, s_\tau | \pi) = P(o_\tau | s_\tau) Q(s_\tau | \pi)). The expected free energy over the entire horizon is summed over future time steps, and the policy posterior is taken as
with (\sigma) the softmax function. Beliefs about states and policies are optimised by gradient descent on variational free energy, and model parameters are learned similarly.
Decompositions. Expected free energy can be rewritten in two useful forms:
Risk plus ambiguity:
(G(\pi, \tau) = D_{\mathrm{KL}}[Q(s_\tau | \pi) \parallel P(s_\tau)] + \mathbb{E}{Q(s\tau | \pi)}[H(P(o_\tau | s_\tau))]),
where the first term is risk (deviation from prior preferences over states) and the second is ambiguity (expected uncertainty in observations given states).
Information gain about latent states can be extended to include model parameters when learning is enabled, corresponding to novelty (Schwartenbeck et al., 2019).
Limiting cases.
Optimal Bayesian design (Lindley, 1956; Chaloner & Verdinelli, 1995): Removing prior preferences about outcomes ((\log P(o_\tau) = \text{constant})) leaves only the intrinsic value term (expected information gain). Active inference then reduces to maximising mutual information between latent states and observations, which underwrites salience in visual search (Itti & Baldi, 2009) and deployment of saccadic eye movements.
Bayesian decision theory (Savage, 1972; Harsanyi, 1978): Removing ambiguity and posterior predictive entropy (or equivalently, removing the information gain term) leaves only the extrinsic value term, i.e., maximisation of expected utility. This is equivalent to standard reward maximisation in reinforcement learning (Sutton & Barto, 1998).
T-maze simulation. A foraging maze environment (described in Friston et al., 2017; Friston et al., 2015) consists of a T-shaped maze with a mouse starting at the centre. There are four possible actions (move to centre, lower arm, upper-right arm, upper-left arm) and eight latent states (four locations × two contexts: cheese in right or left arm). At the centre the agent receives an ambiguous cue; at the lower arm it receives a definitive cue revealing the cheese location. The upper arms contain cheese with probability 98% according to the true context. The agent cannot learn the context across trials – each trial is independent. Prior preferences assign a log preference of +6 to the outcome of correctly located cheese and –6 to the incorrect location (zero otherwise). Performance scores are recorded externally: +5 for visiting the cue then the correct cheese arm, +10 for going directly to the correct cheese, and –10 for failing to find cheese.
Three simulated agents (active inference, optimal Bayesian design, Bayesian decision theory) were run for 50 trials with context shifting between white (trials 1–9, 13–29, 31–50) and black (trials 10–12, 30). All other model parameters were identical.
The active inference agent consistently selected policy 8 (go to cue then left arm) across all trials, appropriately resolving context uncertainty before exploiting the reward. It accumulated the highest score.
The optimal Bayesian design agent selected only information-seeking policies (go to cue and stay there, or go to cue then back to centre). After resolving uncertainty, it “got bored” and did not go to the cheese, resulting in low cumulative score.
The Bayesian decision theory agent exhibited exploitative but uninformed behaviour: it sometimes remained at centre or moved to the cue by chance, but without resolving context it often went to the wrong arm (right arm on the first trial) and achieved a moderate score.
These results illustrate that only the expected free energy functional induces the Bayes-optimal balance of exploration and exploitation. The contrast with reinforcement learning is noted: standard RL methods use (\varepsilon)-greedy or intrinsic motivation rewards (e.g., VIME, Pathak et al.) to encourage exploration, but these are approximations to the principled epistemic value term in active inference. Bayesian RL schemes (Thompson sampling, UCB) are closer but often ignore uncertainty about latent states in the expected free energy sense (Houthooft et al., 2016).
The implementation used variational message passing (SPM function spm_MDP_VB_X.m) with gradient descent on free energy functionals. The generative model had a Kronecker product structure over location and context factors. Ten policies were considered (see Figure 3 description). The active inference agent’s behaviour demonstrates that sentient systems can self-organise to non-equilibrium steady states by minimising expected free energy, intrinsically balancing epistemic and pragmatic drives. Future work should examine how structure and parameters of the generative model (e.g., precision of prior preferences, hierarchical levels) can account for individual differences in human exploration–exploitation behaviour (Laureiro-Martínez et al., 2010; Gottlieb et al., 2013).