What Is a Policy?

A policy is the smallest description of agent behavior that supports learning. The naive version is a table: state in, action out.

TABLE = {
    "s0": "left",
    "s1": "right",
    "s2": "stay",
}
 
def act(state):
    return TABLE[state]
 
state = "s0"
action = act(state)

The program has no randomness, no learning rule, and no uncertainty. The same state always produces the same action. The complete behavior is readable in three lines of dictionary entries. That simplicity is the reason the table appears first: every later complication removes a specific limitation of the table.

The underlying structure#

A policy is a mapping from states to distributions over actions. Formally, a policy is a function $\pi(a \mid s)$ that assigns a probability to each action $a$ in state $s$. Acting is sampling from that distribution. A deterministic policy is a special case in which all probability mass sits on a single action.

Three structures support the definition. First, a state space $\mathcal{S}$ and an action space $\mathcal{A}$ define the input and output domains. Second, a probability simplex constrains every distribution: probabilities are non-negative and sum to one. Third, a trajectory distribution combines the policy with environment dynamics to produce state-action sequences:

The expected return follows from that distribution:

The expectation runs over trajectories. A single rollout is a sample from $J(\pi)$, not a measurement of the expectation.

The interaction loop makes the roles concrete:

import random
 
def step(env, action):
    env["state"] = env["next_state"](action)
    return env["reward"](action)
 
def run(env, policy, steps=100, seed=0):
    rng = random.Random(seed)
    obs = env["initial_state"]()
    total = 0.0
    for _ in range(steps):
        action = policy(obs, rng)
        obs, reward = step(env, action)
        total += reward
    return total

The environment defines transitions and rewards. The policy defines choices. The random generator defines samples. The interface remains stable across discrete and continuous cases. The bottleneck is never the interface. The bottleneck is the distribution family.

Bottleneck one: a deterministic policy stops learning#

A deterministic table that always selects the currently best action never collects the data that would correct the table. Action selection collapses before the value estimates are accurate, and the error is unobservable because the alternative action never occurs.

The smallest fix places probability mass on every action. Epsilon-greedy is the simplest version:

import random
 
def make_epsilon_greedy(table, epsilon=0.1, rng_seed=0):
    rng = random.Random(rng_seed)
    actions = list(table.values())
 
    def policy(state, rng=None):
        if rng.random() < epsilon:
            return rng.choice(actions)
        return table[state]
 
    return policy

Epsilon-greedy assigns $1 - \epsilon$ to the greedy action and $\epsilon / |\mathcal{A}|$ to each other action. Exploration is uniform and does not depend on the value gap between actions.

A softmax policy makes the mass depend on scores:

import math
 
def softmax_policy(scores, temperature=1.0):
    logits = [s / temperature for s in scores]
    m = max(logits)
    weights = [math.exp(x - m) for x in logits]
    total = sum(weights)
    probs = [w / total for w in weights]
 
    def policy(state, rng):
        r = rng.random()
        acc = 0.0
        for action, p in enumerate(probs):
            acc += p
            if r <= acc:
                return action
        return len(probs) - 1
 
    return policy, probs

Temperature controls spread. A temperature near zero approaches determinism. A temperature near infinity approaches uniformity. Entropy measures the spread:

import math
 
def entropy(probs):
    return -sum(p * math.log(p + 1e-12) for p in probs)

Entropy functions as an exploration bonus. Higher entropy explores more and converges more slowly. Collapse to zero entropy halts exploration. The entropy bonus in soft actor-critic and the entropy term in PPO address exactly that collapse.

Bottleneck two: one distribution cannot cover every action space#

The softmax construction assumes a small, fixed action set. Continuous control has no finite list to normalize over, and a tabular distribution fails for the same reason. The distribution needs a shape suited to the action space.

A Gaussian policy places mass near a learned mean with a learned spread:

import numpy as np
 
def make_gaussian_policy(mean, log_std, rng):
    mean = np.asarray(mean, dtype=np.float32)
    std = np.exp(np.asarray(log_std, dtype=np.float32))
 
    def policy(state, rng=None):
        return rng.normal(loc=mean, scale=std)
 
    return policy, {"mean": mean, "std": std}

Sampling deviates from the mean at a rate the standard deviation controls. A state-dependent mean and standard deviation generalize the construction:

import numpy as np
import torch.nn as nn
 
class GaussianPolicy(nn.Module):
    def __init__(self, obs_dim, act_dim):
        super().__init__()
        self.mean_net = nn.Sequential(
            nn.Linear(obs_dim, 64), nn.Tanh(),
            nn.Linear(64, act_dim),
        )
        self.log_std_net = nn.Sequential(
            nn.Linear(obs_dim, 64), nn.Tanh(),
            nn.Linear(64, act_dim),
        )
 
    def distribution_params(self, obs):
        mean = self.mean_net(obs)
        log_std = self.log_std_net(obs).clamp(-20, 2)
        return mean, log_std.exp()

Deterministic policies in continuous control take a different route. Noise is injected around the deterministic action, separating the policy from the exploration source. The policy itself remains deterministic and the behavior policy becomes stochastic. The learned artifact is the deterministic one while the collected data comes from the noisy one. The distinction matters for logging and evaluation.

Bounded action spaces add one more distortion. A Gaussian is unbounded. Squashing through a tanh and correcting for the Jacobian keeps the density valid:

import torch
import torch.nn.functional as F
 
def squash_correction(u):
    # log |det Jacobian| for tanh(y), y in (-1, 1)
    return 2 * u - 2 * F.softplus(2 * u)
 
def sample_squashed(mean, std, rng):
    z = rng.normal(loc=0.0, scale=1.0)
    u = mean + std * z
    a = torch.tanh(u)
    log_prob = torch.distributions.Normal(mean, std).log_prob(u)
    log_prob = log_prob - squash_correction(u)
    return a, log_prob

Without the correction, likelihood ratios are biased and policy gradients point in the wrong direction.

Bottleneck three: hidden information needs randomization#

Game theory supplies a second reason for stochasticity. In an imperfect-information game — poker, auctions, bluffing — a deterministic policy is predictable, and a predictable policy is exploitable.

Rock-paper-scissors illustrates the mechanism. Any deterministic strategy loses to a fixed counter-strategy. The equilibrium strategy is a distribution, typically uniform. A policy that collapses to one action is strictly worse because the game has no deterministic equilibrium.

def equilibrium_policy(state, rng):
    return rng.choice(["rock", "paper", "scissors"], p=[1/3, 1/3, 1/3])
 
def counter_table(state, rng):
    counter = {"rock": "paper", "paper": "scissors", "scissors": "rock"}
    return counter[state]

The same logic runs through partially observable environments. When the state is incompletely observed, one observation corresponds to multiple underlying states with different optimal actions. A policy conditioned only on the observation cannot distinguish those states, and a distribution represents the remaining uncertainty. The stochasticity encodes a posterior.

A distribution is also a commitment to a frequency. A policy that selects action $a$ with probability 0.3 selects it about 30 percent of the time over many visits to the same state. Frequency makes behavior legible to an opponent or an environment. A predictable stochasticity can be modeled, and in adversarial settings the model is the equilibrium.

import random
 
def empirical_frequency(policy, state, draws=10_000, seed=0):
    rng = random.Random(seed)
    counts = {}
    for _ in range(draws):
        a = policy(state, rng)
        counts[a] = counts.get(a, 0) + 1
    total = sum(counts.values())
    return {a: c / total for a, c in counts.items()}

Correlated action spaces add a further limitation. Independent per-dimension Gaussians cannot express that actions must lie on a low-dimensional manifold. A factorized distribution samples each dimension independently, wasting probability mass off-manifold. Two fixes follow. An autoregressive distribution samples dimensions sequentially, conditioning each on the previous ones. A mixture of Gaussians places mass on several modes instead of one. Both increase expressiveness at the cost of sampling complexity.

import numpy as np
 
def mixture_density(x, means, log_stds, logits):
    # x: (d,), means: (k, d), log_stds: (k, d), logits: (k,)
    stds = np.exp(log_stds)
    log_weights = logits - np.log(np.sum(np.exp(logits)))
    log_comps = []
    for k in range(len(means)):
        log_comp = -0.5 * np.sum(((x - means[k]) / stds[k]) ** 2)
        log_comp += -np.sum(np.log(stds[k]))
        log_comps.append(log_comp + log_weights[k])
    m = max(log_comps)
    total = m + np.log(sum(np.exp(v - m) for v in log_comps))
    return total

A unimodal policy cannot represent a decision that required a choice between two distant behaviors. The mixture can.

What to measure#

A policy stated as a distribution is a complete specification of behavior. Given the state, the distribution, and a random number generator, the entire trajectory follows. Determinism is a choice of where to place probability mass. Stochasticity is the general case that keeps every other choice representable.

Evaluation of a stochastic policy requires care. The expected return is an expectation over the distribution, so a single rollout is a sample rather than a number. A protocol that reports one deterministic rollout of a stochastic policy measures the luck of the seed rather than the policy.

def evaluate(policy, env, episodes=20, steps=100):
    returns = []
    for ep in range(episodes):
        returns.append(run(env, policy, steps, seed=ep))
    mean = sum(returns) / len(returns)
    var = sum((r - mean) ** 2 for r in returns) / len(returns)
    return {"mean": mean, "std": var ** 0.5, "returns": returns}

A complete measurement checklist:

• Report entropy alongside return. Zero entropy plus high training return signals coverage failure rather than mastery.
• Report the number of distinct actions per state, or the standard deviation for continuous actions. One action per state signals collapse.
• Evaluate over multiple seeds and report mean and variance. One seed is an anecdote about the generator.
• Check exploitability in adversarial settings. Test against a fixed counter-strategy and against the equilibrium. A large gap between the two signals a deterministic leak.
• Separate the learned policy from the behavior policy. Log which distribution generated each action, especially when noise is injected around a deterministic actor.
• Verify bounded-action densities. Without a Jacobian correction, log-probabilities are wrong and importance weights drift.
• Test mode coverage. A policy that must choose between distant behaviors needs a mixture or an autoregressive form, and the test is whether both modes appear in samples.
• Keep the deterministic representation for deployment only after the stochastic checks pass. Dropping the distribution discards exploration, hidden-state posteriors, and equilibrium protection in one step.

The literature that treats "policy" as synonymous with "deterministic function" uses the special case as if it were the whole. The distribution is the notion that survives contact with exploration, hidden information, and evaluation. Mass placement is the decision. The agent takes every action the distribution allows, at the rate the distribution specifies.