The Key Equation Behind Probability

Probability thinking—assigning likelihoods rather than single answers—sits at the core of how both brains and machine-learning systems handle uncertainty. A foggy walk-home example captures the point: when sensory information is ambiguous, the mind effectively carries a probability distribution over possible interpretations. That same probabilistic machinery powers generative modeling, where systems learn a distribution from data, then sample from it to produce new, plausible outcomes.

The discussion begins with probability distributions: a mapping from every possible state to a number between 0 and 1, with the constraint that all probabilities sum to one. For discrete cases like a fair six-sided die, each face gets probability 1/6. For continuous variables like adult height, the distribution must integrate to one (area under the curve), which can be pictured as many tiny bins. When the “state” includes multiple variables—height and weight, or even all pixels in a 100 by 100 image—the distribution lives in a high-dimensional space, making visualization impossible but the underlying idea unchanged.

To use such distributions, the key operation is sampling: drawing new outcomes according to the learned probabilities. A die roll is the simplest example; a generative AI model extends the same logic to complex data, producing images that reflect structure in natural data rather than independent random noise per pixel.

From there, the math is built around “surprise.” Surprise should be larger when an event is unlikely and should add across independent events. The function that satisfies these requirements is proportional to log(1/p). Averaging surprise over the distribution yields entropy, a measure of inherent uncertainty. The transcript contrasts a fair coin with a “thick coin” that lands on its side 2% of the time: the side outcome creates rare but highly surprising events, raising the overall entropy.

A crucial real-world twist follows: the true distribution is usually hidden. People and models rely on an internal belief distribution Q, not the real one P. When observations come from P but predictions use Q, the expected surprise becomes cross entropy. Cross entropy is always at least the entropy of P, meaning a wrong model can only increase expected surprise. It also behaves asymmetrically: believing a fair coin is rigged produces a different cross-entropy than believing a rigged coin is fair, because rare outcomes under the mistaken model can dominate the average.

To isolate the extra surprise caused specifically by model mismatch, the transcript introduces Kullback-Leibler divergence (KL divergence). By subtracting entropy from cross entropy, KL divergence measures the additional penalty of using Q instead of P. In machine learning training, this matters because minimizing KL divergence is equivalent to minimizing cross entropy: the entropy term H(P) does not depend on the model parameters, so it shifts values but not which model Q is optimal. That equivalence helps explain why cross-entropy objectives dominate generative modeling, even when the true distribution cannot be directly computed.