Entropy and Normal Distributions
I started writing this page because I wanted to understand what’s so special about normal distributions (also called Gaussian distributions, which I’ll use interchangeably). They appear in many corners of statistics, physics, and information theory, and the reasons in each case seemed unrelated to me. There’s an explicit formula for the normal distribution (i.e., for variance $\sigma^2$ and centered at mean zero, $p(x) = \frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{x^2}{2\sigma^2}}$), but to me this offers no “moral” explanation for its qualitative appearances.
Normal distributions in physics and statistics
Here’s the example from statistical physics. Consider a large number of identical particles moving around in a box with different speeds and directions. All particle-box and particle-particle collisions are elastic. After a long time, how will the velocity vectors of the particles be distributed? What about the positions of the particles? Below is a simulation, and at the right are live histograms of the y-position (height), kinetic energy, x-velocity, and y-velocity of the particles. You can tune the number of particles N, temperature T (= mean kinetic energy), strength of gravity g, and number of bins for y-position.
What you’ll observe is that the first two histograms equilibriate to exponential curves, while the third and fourth equilibriate to normal distributions, drawn in red. This system empirically exhibits Maxwell-Boltzmann statistics, and can be interpreted as saying that the normal distribution (resp. exponential distribution) on physical variables (resp. energy) is the “thermal equilibrium” for any system.
The real-life upshot is that physical systems in contact with a state at thermal equilibrium inherit Gaussian noise (which is why voltage/current fluctuations in a resistor are Gaussian).
Here’s the example from statistics. Flip a coin a large number of times, and then count the proportion of times it came up heads. The most probable outcome is that exactly half of the outcomes are heads, but how does the probability distribution look for proportions away from the halfway mark? This is the binomial distribution. As , this curve (appropriately rescaled horizontally by a factor of ) approaches the shape of the normal distribution. This fact can be physically manifested with a device called a Galton board.
More generally, the same limiting bell-curve shape appears in describing the outcome of any repeated-trial experiment: e.g., if you were throwing a multi-sided die a large number of times and averaging the results, or flipping an unfair coin a large number of times and counting the fraction of heads (it would just be centered somewhere different from 1/2). This is the central limit theorem.
(If you shrink down the left-right steps to size $1/\sqrt{N}$ and also let $N \to \infty$, you get Brownian motion. This is why the probability distribution describing the position of a particle undergoing Brownian motion evolves like a normal distribution whose variance grows over time. That’s also how heat spreads out from a singular heat source in a flat material that’s isotropic, i.e. where heat spreads at an equal rate in all directions.)
Entropy
The best explanation I’ve found linking these appearances comes from entropy. Out of all probability distributions with a given variance, the normal distribution has the most entropy out of any distribution with the same variance.
Since variance is a measure of “energy”, and energy is conserved (first law of thermodynamics), it follows that in the first example, the distribution of velocities will tend towards one of maximal entropy (second law of thermodynamics), and that’s a normal distribution.
I first learned the term entropy in my high school chemistry class, alongside the vague definition as “disorder”, and second law of thermodynamics: entropy is always increasing. It bothered me because my teacher never told me how we could measure entropy, or even define it mathematically — then what does it mean to say entropy is increasing?
Next, I learned about entropy in college, while in a theoretical computer science class. We were touching upon some concepts of information theory, and I learned about how entropy is relevant to problems of information compression a la Shannon’s channel coding theorem). I was given an explicit mathematical expression, which was good. But I still didn’t really feel what entropy is deep in my bones. Why was this formula the same as the entropy used in physics?
As I have done professional work in science and technology in the 2020s, it’s been impossible to ignore the effects of AI. Large language models have generalized to a vast number of applications in knowledge work. It turns out that the mathematical underpinnings of modern machine learning draw heavily upon concepts from both physics (through e.g. diffusion models) and information theory (through language modeling), and so by trying to develop a better understanding of the field, I once again ran into the same concepts: Gaussian noise, entropy, etc.
The point of this page is to define entropy and try to develop multiple mental models on what it is, thereby connecting these various viewpoints on normal distributions.
What is entropy?
I want to step back and give a very simple, mathematical formulation of what entropy is.
Entropy measures how uncertain or disparate the outcomes of a random event are.
It has a very concrete formula. If a random event $X$ can take on $n$ different possible outcomes with probabilities $p_1, p_2, \ldots, p_n$ (whose sum equals $1$) then the entropy, $H(X)$, is negative the sum of each probability value times its logarithm.
$$H(X) = -\sum\limits_{i=1}^{n}p_i\log(p_i)$$
For a continuous probability distribution (such as the spread of the velocities of particles in a box) which has a whole continuum of possible outcomes, the entropy becomes an integral, e.g. .
Entropy (unlike e.g. mean or standard deviation) doesn’t require that the event outcomes are structured in any way, such as being numerical values. The measure of “spread-out-ness” has nothing to do with such values, only that the outcomes are distinct. So in a sense entropy is one of the most basic concepts in probability theory.
A simple example to ground ourselves is an event with just two possible outcomes, e.g. flipping an unfair coin which turns up heads with probability $p$ and tails with probability $1-p$. The entropy is symmetric with respect to interchanging $p$ and $1-p$ (which is the same as replacing heads for tails). When $p$ is close to $0$ or close to $1$, the entropy is almost zero, and the entropy is highest when $p=1/2$.
The logarithm here can be taken in any base. In coding and information theory it’s preferred to be log base 2 (as I’ll justify in the next section), and in statistical mechanics it’s preferred to be log base e (as I’ll justify in the section after that). Using log base 2 for now means that flipping a fair coin is an entropy 1 random event.
An interpretation of entropy in plain English is “the expected amount of information gained from one sample”, where unexpected outcomes have higher information content than lower information ones — more precisely, a probability-$p$ outcome has “information content: equal to $-\log(p)$.1
Why logarithms? In what sense is $-\log(p)$ a sensible way to measure “information content”? That’s is a nice segue to Shannon’s theory of communication.
Entropy and information theory
Entropy has a natural occurrence in the theory of communication and codes, going back to Claude Shannon’s seminal paper on information theory, A Mathematical Theory of Communication.2
In working through these ideas for myself, I learned of this old and relevant post from Chris Olah, which already nicely ties together all of the ideas I was trying to capture in this section, with great examples and visuals. In lieu of giving the details myself, I’m going to simply leave a reference to that post because it already says everything I wanted to say and more.
The core point is as follows. Let’s say you’re trying to communicate symbols across a communication channel, where there are possible symbols occurring with respective frequencies . Your communication channel only allows sending 0s and 1s. You want to encode the symbols into encoded bit strings in such a way that (1) any bit string that arrives at the other end can be uniquely broken into a concatenation of encoded bit strings and thus uniquely decoded into the base symbols, and (2) the expected bit-rate of the encoded language is as small as possible. It turns out that the entropy we defined before is the lowest-possible bit-rate that’s achievable.
More recently, there’s an excellent recent video from 3Blue1Brown about entropy and information theory, the first in an ongoing series which I hope will speak to its applications in natural language processing.
Entropy in thermodynamics
Imagine that you have a fluid (such as a liquid) living in a collection of chambers of a tank, labeled through . Let’s say that you somehow know a fraction of the fluid is in the first chamber (e.g. with a pressure and temperature sensor), in the second chamber, etc. where .
Now suppose that there are total fluid particles, for some number that’s much larger than . How many possible different ways are there to sort these particles into the chambers so that we observe the same fractional distribution described above?
The point here is the distinction between the macroscopic view of the system and the microscopic one. The macroscopic state we observe is that each chamber has a certain fraction of the particles. The microscopic state keeps track of each individual particle. The macroscopic view is like a census-level view of a population, while the microscopic one considers the individuals as distinct. There are many possible microscopic states which lead to the same macroscopic state. How many, though? Call that number $\Omega$.
The total number of microstates can be written explicitly as a ratio of factorials $$\Omega = \frac{N!}{(Np_1)!(Np_2)!\ldots (Np_n)!}$$ Taking the natural logarithm of both sides gives $$\log \Omega = \log(N!) – \sum\limits_{i=1}^n \log((Np_i)!)$$ Stirling’s approximation tells us that the natural logarithm of a factorial grows in a controlled manner for large values: $\log(k!) \approx k\log(k) – k + \frac{1}{2}\log(2\pi k) + O(1/k)$. Applying this approximation to the sum given yields (after algebraic simplification)
$$\log \Omega \approx N\log(N) – N + \frac{1}{2}\log(2\pi N) – \sum\limits_{i=1}^n (Np_i\log(Np_i) – Np_i – \frac{1}{2}\log(2\pi Np_i))$$
After using the fact that $\sum\limits_{i=1}^n p_i=1$, many terms cancel here, leaving us with
$$\log \Omega \approx -N\sum\limits_{i=1}^n p_i\log(p_i) + O(\log(N))$$
$$= NH(X) + O(\log(N))$$
In other words, the dominant term in the number of possible microstates is the entropy of the distribution, with the relation being of the form $\frac{\log \Omega}{N} = H(X)$ as $N \rightarrow \infty$. The quantity $\log \Omega$ has an information-theoretic interpretation, as the approximate number of bits required to specify a given microstate, given the macrostate.
If $\Omega \approx e^{N H(X)}$, then when $N$ is very large, the macrostate with the highest entropy — i.e. with an equal fraction associated to every chamber — has a LOT more associated microstates than any other macrostate. That’s why we always see the increase to maximal entropy.
But we don’t always see the uniform distribution between chambers. For example, what if the chambers have different sizes? What if there’s a force pulling particles towards one of the chambers? This entropy calculation says nothing about the physics of the system, such as whether one chamber is preferred over another for some reason, or even the process of how particles transition from one chamber to another. The next section runs through a few examples of entropy-maximizing distributions under constraints, which can be thought of as analogous to adding simple physical laws to the simulation above.
Entropy-maximization under constraints: three examples
We already have seen that for a random event $X$ with $n$ possible outcomes with probabilities $p_1, p_2, \ldots, p_n$, the entropy $H(X)$ is maximized when $p_1=p_2=\ldots=p_n=\frac{1}{n}$. This is reflected by the gas simulation in the $n$-chamber tank from the last section. In other words, among all probability distributions on a finite set, entropy is maximized when the distribution is uniform.
What if we include a force pulling particles to the left (analogous to gravitational pull)? Then we’d expect to find more particles in the leftmost chamber. It turns out the fraction of particles in each chamber follow an exponentially decaying curve (a Gibbs distribution). This is a consequence of the following fact: among all continuous probability distributions $p(x)$ taking values in the positive real numbers having a fixed mean value $E[X] = c$, the one with the highest entropy is the exponential distribution $p(x) = \frac{1}{c}e^{-x/c}$. The number $c$ here is the mean potential energy at equilibrium. (More simply, since kinetic and potential energy interchange for a single particle, it’s easier to plot a histogram of KE + PE among particles and see that this, too, follows a Gibbs distribution whose mean value remains unchanging under evolution of the system. The fact that it’s unchanging is by conservation of energy, which is the important physics input here.)
What about the distribution on the x- and y-components of velocity, as in the gas simulation at the top of this page?
Let’s go back to the zero-gravity case, so there’s no potential energy — only kinetic. Since kinetic energy goes as the square of velocity, and the mean velocity of the particles is zero at equilibrium (the center of mass has to be at the center of the box), the total energy of the system is the variance of the distribution on velocity. Conservation of energy thus becomes a constraint on this variance, and the emergence of the normal distribution is captured in the following mathematical fact: among all continuous probability distributions $p(x)$ taking values in the real numbers having mean $0$ and a fixed variance $\sigma^2$, the one with the highest entropy is the normal distribution $p(x) = \frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{x^2}{2\sigma^2}}$.
Derivations of these statements are given in these notes from Keith Conrad, in section 4, using cross-entropy.
Summary and sequel
The main message here is that normal distributions appear so frequently in equilibrium in physics because
- energy is conserved, and thermal equilibrium means that the average energy (temperature) within the components of a system is fixed;
- thus this physical law into a constraint on the mean of the energy distribution;
- energy is often given by the square of measured quantities (like position, velocity, current, etc), so this mean constraint converts into a variance constraint on the measure quantity of the system;
- the distribution with the highest entropy, subject to a fixed variance, is a normal distribution.
However, the widespread appearance in statistics (i.e. the Central Limit Theorem) appears at first glance to come from a different place. It turns out that there is an interpretation of the Central Limit Theorem in terms of entropy maximization — namely, that mixing random variables by adding them (i.e., taking the convolution of their densities) increases entropy. I went down quite a rabbit hole to try to make sense of this (see in order, 1, 2, 3, 4), and it’s more involved than I had thought, so I’m going to save it for a later post.
- Each day, either the sun rises (with probability near 1) or doesn’t rise (with probability near 0). The latter is much more surprising and gives us much more information about the world, precisely because it’s much lower probability in our current world model.
- There’s a story that when Shannon had mathematically defined what is now known as “entropy”, he was going to call it “information” or “uncertainty”. However, Von Neumann urged him to call it “entropy”, firstly because it already has that name in thermodynamics, and secondly (and more importantly) because nobody really understand entropy, and so if Shannon used that term he would win in arguments about it. Politically savvy!