Where did all my probability go?

Let $X$ be a one-dimensional normal variable with mean 0 and standard deviation $\sigma$. What is the limit of $P(X>a|\sigma )$ when $\sigma$ tends to infinity?

We can compute the probability $P(X>a|\sigma )$ as an integral of the normal density function:

$$P(X>a|\sigma )={\int }_a^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx.$$

As $\sigma$ goes to infinity, $\frac{1}{\sqrt{2\pi }\sigma }$ goes to 0, and $e^{-x^2/2{\sigma }^2}$ goes to 1. Their product, therefore, goes to 0, and so

$$\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>a|\sigma )={\int }_a^{+\mathrm{\infty }}\underset{\sigma \to +\mathrm{\infty }}{lim}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx=0.$$

But this does not make sense, does it? As the standard deviation increases, the normal distribution becomes more spread out, and the probability that $X$ will exceed a fixed threshold $a$ should increase, not fall down to 0.

Indeed, the same argument could be used to “show” that the probability $P(X\le a|\sigma )$ also tends to 0 as $\sigma$ increases. So where does all the probability go?

The problem in our reasoning is swapping the integral and the limit operations, i.e. equating

$$\underset{\sigma \to +\mathrm{\infty }}{lim}{\int }_a^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx$$

to

$${\int }_a^{+\mathrm{\infty }}\underset{\sigma \to +\mathrm{\infty }}{lim}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx.$$

But why exactly this is a problem requries a little digging.

Proper integrals

We might suspect that the reason swapping the integral and the limit didn’t work is that we are integrating over an infinite interval $[a;+\mathrm{\infty })$; i.e. that our integral is improper.

One reason to think so is that we would be right to conclude that $$\underset{\sigma \to +\mathrm{\infty }}{lim}P(a\le X\le b|\sigma )=0,$$ where the integration happens over a finite interval $[a,b]$.

The improperness of the integral does play a role, as we shall see below; but by itself, properness is neither necessary nor sufficient to swap the integral and the limit.

A. Ya. Dorogovtsev in his mathematical analysis textbook gives the following example (section 13.2.3):

$$f(x,y)=\frac{1}{y}(1-x^{1/y})x^{1/y},$$

where $x\in [1/2,1]$ and $y\in (0,1]$.

For any given $y$, $f(\cdot ,y):[1/2,1]\to \mathbb{R}$ is a continuous function on a closed interval, so ${\int }_{1/2}^{1}f(x,y)dx$ is proper. And yet, it can be shown that

$$\mathrm{\forall }x\in [1/2,1]\underset{y\to 0+}{lim}f(x,y)=0$$ while $$\underset{y\to 0+}{lim}{\int }_{1/2}^{1}f(x,y)dx=1/2.$$

Let’s look at the graph of $f(x,y)$ for different values of $y$ to see what’s going on.

On the one hand, at any given $x$, $f(x,y)$ tends to 0. (Look at how the function value changes for a fixed $x$, say, $x=0.8$ or $x=0.9$. It may be less obvious that the same is true for $x=0.99$; you’ll have to trust that the trend continues or verify it analytically.)

On the other hand, the function as a whole does not seem to converge to 0. This is formalized by the notion of uniform convergence.

We say that $g(x,y)$ converges uniformly to $g(x)$ when $y\to y_0$ iff the maximum vertical distance between the graphs of $g(x,y)$ and $g(x)$, i.e. $\underset{x}{sup}|g(x,y)-g(x)|$, goes to 0.

Uniform convergence on $[a,b]$ is sufficient (although not necessary) to replace $\underset{y\to y_0}{lim}{\int }_a^{b}g(x,y)dx$ with ${\int }_a^b\underset{y\to y_0}{lim}g(x,y)dx$.

Clearly, our $f(x,y)$ converges to 0 non-uniformly: not only does $\underset{x}{sup}|f(x,y)|$ not tend to 0, it tends to infinity. Non-uniform convergence allows a travelling wave. Such a wave can contribute to the integral while escaping the point-wise convergence analysis (the function converges at every point, but only after the wave has passed).

Improper integrals

Let’s revisit our first example, the limit $$\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>a|\sigma )dx=\underset{\sigma \to +\mathrm{\infty }}{lim}{\int }_a^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx.$$

Here again there is a wave traveling to the right (and a symmetric one traveling to the left, not shown here).

Because the wave travels with a finite speed, we may suspect non-uniform convergence: no matter how large $\sigma$ is, there are always distant enough points at which the density will temporarily increase as $\sigma$ continues to increase.

And yet, the height of the wave diminishes, and $\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}$ does converge to 0 uniformly:

$$\underset{x}{sup}\left|\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}\right|=\frac{1}{\sqrt{2\pi }\sigma }\to 0(\sigma \to +\mathrm{\infty }).$$

The reason why this is not enough to bring the limit under the integral is that here we are dealing with an improper integral. Improper integrals are themselves limits; recall that the improper integral is defined as

$${\int }_a^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx=\underset{b\to +\mathrm{\infty }}{lim}{\int }_a^b\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx.$$

Before we get to bring $\underset{\sigma \to +\mathrm{\infty }}{lim}$ under ${\int }_a^b$, we need to sneak it past $\underset{b\to +\mathrm{\infty }}{lim}$ first. For that, the limit $$\underset{b\to +\mathrm{\infty }}{lim}{\int }_a^b\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx$$ itself needs to converge uniformly as a function of $\sigma$; i.e. $$\underset{\sigma }{sup}\left|{\int }_b^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx\right|\to 0(b\to +\mathrm{\infty }).$$

Look, this is the same quantity that we started with: because the integral is a positive and increasing function of $\sigma$,

$$\begin{array}{rl}\underset{\sigma }{sup}\left|{\int }_b^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx\right|& =\underset{\sigma \to +\mathrm{\infty }}{lim}{\int }_b^{+\mathrm{\infty }}\frac{1}{\sqrt{2\pi }\sigma }{e}^{-x^2/2{\sigma }^2}dx\\ & =\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>b|\sigma ).\end{array}$$

If we assume that $\mathrm{\forall }b\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>b|\sigma )=0$, then the integral converges uniformly, we can bring the limit under the integral, and prove that indeed, $\mathrm{\forall }a\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>a|\sigma )=0$.

But there is no reason to believe that the limit is zero, and that explains why our calculation yielding 0 was wrong: the integral does not converge uniformly.

The actual calculation of $\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>a|\sigma )$ is simple: because $\mathrm{\forall }a>0\underset{\sigma \to +\mathrm{\infty }}{lim}P(-a\le X\le a|\sigma )=0$ (thanks to uniform convergence!), all the probability mass goes to infinity, and because of the symmetry, each tail gets a half of it; so

$$\underset{\sigma \to +\mathrm{\infty }}{lim}P(X>a|\sigma )=1/2.$$

But I thought it would be instructive to investigate why the naive calculation did not work, so here we are.