Where did all my probability go?

Let \(X\) be a one-dimensional normal variable with mean 0 and standard deviation \(\sigma\). What is the limit of \(P(X>a|\sigma)\) when \(\sigma\) tends to infinity?

We can compute the probability \(P(X>a|\sigma)\) as an integral of the normal density function:

\[ P(X>a|\sigma)=\int_{a}^{+\infty}\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

As \(\sigma\) goes to infinity, \(\frac{1}{\sqrt{2\pi}\sigma}\) goes to 0, and \(e^{-x^2/2\sigma^2}\) goes to 1. Their product, therefore, goes to 0, and so

\[ \lim_{\sigma\to+\infty}P(X>a|\sigma)= \int_{a}^{+\infty}\lim_{\sigma\to+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx =0. \]

But this does not make sense, does it? As the standard deviation increases, the normal distribution becomes more spread out, and the probability that \(X\) will exceed a fixed threshold \(a\) should increase, not fall down to 0.

Indeed, the same argument could be used to “show” that the probability \(P(X\leq a|\sigma)\) also tends to 0 as \(\sigma\) increases. So where does all the probability go?

The problem in our reasoning is swapping the integral and the limit operations, i.e. equating

\[ \lim_{\sigma\to+\infty}\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx \]

to

\[ \int_{a}^{+\infty}\lim_{\sigma\to+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

But why exactly this is a problem requries a little digging.

Proper integrals

We might suspect that the reason swapping the integral and the limit didn’t work is that we are integrating over an infinite interval \([a;+\infty)\); i.e. that our integral is improper.

One reason to think so is that we would be right to conclude that \[\lim_{\sigma\to+\infty} P(a\leq X\leq b|\sigma) = 0,\] where the integration happens over a finite interval \([a,b]\).

The improperness of the integral does play a role, as we shall see below; but by itself, properness is neither necessary nor sufficient to swap the integral and the limit.

A. Ya. Dorogovtsev in his mathematical analysis textbook gives the following example (section 13.2.3):

\[ f(x,y)=\frac1y\left(1-x^{1/y}\right)x^{1/y}, \]

where \(x\in[1/2,1]\) and \(y\in(0,1]\).

For any given \(y\), \(f(\cdot,y):[1/2,1]\to\mathbb{R}\) is a continuous function on a closed interval, so \(\int_{1/2}^1 f(x,y)dx\) is proper. And yet, it can be shown that

\[ \forall x\in[1/2,1]\; \lim_{y\to 0+} f(x,y) = 0 \] while \[ \lim_{y\to 0+}\int_{1/2}^1 f(x,y)dx = 1/2. \]

Let’s look at the graph of \(f(x,y)\) for different values of \(y\) to see what’s going on.

On the one hand, at any given \(x\), \(f(x,y)\) tends to 0. (Look at how the function value changes for a fixed \(x\), say, \(x=0.8\) or \(x=0.9\). It may be less obvious that the same is true for \(x=0.99\); you’ll have to trust that the trend continues or verify it analytically.)

On the other hand, the function as a whole does not seem to converge to 0. This is formalized by the notion of uniform convergence.

We say that \(g(x,y)\) converges uniformly to \(g(x)\) when \(y\to y_0\) iff the maximum vertical distance between the graphs of \(g(x,y)\) and \(g(x)\), i.e. \(\sup_x |g(x,y)-g(x)|\), goes to 0.

Uniform convergence on \([a,b]\) is sufficient (although not necessary) to replace \(\lim_{y\to y_0}\int_a^b g(x,y)dx\) with \(\int_a^b \lim_{y\to y_0}g(x,y)dx\).

Clearly, our \(f(x,y)\) converges to 0 non-uniformly: not only does \(\sup_x |f(x,y)|\) not tend to 0, it tends to infinity. Non-uniform convergence allows a travelling wave. Such a wave can contribute to the integral while escaping the point-wise convergence analysis (the function converges at every point, but only after the wave has passed).

Improper integrals

Let’s revisit our first example, the limit \[ \lim_{\sigma\to+\infty}P(X>a|\sigma)dx = \lim_{\sigma\to+\infty}\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

Here again there is a wave traveling to the right (and a symmetric one traveling to the left, not shown here).

Because the wave travels with a finite speed, we may suspect non-uniform convergence: no matter how large \(\sigma\) is, there are always distant enough points at which the density will temporarily increase as \(\sigma\) continues to increase.

And yet, the height of the wave diminishes, and \(\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}\) does converge to 0 uniformly:

\[ \sup_x \left|\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}\right|= \frac{1}{\sqrt{2\pi}\sigma}\to 0\quad (\sigma\to+\infty). \]

The reason why this is not enough to bring the limit under the integral is that here we are dealing with an improper integral. Improper integrals are themselves limits; recall that the improper integral is defined as

\[ \int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx = \lim_{b\to+\infty}\int_{a}^{b} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

Before we get to bring \(\lim_{\sigma\to+\infty}\) under \(\int_a^b\), we need to sneak it past \(\lim_{b\to+\infty}\) first. For that, the limit \[\lim_{b\to+\infty}\int_{a}^{b} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\] itself needs to converge uniformly as a function of \(\sigma\); i.e. \[\sup_{\sigma} \left|\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\right|\to 0\quad (b\to+\infty).\]

Look, this is the same quantity that we started with: because the integral is a positive and increasing function of \(\sigma\),

\[ \begin{split} \sup_{\sigma} \left|\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\right|&= \lim_{\sigma\to+\infty}\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\\&= \lim_{\sigma\to+\infty}P(X>b|\sigma). \end{split} \]

If we assume that \(\forall b\:\lim_{\sigma\to+\infty}P(X>b|\sigma)=0\), then the integral converges uniformly, we can bring the limit under the integral, and prove that indeed, \(\forall a\:\lim_{\sigma\to+\infty}P(X>a|\sigma)=0\).

But there is no reason to believe that the limit is zero, and that explains why our calculation yielding 0 was wrong: the integral does not converge uniformly.

The actual calculation of \(\lim_{\sigma\to+\infty}P(X>a|\sigma)\) is simple: because \(\forall a>0\:\lim_{\sigma\to+\infty}P(-a\leq X\leq a|\sigma)=0\) (thanks to uniform convergence!), all the probability mass goes to infinity, and because of the symmetry, each tail gets a half of it; so

\[\lim_{\sigma\to+\infty}P(X>a|\sigma)=1/2.\]

But I thought it would be instructive to investigate why the naive calculation did not work, so here we are.