# Where did all my probability go?

Published on

Let $X$ be a one-dimensional normal variable with mean 0 and standard deviation $\sigma$. What is the limit of $P(X>a|\sigma)$ when $\sigma$ tends to infinity?

We can compute the probability $P(X>a|\sigma)$ as an integral of the normal density function:

$P(X>a|\sigma)=\int_{a}^{+\infty}\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx.$

As $\sigma$ goes to infinity, $\frac{1}{\sqrt{2\pi}\sigma}$ goes to 0, and $e^{-x^2/2\sigma^2}$ goes to 1. Their product, therefore, goes to 0, and so

$\lim_{\sigma\to+\infty}P(X>a|\sigma)= \int_{a}^{+\infty}\lim_{\sigma\to+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx =0.$

But this does not make sense, does it? As the standard deviation increases, the normal distribution becomes more spread out, and the probability that $X$ will exceed a fixed threshold $a$ should increase, not fall down to 0.

Indeed, the same argument could be used to “show” that the probability $P(X\leq a|\sigma)$ also tends to 0 as $\sigma$ increases. So where does all the probability go?

The problem in our reasoning is swapping the integral and the limit operations, i.e. equating

$\lim_{\sigma\to+\infty}\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx$

to

$\int_{a}^{+\infty}\lim_{\sigma\to+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx.$

But why exactly this is a problem requries a little digging.

## Proper integrals

We might suspect that the reason swapping the integral and the limit didn’t work is that we are integrating over an infinite interval $[a;+\infty)$; i.e. that our integral is improper.

One reason to think so is that we would be right to conclude that $\lim_{\sigma\to+\infty} P(a\leq X\leq b|\sigma) = 0,$ where the integration happens over a finite interval $[a,b]$.

The improperness of the integral does play a role, as we shall see below; but by itself, properness is neither necessary nor sufficient to swap the integral and the limit.

A. Ya. Dorogovtsev in his mathematical analysis textbook gives the following example (section 13.2.3):

$f(x,y)=\frac1y\left(1-x^{1/y}\right)x^{1/y},$

where $x\in[1/2,1]$ and $y\in(0,1]$.

For any given $y$, $f(\cdot,y):[1/2,1]\to\mathbb{R}$ is a continuous function on a closed interval, so $\int_{1/2}^1 f(x,y)dx$ is proper. And yet, it can be shown that

$\forall x\in[1/2,1]\; \lim_{y\to 0+} f(x,y) = 0$ while $\lim_{y\to 0+}\int_{1/2}^1 f(x,y)dx = 1/2.$

Let’s look at the graph of $f(x,y)$ for different values of $y$ to see what’s going on.

On the one hand, at any given $x$, $f(x,y)$ tends to 0. (Look at how the function value changes for a fixed $x$, say, $x=0.8$ or $x=0.9$. It may be less obvious that the same is true for $x=0.99$; you’ll have to trust that the trend continues or verify it analytically.)

On the other hand, the function as a whole does not seem to converge to 0. This is formalized by the notion of uniform convergence.

We say that $g(x,y)$ converges uniformly to $g(x)$ when $y\to y_0$ iff the maximum vertical distance between the graphs of $g(x,y)$ and $g(x)$, i.e. $\sup_x |g(x,y)-g(x)|$, goes to 0.

Uniform convergence on $[a,b]$ is sufficient (although not necessary) to replace $\lim_{y\to y_0}\int_a^b g(x,y)dx$ with $\int_a^b \lim_{y\to y_0}g(x,y)dx$.

Clearly, our $f(x,y)$ converges to 0 non-uniformly: not only does $\sup_x |f(x,y)|$ not tend to 0, it tends to infinity. Non-uniform convergence allows a travelling wave. Such a wave can contribute to the integral while escaping the point-wise convergence analysis (the function converges at every point, but only after the wave has passed).

## Improper integrals

Let’s revisit our first example, the limit $\lim_{\sigma\to+\infty}P(X>a|\sigma)dx = \lim_{\sigma\to+\infty}\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx.$

Here again there is a wave traveling to the right (and a symmetric one traveling to the left, not shown here).

Because the wave travels with a finite speed, we may suspect non-uniform convergence: no matter how large $\sigma$ is, there are always distant enough points at which the density will temporarily increase as $\sigma$ continues to increase.

And yet, the height of the wave diminishes, and $\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}$ does converge to 0 uniformly:

$\sup_x \left|\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}\right|= \frac{1}{\sqrt{2\pi}\sigma}\to 0\quad (\sigma\to+\infty).$

The reason why this is not enough to bring the limit under the integral is that here we are dealing with an improper integral. Improper integrals are themselves limits; recall that the improper integral is defined as

$\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx = \lim_{b\to+\infty}\int_{a}^{b} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx.$

Before we get to bring $\lim_{\sigma\to+\infty}$ under $\int_a^b$, we need to sneak it past $\lim_{b\to+\infty}$ first. For that, the limit $\lim_{b\to+\infty}\int_{a}^{b} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx$ itself needs to converge uniformly as a function of $\sigma$; i.e. $\sup_{\sigma} \left|\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\right|\to 0\quad (b\to+\infty).$

Look, this is the same quantity that we started with: because the integral is a positive and increasing function of $\sigma$,

$\begin{split} \sup_{\sigma} \left|\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\right|&= \lim_{\sigma\to+\infty}\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\\&= \lim_{\sigma\to+\infty}P(X>b|\sigma). \end{split}$

If we assume that $\forall b\:\lim_{\sigma\to+\infty}P(X>b|\sigma)=0$, then the integral converges uniformly, we can bring the limit under the integral, and prove that indeed, $\forall a\:\lim_{\sigma\to+\infty}P(X>a|\sigma)=0$.

But there is no reason to believe that the limit is zero, and that explains why our calculation yielding 0 was wrong: the integral does not converge uniformly.

The actual calculation of $\lim_{\sigma\to+\infty}P(X>a|\sigma)$ is simple: because $\forall a>0\:\lim_{\sigma\to+\infty}P(-a\leq X\leq a|\sigma)=0$ (thanks to uniform convergence!), all the probability mass goes to infinity, and because of the symmetry, each tail gets a half of it; so

$\lim_{\sigma\to+\infty}P(X>a|\sigma)=1/2.$

But I thought it would be instructive to investigate why the naive calculation did not work, so here we are.