# Surprising reciprocity

June 2, 2016

I have two correlated random variables, $X$ and $Y$, with zero mean and equal variance. I tell you that the best way to predict $Y$ based on the knowledge of $X$ is $y = a x$. Now, you tell me, what is the best way to predict $X$ based on $Y$?

Your intuition might tell you that if $y = ax$, then $x = y/a$. This is correct most of the time… but not here. The right answer will surprise you.

So what is the best way to predict $Y$ based on $X$ and vice versa? Let’s find the $a$ that minimizes the mean squared error $E[(Y-aX)^2]$:

$E[(Y-aX)^2] = E[Y^2-2aXY+a^2X^2]=(1+a^2)\mathrm{Var}(X)-2a\mathrm{Cov}(X,Y);$

$\frac{\partial}{\partial a}E[(Y-aX)^2] = 2a\mathrm{Var}(X)-2\mathrm{Cov}(X,Y);$

$a=\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}=\mathrm{Corr}(X,Y).$

Notice that the answer, the (Pearson) correlation coefficient, is symmetric w.r.t. $X$ and $Y$. Thus it will be the same whether we want to predict $Y$ based on $X$ or $X$ based on $Y$!

How to make sense of this? It may help to consider a couple of special cases first.

First, suppose that $X$ and $Y$ are perfectly correlated and you’re trying to predict $Y$ based on $X$. Since $X$ is such a good predictor, just use its value as it is ($a=1$).

Now, suppose that $X$ and $Y$ are uncorrelated. Knowing the value of $X$ doesn’t tell you anything about the value of $Y$ (as far as linear relationships go). The best predictor you have for $Y$ is its mean, $0$.

Finally, suppose that $X$ and $Y$ are somewhat correlated. The correlation coefficient is the degree to which we should trust the value of $X$ when predicting $Y$ versus sticking to $0$ as a conservative estimate.

This is the key idea—to think about $a$ in $y=ax$ not as a degree of proportionality, but as a degree of “trust”.