Orthogonality & Uncorrelatedness

Recently I was pondering over orthogonality... a simple concept in the domain of geometry, characterizing two vectors in Euclidian space with 90° angle spanned between them. The orthogonality means that the inner product of two vectors has to be equal to zero.

Our TiU Econometrics grad course is drawing heavily on this notion, since it proves very handy in explaining mechanics underlying the Ordinary Least Squares (OLS) regression. And that's how I got tangled in the concept.

So without further ado, here's a small puzzle:

Advanced econometrics textbooks equate the orthogonality with uncorrelatedness. The two concepts are hence considered equivalent,

Othogonal <=> uncorrelated.

However, we could easily come up with examples which break one part of the equivalence whilst sustaining the other. A straightforward illustration is a pair of individual-specific dummies. These are orthogonal by construction, however negatively correlated with each other!

Similarly, the opposite case of the equivalence violation could arise by taking again an individual-specific dummy vector, and a vector of constant terms. This pair of vectors is uncorrelated, however when plotted into the Euclidean space, the angle between the two is just 45°!

So, that is quite confusing, right? Well fear not - we do not have to lose faith in OLS. It turns out that the notion of orthogonality in geometry and in econometrics are actually two slightly different beasts. To satisfy the textbook equivalence of uncorrelatedness, the geometrical orthogonality has to be applied not to the original vectors, but to their de-meaned forms! The orthogonality of the original vectors has in most cases no consequence for uncorrelatedness of a & b.

This is in fact very natural, since correlation is based on covariance, which is a result of integration over a product of de-meaned realizations of two random variables.

The culprit of this semantic difference between the geometric and econometric interpretation of orthogonality is actually the constant term, which acts as a de-meaning agent in the context of OLS regression. But since the constant is almost always present in the econometric models, the de-meaning becomes somewhat implicit. The econometric orthogonality turns into something what I would probably call "centered orthogonality".

This identification has also straightforward implication for the inner product of the original vectors. These two vectors are (econometrically) orthogonal, when their inner product is equal to n-times the product of their means, with n being the number of their scalar components.



Oh, the semantics...