Refresher: Variance, Covariance, Correlation

Means, Variances and Standard Deviations

The mean is a measure of central tendency for a distribution. To get the mean, we add up all the numbers and divide by how many numbers we have.

It is one way to get “typical” value for a set of numbers, and is the most commonly what we are meaning when we use the word “average”.
One way of thinking of the mean is as the balancing point on a see-saw. It’s the value that balances out the all the deviations to the observed values. In Figure 1, the mean is 4 - the point at which all positive deviations sum to the same as all the negative deviations.

Figure 1: The mean is the point that balances the weight of the data. This means that it is sensitive to outliers. The one datapoint at 9 (+5 deviation) carries the same weight as two observations at 1 (-3 deviation) and 2 (-2 deviation) combined!

If the mean is trying to get the middle of the see-saw, variance is a way of capturing its width. It takes all the deviations, squares them (because we don’t care about the direction), and then divides by the number of deviations (minus 1, because math!1). So variance gives us the typical squared distance from an observation to the mean.

Because no one really works in squared units, we often prefer to use the standard deviation, which simply takes the square root of the variance. In that way, it provides an estimate of the typical distance (in the original measured units, not squared) from an observation to the mean.

Ultimately, variance and standard deviation capture the scale of a distribution (how spread out it it is), whereas the mean captures its location (where the centre is).

Covariance and Correlation

When we move from one variable to two, each variable has a mean and a variance, but we also want to describe how the two variables relate to one another (how they ‘co-relate’):

Do higher values on X co-occur with higher values on Y? or with lower values on Y? or does the value of X give us no information about the value of Y? Covariance captures just that idea - how do two variables vary together?

  • If values of X are generally high when values of Y are also high, then the two variables have positive covariance.
  • If values of X are generally low when values of Y are high, then the two variables have negative covariance.
  • If values of X are don’t pattern with values of Y, then the two items have near-zero covariance.

Covariance works by looking at the deviations from the means on both variables at the same time. In Figure 2, if an observation is above the mean on both, it is in the top right quadrant. If it is below the mean on both, it is in the bottom left. If it is above the mean on one but below the mean on the other, then it will be in one of the other ones.

Taking all of these together, calculating covariance is like calculating the average area of all the coloured squares (red = positive, blue = negative) shown in Figure 2.

Figure 2: Covariance, animated.

One thing about covariance is that it is sensitive to the units of measurement. The covariance between height-in-centimeters and weight-in-grams will be a much bigger number than the covariance between height-in-meters and weight-in-kilograms.

Correlation solves this issue, and you can think of correlation as a standardised covariance. It has a scale from negative one to one, on which the distance from zero indicates the strength of the relationship. Just like covariance, positive/negative values reflect the nature of the relationship. Another way to think about correlation is that it is covariance but where we standardise both variables first, meaning that we have them on the same scale (mean of zero, standard deviation of one), as in Figure 3.

Figure 3: Correlation, animated.

covariance/correlation matrices

  • Variance & Standard Deviation: How spread out a single variable is
  • Covariance & Correlation: How the spread of two variables moves together

What about when we have lots and lots of variables, and we want to see how they all covary? This is where the covariance/correlation matrix comes in. It presents all the covariances or correlations between all the different pairs of variables.

When we give functions like cov() and cor() a dataframe of numeric variables, it will return a square matrix. So if we are looking at a correlation matrix for 5 variables (x,y,z,w,v), then each entry would show:

x y z w v
x cor(x,x) cor(x,y) cor(x,z) cor(x,w) cor(x,v)
y cor(y,x) cor(y,y) cor(y,z) cor(y,w) cor(y,v)
z cor(z,x) cor(z,y) cor(z,z) cor(z,w) cor(z,v)
w cor(w,x) cor(w,y) cor(w,z) cor(w,w) cor(w,v)
v cor(v,x) cor(v,y) cor(v,z) cor(v,w) cor(v,v)

An actual example would look like:

     x    y    z    w    v
x 1.00 0.01 0.47 0.81 0.42
y 0.01 1.00 0.01 0.01 0.00
z 0.47 0.01 1.00 0.52 0.27
w 0.81 0.01 0.52 1.00 0.46
v 0.42 0.00 0.27 0.46 1.00

There are a couple things to notice here:

  1. the order doesn’t matter - cor(x,y) and cor(y,x) are the same!
  2. the diagonals are the correlation of a variable with itself. This will be 1, because a variable is always perfectly correlated with itself. If we were looking at a covariance matrix, these would be the variance of each variable

When we work with lots of variables, data might be missing in lots of different places.

In the dataset below, only the last three rows have no missing NAs in them. And looking down the columns, every variable has at least one NA in it somewhere:

somedata
# A tibble: 8 × 5
      x     y     z     w     v
  <dbl> <dbl> <dbl> <dbl> <dbl>
1    NA     8     3     4     2
2     6    NA     7     7     8
3     3     2    NA     3     5
4     9     6     3    NA     3
5     2     8     6     7    NA
6     7     3     6     9     3
7     3     6     2     3     5
8     3     2     4     6     6

Because every variable contains an NA, the correlations in the correlation matrix created by cor() will not work:

cor(somedata)
   x  y  z  w  v
x  1 NA NA NA NA
y NA  1 NA NA NA
z NA NA  1 NA NA
w NA NA NA  1 NA
v NA NA NA NA  1

An alternative is to specify that we want only the complete observations (the rows where there is no missing data on any of the variables).
This code will essentially calculate correlations on the last three rows of the dataset:

cor(somedata, use = "complete.obs")
       x       y      z      w       v
x  1.000 -0.2774  0.866  0.866 -0.9449
y -0.277  1.0000 -0.721 -0.721 -0.0524
z  0.866 -0.7206  1.000  1.000 -0.6547
w  0.866 -0.7206  1.000  1.000 -0.6547
v -0.945 -0.0524 -0.655 -0.655  1.0000

Another alternative is to specify pairwise complete observations. What this means is that for each correlation in the matrix, it will use the complete observations for that pair of variables. For example, the correlation between variables x and y will be based on all rows except the rows 1 and 2. the correlation between x and z will be based on all except rows 1 and 3, and so on.
What is important here is to be transparent about how many observations are being used to calculate each correlation. In the resulting matrix below, the correlation between variable w and v are based on more data than the others.

cor(somedata, use = "pairwise.complete.obs")
        x       y       z       w       v
x  1.0000 -0.0142  0.0369  0.6246 -0.4525
y -0.0142  1.0000 -0.1905 -0.1059 -0.6686
z  0.0369 -0.1905  1.0000  0.8821  0.4791
w  0.6246 -0.1059  0.8821  1.0000  0.0902
v -0.4525 -0.6686  0.4791  0.0902  1.0000

Footnotes

  1. technically it’s because for the \(n\) deviations we have, we actually only have \(n-1\) bits of information, because the deviations (by definition) must all sum to zero. Which means once you tell me any \(n-1\) of the deviations, I can work out the last one↩︎