Assumptions for LMs

Why do we have to make assumptions?

We can think of a linear model as a way to describe the process of how our data was generated.

Models are simplifications of processes going on in the world. In order to simplify these processes, a model will specify or “hard-code” certain aspects of the data generating process, and those aspects cannot be changed. Unchangeable aspects of a model’s data generating process are things that we have to assume are true about the data and relationships that we’re modelling, in order to justify a linear model as the most appropriate tool.

Assumptions are not like significance tests: it’s not the case that the assumption is either “accepted” or “rejected”. Our decision process is blurrier than that. When evaluating assumptions, we are asking ourselves: Are we satisfied that the unchangeable aspects of the model are reasonable enough simplifications, so that the model will still give us reasonable parameter estimates?

An acronym for the four assumptions we must make for a linear model is “LINE”:

  • L = Linearity of association
  • I = Independence of errors
  • N = Normality of errors
  • E = Equal variance of errors

Example data

We’ll reuse the recall accuracy data from the Block 1 flash cards.

recalldata <- read_csv('https://uoepsy.github.io/data/recalldata.csv')
head(recalldata)
# A tibble: 6 x 4
  ppt   recall_accuracy recall_confidence   age
  <chr>           <dbl>             <dbl> <dbl>
1 ppt_1              72              66.6    72
2 ppt_2              66              47.1    35
3 ppt_3              47              43.8    48
4 ppt_4              84              58.9    52
5 ppt_5              84              75.1    46
6 ppt_6              58              53.5    41

We’ll fit the same old multiple regression model.

recall_mdl <- lm(
  recall_accuracy ~ recall_confidence + age,
  data = recalldata
)

L = Linearity of association

A linear model can only model associations between predictor and outcome in terms of a straight line. Within the basic linear model machinery, this cannot be changed, and we just have to accept it. In other words, we have to assume it’s true.

If an association doesn’t follow a straight line, then a linear model cannot accurately capture it.

Think of predicting height as a function of age, or height ~ age, for example. Height rises steeply when ages are small and then eventually tapers off (and may even start to decrease in old age!).

If we used a linear model to model height ~ age, then our model will give us the best straight line it can.

To be satisfied with a linear model of that data, we have to assume that the best possible straight line is a good representation of the data.

Check linearity using component–residual plots

  • Component: The association between one particular predictor and the outcome, with all the other predictors held constant. Think of this as each predictor’s individual contribution to the overall linear model.
  • Residual: The differences between the line the model predicts and the actual observed data points.

We use the function crPlots() from the library car.

library(car)
crPlots(recall_mdl)

The x axis shows each predictor across its full range. The y axis shows that predictor’s “component” (what that particular predictor contributes to the model’s estimates), plus each data point’s residual.

The dashed blue line shows what we’d expect if the associations were perfectly linear. The wobbly pink line shows how the associations actually look. Are we satisfied with how closely the two lines match?

In this case, I’d say we are.

If we weren’t, what would we do? One option would be to transform the outcome or predictor variable, fit the model, and then try again. Sometimes taking the logarithm of the predictor or the outcome can make the linearity of association in our data much better than it was with the untransformed version!

For Block 3: A different check for interaction models

When there is an interaction in the model, assessing linearity becomes difficult. In fact, crPlots() will not work. Instead, you’ll need to use a residuals-vs-fitted plot (see the “Assumption of equal variance” section below) and see how closely the wobbly blue line matches the horizontal black dashed line.

I = Independence of errors

A linear model can only model errors \(\epsilon\) as residuals that are independent from one another. Within the basic linear model machinery, this cannot be changed.

To be satisfied with a simple linear model of our data, we have to assume that independent residuals are a good representation of the data.

Put more simply: we don’t want the data points to have any underlying relationships or groupings that cause them to influence one another.

One common source of non-independence are data points that come from the same source (e.g., in repeated-measures data, the same person will contribute multiple observations). In DAPR3, you’ll learn how to deal with this scenario.

For DAPR2, as long as each individual data point comes from a separate source (e.g., each person only contributes one observation to the dataset), we can assume that errors in our data will be independent.

N = Normality of errors

A linear model can only model errors \(\epsilon\) as residuals that are normally distributed. Within the basic linear model machinery, this cannot be changed.

To be satisfied with a linear model of our data, we have to assume that normally-distributed residuals are a good representation of the data.

We can check how willing we are to make this assumption visually, using Q-Q plots of residuals.

A Q-Q plot (Quantile-Quantile plot) shows how closely a model’s residuals (represented by data points) match the ideal normal distribution (represented by the dashed diagonal line). The closer the match, the more normally-distributed the residuals are.

plot(recall_mdl, which = 2)

This Q-Q plot shows some deviations from normality at each extreme, especially for data points 14, 17, and 12, which have larger positive residuals than we’d expect if the data were perfectly normally distributed.

That said: No data in real life is really ever perfectly normally distributed. Some divergence is inevitable. So it’s a judgement call: is the divergence big enough that you wouldn’t want to assume normality of errors?

In this case, I will try transforming the data to see if we can get a better, more normal, distribution of residuals. I’ll explain this, and we’ll go through all checks for that model, in the “Transform outcome data and try again” section below.

E = Equal variance of errors

A linear model can only model errors as residuals that have equal variance. Within the basic linear model machinery, this cannot be changed.

To be satisfied with a linear model of our data, we have to assume that residuals with equal variance are a good representation of the data.

Another term for “equal variance” is “homoscedasticity”. Another term for “unequal variance” is “heteroscedasticity”.

We can check how willing we are to make this assumption using residual-vs-fitted plots. A “residuals vs fitted” plot shows:

  • x axis: the predicted outcome values (aka the “fitted values” across their whole range).
  • y axis: the residuals of each data point.

Here’s an example of what the residuals-vs-fitted plots looks like when variance is equal (this is what we want to see).

The vertical spread of data points is pretty even across both/all values of the predictor.

Now, here’s an example of what data looks like when variance is not equal (this is what we want to avoid).

The most common and classic version of unequal variance is the funnel/fan/cone shape that appears in the plots above. There, the vertical spread of data points is not even across both/all values of the predictor.


Let’s check the variance of our model.

plot(recall_mdl, which = 1)

Here, the vertical spread of data points seems fairly even. Based on this, I’m happy to make the assumption of equal variance.

For Block 3: Linearity assumption for interaction models

To check linearity in interaction models, since component-residual plots are impossible, you can look at the wobbly red line in the residual-vs-fitted plot. The closer the wobbly red line is to the dashed horizontal line at y = zero, the more linear the modelled associations.

Transform outcome data and try again

Because the Q-Q plot for the assumption of normality looks a little suspicious to me, I’m going to try log-transforming the outcome variable to see if that improves the situation.

recalldata <- recalldata |>
  mutate(
    logacc = log(recall_accuracy)
  )

What’s changed?

p1 <- recalldata |>
  ggplot(aes(x = recall_accuracy)) +
  geom_density() +
  labs(title = 'Raw recall accuracy')

p2 <- recalldata |>
  ggplot(aes(x = logacc)) +
  geom_density() +
  labs(title = 'Log-transformed accuracy')

p1 + p2

The distribution has a similar shape, it’s slightly less right-skewed now. A big difference is the scale of the x axis. Instead of measuring the outcome in points, we’re now measuring the outcome in log units. This will impact all our coefficient interpretations.

Fit a new model with the log-transformed version of recall accuracy as the outcome:

recall_mdl2 <- lm(
  logacc ~ recall_confidence + age,
  data = recalldata
)

Check linearity of association

crPlots(recall_mdl2)

Linearity still looks reasonable to me.

Check normality of errors

plot(recall_mdl2, which = 2)

The match to the ideal diagonal line is better now than it was before! So if the equal variance assumption still holds up, then I would interpret and report this model instead of the original one.

Check equal variance

plot(recall_mdl2, which = 1)

Well, now it looks a little bit like we’ve got a backwards funnel kind of shape in the residuals-vs-fitted plot. The data points on the left of the plot have a much wider vertical spread than the data points on the right of the plot.

It is definitely tough to tell, since there’s so little data. But I’m not too sure about this one…

I’d say we have two options here:

  • Report whichever version of the model you feel better about, explaining that some of the assumptions look a bit borderline; or
  • Because the assumptions that we’re not totally willing to make both concern the model’s errors, a reasonable solution might be to bootstrap the linear model.

A data transformation involves the replacement of a variable (e.g., \(y\)) by a function of that variable in order to change the shape of a distribution or association (e.g., to help reduce skew). We can transform the outcome variable prior to fitting the model, using something such as log(y) or sqrt(y). This will sometimes allow us to estimate a model for which our assumptions are satisfied.

Some of the most common (not an exhaustive list) transformations are:

  • Log (log(y)): Often used for reducing right skewness. Note, this transformation cannot be applied to zero or negative values (make sure to check your data!)
  • Square root (sqrt(y)): Also often used for reducing right skewness. This transformation can be applied to zero values (but not negative), and is commonly applied to count data
Figure 1: A model of a transformed outcome variable can sometimes avoid violations of assumptions that arise when modeling the outcome variable directly. Data from https://uoepsy.github.io/data/trouble1.csv

The major downside of this is that we are no longer modelling \(y\), but some transformation \(f(y)\) (\(y\) with some function \(f\) applied to it). Interpretation of the coefficients changes accordingly, such that we are no longer talking in terms of changes in y, but changes in \(f(y)\). When the transformation function used is non-linear (see the Right-Hand of Figure 2) a change in \(f(y)\) is not the same for every \(y\).

Figure 2: The log transformation is non-linear