Write Up & Block 3 Recap

Learning Objectives

At the end of this lab, you will:

1. Understand how to write-up and provide interpretation of a 4x2 factorial ANOVA¹

What You Need

1. Be up to date with lectures
2. Have completed Labs 1-4

Required R Packages

Remember to load all packages within a code chunk at the start of your RMarkdown file using library(). If you do not have a package and need to install, do so within the console using install.packages(" "). For further guidance on installing/updating packages, see Section C here.

For this lab, you will need to load the following package(s):

• tidyverse
• psych
• kableExtra
• sjPlot
• interactions
• emmeans

Lab Data

You can download the data required for this lab here or read it in via this link https://uoepsy.github.io/data/laptop_vs_longhand.csv

Section A: Write-Up

In this lab you will be presented with the output from a statistical analysis, and your job will be to write-up and present the results. We’re going to use two simulated datasets based on a paper (the same two that you have worked on in lectures this week) concerning academic outcomes, student/class characteristics, and attendance.

The aim in writing should be that a reader is able to more or less replicate your analyses without referring to your R code. This requires detailing all of the steps you took in conducting the analysis.

The point of using RMarkdown is that you can pull your results directly from the code. If your analysis changes, so does your report!

Make sure that your final report doesn’t show any R functions or code. Remember you are interpreting and reporting your results in text, tables, or plots, targeting a generic reader who may use different software or may not know R at all. If you need a reminder on how to hide code, format tables, etc., make sure to review the rmd bootcamp.

Important - Write-Up Examples & Plagiarism

The example write-up sections included below are not perfect - they instead should give you a good example of what information you should include within each section, and how to structure this. For example, some information is missing (e.g., description of data checks, interpretation of descriptive statistics), some information could be presented more clearly (e.g., variable names in tables, table/figure titles/captions, and rationales for choices), and writing could be more concise in places (e.g., discussion section is quite long).

Further, you must not copy any of the write-up included below for future reports - if you do, you will be committing plagiarism, and this type of academic misconduct is taken very seriously by the University. You can find out more here.

Study Overview

Research Aim

Explore the associations among study time and note-taking medium on test scores.

Research Questions

• RQ1: Do differences in test scores between study conditions differ by the note-taking medium used?
• RQ2: Are there differences in test scores between participants when comparing pairs of study and note-taking conditions? If so, what are these specific differences?

Note Taking: Data Codebook

Description

The data used for this write-up exercise are simulated, drawing on a research paper that explored the association between student note taking, and performance when answering different types of questions. The simulated data are loosely based on the findings of this work, and acted to expand upon the methods and results reported in the paper:

Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological Science, 25(6), 1159–1168. https://doi.org/10.1177/0956797614524581

In the current study, participants were invited to take part in a study investigating the the medium of note taking and study time on test scores. The sample comprised of 160 students who took notes on a lecture via one of two mediums - either on a laptop or longhand (i.e., using pen and paper). After watching the lecture and taking notes, they then randomly allocated to one of four study time conditions, either engaging in no, minimal, moderate, or extensive study of the notes taken on their assigned medium. After engaging in study for their allocated time, participants took a test on the lecture content. The test involved a series of questions, where participants could score a maximum of 100 points.

Data Dictionary

The data in laptop_vs_longhand.csv contain three attributes collected from a simulated sample of \(n=160\) hypothetical individuals, and includes:

Variable	Description
test_score	Test Score (0-100)
medium	Medium of note-taking (levels = Longhand, Laptop)
study	Study time (levels = No, Minimal, Moderate, Extensive)

Preview

The first six rows of the data are:

test_score	medium	study
47.47812	Laptop	No
50.41772	Laptop	No
49.88763	Laptop	No
48.47961	Laptop	No
48.44368	Laptop	No
48.02715	Laptop	No

Setup

Setup

1. Create a new RMarkdown file
2. Load the required package(s)
3. Read the laptop_vs_longhand dataset into R, assigning it to an object named notes

Solution

library(tidyverse)
library(psych) 
library(kableExtra)
library(sjPlot)
library(interactions)
library(emmeans)

#read in data
notes <- read_csv("https://uoepsy.github.io/data/laptop_vs_longhand.csv")

Analysis Code

Try to answer the research question above without referring to the provided analysis code below, and then check how your script matches up - is there anything you missed or done differently? If so, discuss the differences with a tutor - there are lots of ways to code to the same solution!

Provided Analysis Code

######Step 1 is always to read in the data, then to explore, check, describe, and visualise it.

#check coding of variables - are they coded as they should be?
str(notes)

spc_tbl_ [160 × 3] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
 $ test_score: num [1:160] 47.5 50.4 49.9 48.5 48.4 ...
 $ medium    : chr [1:160] "Laptop" "Laptop" "Laptop" "Laptop" ...
 $ study     : chr [1:160] "No" "No" "No" "No" ...
 - attr(*, "spec")=
  .. cols(
  ..   test_score = col_double(),
  ..   medium = col_character(),
  ..   study = col_character()
  .. )
 - attr(*, "problems")=<externalptr> 

head(notes)

# A tibble: 6 × 3
  test_score medium study
       <dbl> <chr>  <chr>
1       47.5 Laptop No   
2       50.4 Laptop No   
3       49.9 Laptop No   
4       48.5 Laptop No   
5       48.4 Laptop No   
6       48.0 Laptop No   

#check for NAs - none in dataset, so no missing values
table(is.na(notes))

FALSE 
  480 

#make variables factors
notes <- notes %>%
    mutate(medium = as_factor(medium),
           study = as_factor(study))

#create descriptives table
descript <- notes %>% 
    group_by(study, medium) %>%
   summarise(
       M_Score = round(mean(test_score), 2),
       SD_Score = round(sd(test_score), 2),
       SE_Score = round(sd(test_score)/sqrt(n()), 2),
       Min_Score = round(min(test_score), 2),
       Max_Score = round(max(test_score), 2)
    )
descript

# A tibble: 8 × 7
# Groups:   study [4]
  study     medium   M_Score SD_Score SE_Score Min_Score Max_Score
  <fct>     <fct>      <dbl>    <dbl>    <dbl>     <dbl>     <dbl>
1 No        Laptop      48.1        2     0.45      44.9      51.2
2 No        Longhand    51.0        2     0.45      48.0      54.9
3 Minimal   Laptop      55.6        2     0.45      52.1      60.1
4 Minimal   Longhand    60.9        2     0.45      57.8      65.3
5 Moderate  Laptop      59.3        2     0.45      55.8      63.0
6 Moderate  Longhand    80.7        2     0.45      76.3      84.6
7 Extensive Laptop      61.2        2     0.45      56.9      64.3
8 Extensive Longhand    90.6        2     0.45      86.8      94.3

#boxplot
p1 <- ggplot(data = notes, aes(x = study, y = test_score, color = medium)) + 
  geom_boxplot() + 
    ylim(0,100) +
    labs(x = "Study Condition", y = "Test Score")
p1

#plot showing the mean score for each condition
# p2 is useful to notice that lines do not run in parallel - suggests interaction
p2 <- ggplot(descript, aes(x = study, y = M_Score, color = medium)) + 
  geom_point(size = 3) +
  geom_linerange(aes(ymin = M_Score - 2 * SE_Score, ymax = M_Score + 2 * SE_Score)) +
  geom_path(aes(x = as.numeric(study)))
p2

######Step 2 is to run your model(s) of interest to answer your research question, and make sure that the data meet the assumptions of your chosen test

#set reference levels
notes$medium <- fct_relevel(notes$medium , "Longhand")
notes$study <- fct_relevel(notes$study , "No")

#build model
notes_mdl <- lm(test_score ~ study*medium, data = notes)

#check assumptions - note should check diagnostics here too!
par(mfrow=c(2,2))
plot(notes_mdl)

par(mfrow=c(1,1))

# look at model output - summary()
summary(notes_mdl)

Call:
lm(formula = test_score ~ study * medium, data = notes)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.3485 -1.4764 -0.1018  1.4039  4.5321 

Coefficients:
                            Estimate Std. Error t value Pr(>|t|)    
(Intercept)                  51.0200     0.4472 114.084  < 2e-16 ***
studyMinimal                  9.8900     0.6325  15.637  < 2e-16 ***
studyModerate                29.6700     0.6325  46.912  < 2e-16 ***
studyExtensive               39.5600     0.6325  62.550  < 2e-16 ***
mediumLaptop                 -2.9000     0.6325  -4.585 9.41e-06 ***
studyMinimal:mediumLaptop    -2.4400     0.8944  -2.728  0.00712 ** 
studyModerate:mediumLaptop  -18.4900     0.8944 -20.672  < 2e-16 ***
studyExtensive:mediumLaptop -26.5200     0.8944 -29.650  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2 on 152 degrees of freedom
Multiple R-squared:  0.9803,    Adjusted R-squared:  0.9794 
F-statistic:  1081 on 7 and 152 DF,  p-value: < 2.2e-16

#table results
tab_model(notes_mdl, 
          pred.labels = c('Intercept', 'Study - Minimal', 'Study - Moderate', 'Study - Extensive', 'Medium - Laptop', 'Study - Minimal : Medium - Laptop', 'Study - Moderate : Medium - Laptop', 'Study - Extensive : Medium - Laptop'),
          title = 'RQ1 - Regression Table for Total Scores Model')

RQ1 - Regression Table for Total Scores Model

	test score
Predictors	Estimates	CI	p
Intercept	51.02	50.14 – 51.90	<0.001
Study - Minimal	9.89	8.64 – 11.14	<0.001
Study - Moderate	29.67	28.42 – 30.92	<0.001
Study - Extensive	39.56	38.31 – 40.81	<0.001
Medium - Laptop	-2.90	-4.15 – -1.65	<0.001
Study - Minimal : Medium - Laptop	-2.44	-4.21 – -0.67	0.007
Study - Moderate : Medium - Laptop	-18.49	-20.26 – -16.72	<0.001
Study - Extensive : Medium - Laptop	-26.52	-28.29 – -24.75	<0.001
Observations	160
R2 / R2 adjusted	0.980 / 0.979

#int model plot
plt_notes_mdl <- cat_plot(model = notes_mdl, 
                  pred = study, 
                  modx = medium, 
                  main.title = "Scores across Study and Medium",
                  x.label = "Study",
                  y.label = "Score",
                  legend.main = "Medium")
plt_notes_mdl

#####Step 3 somewhat depends on the outcomes of step 2. Here, you may need to consider conducting further analyses before writing up / describing your results in relation to the research question. 

#Perform a pairwise comparison of the mean accuracy (as measured by points accrued) across the 2×2 factorial design, making sure to adjust for multiple comparisons. 

m1_emm <- emmeans(notes_mdl, ~study*medium)

pairs_res <- pairs(m1_emm)
pairs_res 

 contrast                               estimate    SE  df t.ratio p.value
 No Longhand - Minimal Longhand            -9.89 0.632 152 -15.637  <.0001
 No Longhand - Moderate Longhand          -29.67 0.632 152 -46.912  <.0001
 No Longhand - Extensive Longhand         -39.56 0.632 152 -62.550  <.0001
 No Longhand - No Laptop                    2.90 0.632 152   4.585  0.0002
 No Longhand - Minimal Laptop              -4.55 0.632 152  -7.194  <.0001
 No Longhand - Moderate Laptop             -8.28 0.632 152 -13.092  <.0001
 No Longhand - Extensive Laptop           -10.14 0.632 152 -16.033  <.0001
 Minimal Longhand - Moderate Longhand     -19.78 0.632 152 -31.275  <.0001
 Minimal Longhand - Extensive Longhand    -29.67 0.632 152 -46.912  <.0001
 Minimal Longhand - No Laptop              12.79 0.632 152  20.223  <.0001
 Minimal Longhand - Minimal Laptop          5.34 0.632 152   8.443  <.0001
 Minimal Longhand - Moderate Laptop         1.61 0.632 152   2.546  0.1847
 Minimal Longhand - Extensive Laptop       -0.25 0.632 152  -0.395  0.9999
 Moderate Longhand - Extensive Longhand    -9.89 0.632 152 -15.637  <.0001
 Moderate Longhand - No Laptop             32.57 0.632 152  51.498  <.0001
 Moderate Longhand - Minimal Laptop        25.12 0.632 152  39.718  <.0001
 Moderate Longhand - Moderate Laptop       21.39 0.632 152  33.821  <.0001
 Moderate Longhand - Extensive Laptop      19.53 0.632 152  30.880  <.0001
 Extensive Longhand - No Laptop            42.46 0.632 152  67.135  <.0001
 Extensive Longhand - Minimal Laptop       35.01 0.632 152  55.356  <.0001
 Extensive Longhand - Moderate Laptop      31.28 0.632 152  49.458  <.0001
 Extensive Longhand - Extensive Laptop     29.42 0.632 152  46.517  <.0001
 No Laptop - Minimal Laptop                -7.45 0.632 152 -11.779  <.0001
 No Laptop - Moderate Laptop              -11.18 0.632 152 -17.677  <.0001
 No Laptop - Extensive Laptop             -13.04 0.632 152 -20.618  <.0001
 Minimal Laptop - Moderate Laptop          -3.73 0.632 152  -5.898  <.0001
 Minimal Laptop - Extensive Laptop         -5.59 0.632 152  -8.839  <.0001
 Moderate Laptop - Extensive Laptop        -1.86 0.632 152  -2.941  0.0717

P value adjustment: tukey method for comparing a family of 8 estimates 

#plot
plot(pairs_res)

The 3-Act Structure

We need to present our report in three clear sections - think of your sections like the 3 key parts of a play or story - we need to (1) provide some background and scene setting for the reader, (2) present our results in the context of the research question, and (3) present a resolution to our story - relate our findings back to the question we were asked and provide our answer.

Act I: Analysis Strategy

Question 1

Attempt to draft a discussion section based on the above research question and analysis provided.

Analysis Strategy - What to Include***

Your analysis strategy will contain a number of different elements detailing plans and changes to your plan. Remember, your analysis strategy should not contain any results. You may wish to include the following sections:

• Very brief data and design description:
  • Give the reader some background on the context of your write-up. For example, you may wish to describe the data source, data collection strategy, study design, number of observational units.
  • Specify the variables of interest in relation to the research question, including their unit of measurement, the allowed range (for Likert scales), and how they are scored. If you have categorical data, you will need to specify the levels and coding of your variables, and what was specified as your reference level and the justification for this choice.
• Data management:
  • Describe any data cleaning and/or recoding.
  • Are there any observations that have been excluded based on pre-defined criteria? How/why, and how many?
  • Describe any transformations performed to aid your interpretation (i.e., mean centering, standardisation, etc.)
• Model specification:
  • Clearly state your hypotheses and specify your chosen significance level.
  • What type of statistical analysis do you plan to use to answer the research question? (e.g., simple linear regression, multiple linear regression, binary logistic regression, etc.)
  • In some cases, you may wish to include some visualisations and descriptive tables to motivate your model specification.
  • Specify the model(s) to be fitted to answer your given research question and analysis structure. Clearly specify the response and explanatory variables included in your model(s). This includes specifying the type of coding scheme applied if using categorical data.
  • * Specify the assumption and diagnostic checks that you will conduct. Specify what plots you will use, and how you will evaluate these.

*** Note, given time constraints in labs, we have not included any reference to diagnostic checks in this write-up example - you would be expected to include this in your report. You can find more information on diagnostic checks in the S1 Week 9 Lab and S1 Week 9 Lectures.

As noted and encouraged throughout the course, one of the main benefits of using RMarkdown is the ability to include inline R code in your document. Try to incorporate this in your write up so you can automatically pull the specified values from your code. If you need a reminder on how to do this, see Lesson 4 of the Rmd Bootcamp.

Example Write-Up of Analysis Strategy Section

The notes dataset contained information on 160 participants who took part in a study concerning the role(s) of note taking and study time on test scores. Participants took notes on a lecture via one of two mediums - either on a laptop \((n = 80)\) or long-hand using pen and paper \((n = 80)\). They were then randomly allocated to one of four study time conditions, either engaging in no \((n = 40)\), minimal \((n = 40)\), moderate \((n = 40)\), or extensive \((n = 40)\) study of the notes taken on their assigned medium. Participants then answered a series of questions based on the lecture content. The maximum score was 100, where higher scores reflected better test performance.

The aim of this report was to address the following two research questions:

1. Do differences in test scores between study conditions differ by the note-taking medium used?
2. Are there differences in test scores between participants when comparing pairs of study and note-taking conditions? If so, what are these specific differences?

All participant data was complete, and test scores within range i.e., 0-100. Categorical variables were coded as factors, where ‘No’ was designated as the reference level for study condition, and ‘Longhand’ as the reference level for medium.

To investigate whether study condition (No, Minimal, Moderate, Extensive) and note-taking medium (Longhand, Laptop) interacted to influence test scores, a two-way ANOVA model was used. Effects were considered statistically significant at \(\alpha = .05\). Using dummy coding, the following model specification was used:

\[ \begin{align} \text{Test Score} &= \beta_0 \\ &+ \beta_1 \cdot S_\text{Minimal} \\ &+ \beta_2 \cdot S_\text{Moderate} \\ &+ \beta_3 \cdot S_\text{Extensive} \\ &+ \beta_4 \cdot M_\text{Laptop} \\ &+ \beta_5 \cdot (S_\text{Minimal} \cdot M_\text{Laptop}) \\ &+ \beta_6 \cdot (S_\text{Moderate} \cdot M_\text{Laptop}) \\ &+ \beta_7 \cdot (S_\text{Extensive} \cdot M_\text{Laptop}) \\ &+ \epsilon \end{align} \]

where we tested whether there was a significant interaction between study condition and note-taking medium:

\[ H_0: \text{All}~~ \beta_j = 0 ~\text{(for j = 5, 6, 7)} \]

\[ H_1: \text{At least one}~ \beta_j \neq \text{(for j = 5, 6, 7)} \]

As we were using between-subjects datasets, we assumed independence of our error terms. We assumed linearity as all predictor variables were categorical. Equal variances was assessed via partial residual plots (residuals should be evenly spread across the range of fitted values, where the spread should be constant across the range of fitted values), and normality was assessed via a qqplot of the residuals (points should follow along the diagonal line).

To address RQ2 and explore if there are pairwise differences and determine which conditions significantly differed from each other, we will conduct a series of pairwise comparisons. Since we are interested in all pairwise comparisons of means, we will apply a Tukey correction.

Act II: Results

Question 2

Attempt to draft a results section based on your detailed analysis strategy and the analysis provided.

Results - What To Include***

The results section should follow from your analysis strategy. This is where you would present the evidence and results that will be used to answer the research questions and can support your conclusions. Make sure that you address all aspects of the approach you outlined in the analysis strategy (including the evaluation of assumptions and diagnostics***).

In this section, it is useful to include tables and plots to clearly present your findings to your reader. It is important, however, to carefully select what is the key information that should be presented. You don’t want to overload the reader with unnecessary or duplicate information, and you also want to save space in case there is a page limit. Make use of figures with multiple panels where you can.

As a broad guideline, you want to start with the results of any exploratory data analysis, presenting tables of summary statistics and exploratory plots. You may also want to visualise associations between/among variables and report covariances or correlations. Then, you should move on to the results from your model.

Example Write-Up of Results Section

Descriptive statistics are displayed in Table 1.

Table 1: Descriptive Statistics

study	medium	M_Score	SD_Score	SE_Score	Min_Score	Max_Score
No	Longhand	51.02	2	0.45	48.02	54.92
No	Laptop	48.12	2	0.45	44.86	51.16
Minimal	Longhand	60.91	2	0.45	57.84	65.29
Minimal	Laptop	55.57	2	0.45	52.07	60.10
Moderate	Longhand	80.69	2	0.45	76.34	84.64
Moderate	Laptop	59.30	2	0.45	55.77	62.98
Extensive	Longhand	90.58	2	0.45	86.82	94.32
Extensive	Laptop	61.16	2	0.45	56.93	64.32

In the No and Minimal study conditions, there did not appear to be differences in test score between those using a laptop or longhand when note-taking. However, those in the longhand note-taking condition scored higher than those using laptops in the moderate and extensive study conditions. This suggested that there may be an interaction (see Figure 1).

Figure 1: Association between Test Score and Medium / Study Conditions

Test scores were analysed with a 4 (study: no vs minimal vs moderate vs extensive) \(\times\) 2 (medium: longhand vs laptop) between-subjects ANOVA.

The model met assumptions of linearity and independence (see Appendix A, top left panel of Figure 4; residuals were randomly scattered with a mean of zero and there was no clear dependence), homoscedasticity (see Appendix A, bottom left panel of Figure 4; there was a constant spread of residuals), and normality (see Appendix A, top right panel of Figure 4; the QQplot showed very little deviation from the diagonal line).

There was a significant interaction between study condition and note-taking medium \(F(7, 152) = 1081, p < . 001\). Full regression results, including 95% Confidence Intervals, are shown in Table 2.

Table 2: RQ1 - Regression Table for Total Scores Model

	test score
Predictors	Estimates	CI	p
Intercept	51.02	50.14 – 51.90	<0.001
Study - Minimal	9.89	8.64 – 11.14	<0.001
Study - Moderate	29.67	28.42 – 30.92	<0.001
Study - Extensive	39.56	38.31 – 40.81	<0.001
Medium - Laptop	-2.90	-4.15 – -1.65	<0.001
Study - Minimal : Medium - Laptop	-2.44	-4.21 – -0.67	0.007
Study - Moderate : Medium - Laptop	-18.49	-20.26 – -16.72	<0.001
Study - Extensive : Medium - Laptop	-26.52	-28.29 – -24.75	<0.001
Observations	160
R2 / R2 adjusted	0.980 / 0.979

As displayed in Figure 2, results suggested that the difference in scores did differ significantly across the note-taking medium and study conditions, where scores differences appeared to get larger as the period of study increased (i.e., there was little difference between longhand and laptop note-taking conditions when participants engaged in no study, but the gap in test scores seemed to grow as the length of study time increased).

Figure 2: Interaction Plot

To explore the interaction further, and address RQ2, pairwise comparisons were conducted. Tukey’s Honestly Significant Difference comparisons (see Figure 3) indicated that the vast majority of pairwise comparisons were statistically significant. There were only three pairs of comparisons that were not - those in the minimal longhand condition did not significantly differ from those in either the moderate laptop (95% CI [-0.33 - 3.55]) or extensive laptop (95% CI [-2.19 - 1.69]) conditions; and there was no difference between those in the laptop condition who studied either for a moderate or extensive period of time (95% CI [-3.80 - 0.08]). Overall, test differences appeared more pronounced when using the longhand note-taking medium across study conditions.

Figure 3: Tukey HSD Pairwise Comparisons

Act III: Discussion

Question 3

Attempt to draft a discussion section based on your results and the analysis provided.

Discussion - What To Include

In the discussion section, you should summarise the key findings from the results section and provide the reader with a few take-home sentences drawing the analysis together and relating it back to the original question.

The discussion should be relatively brief, and should not include any statistical analysis - instead think of the discussion as a conclusion, providing an answer to the research question(s).

Example Write-Up of Discussion Section

The findings indicated that, in general, people who engaged in no study regardless of note-taking medium performed at chance level. Overall, participants who had taken notes with laptops performed worse on tests regardless of study time in comparison to those who took notes by hand. Our results led us to reject the null hypothesis, as the results indicated that the association between study condition and note-taking medium did interact to influence test scores. The direction of this association was somewhat surprising, as it suggested that laptop use can negatively influence performance on educational tests, and that engaging in hours of study is not enough to mitigate these effects.

Assumptions & Diagnostics Appendix

Question 4

Given we want to keep the report as concise as possible, we may wish to utilize the appendix to present assumption and diagnostic plots. We must however ensure that:

• We have described what assumptions we will check in the analysis strategy, including how we will evaluate them
• We have summarized the evaluations of our assumptions and diagnostic checks in the results section of the main report
• We have accurately refereed to the figures and tables labels presented in the appendix in the main report (if we don’t refer to them, the reader won’t know what they are relevant to!)

Example Assumptions & Diagnostics Appendix

Appendix A

Figure 4: Assumption Plots

Section B: Weeks 1-5 Recap

In the second part of the lab, there is no new content - the purpose of the recap section is for you to revisit and revise the concepts you have learned over the last 4/5 weeks.

Before you expand each of the boxes below, think about how comfortable you feel with each concept.

Specifying Interaction Models

Formula:

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \cdot x_2) + \epsilon \] In R:

There are basically two pieces of information that we need to pass to the lm() function:

1. The formula: The regression formula should be specified in the form y ~ x where \(y\) is the dependent variable (DV), \(x_1\) the first independent variable (IV1), and \(x_2\) the second independent variable (IV2).

2. The data: Specify which dataframe contains the variables specified in the formula.

• run interaction model via lm() function

model_name <- lm(DV ~ IV1*IV2, data = data_name)

OR

model_name <- lm(data_name$DV ~ data_name$IV1 + data_name$IV2 + data_name$IV1:data_name$IV2)

OR

model_name <- lm(data_name$DV ~ data_name$IV1 + data_name$IV2 + data_name$IV1*data_name$IV2)

Note

See S2 Week 1 Lectures, S2 Week 1 Lab, S2 Week 1 Lab, S2 Week 2 Lectures, S2 Week 2 Lab, S2 Week 3 Lectures, S2 Week 3 Lab, S2 Week 4 Lectures, and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Interpreting Coefficients

Interpreting coefficients for A and B in the presence of an interaction A:B

When you include an interaction between \(x_1\) and \(x_2\) in a regression model, you are estimating the extent to which the effect of \(x_1\) on \(y\) is different across the values of \(x_2\).

What this means is that the effect of \(x_1\) on \(y\) depends on/is conditional upon the value of \(x_2\).
(and vice versa, the effect of \(x_2\) on \(y\) is different across the values of \(x_1\)).
This means that we can no longer talk about the “effect of \(x_1\) holding \(x_2\) constant”. Instead we can talk about a marginal effect of \(x_1\) on \(y\) at a specific value of \(x_2\).

When we fit the model \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \cdot x_2) + \epsilon\) using lm():

• the parameter estimate \(\hat \beta_1\) is the marginal effect of \(x_1\) on \(y\) where \(x_2 = 0\)
  
• the parameter estimate \(\hat \beta_2\) is the marginal effect of \(x_2\) on \(y\) where \(x_1 = 0\)
  

N.B. Regardless of whether or not there is an interaction term in our model, all parameter estimates in multiple regression are “conditional” in the sense that they are dependent upon the inclusion of other variables in the model. For instance, in \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \epsilon\) the coefficient \(\hat \beta_1\) is conditional upon holding \(x_2\) constant.

Interpreting the interaction term A:B

The coefficient for an interaction term can be thought of as providing an adjustment to the slope.

Let’s take the following model as an example:

lm(formula = y ~ x1 * x2, data = df)
...

...
Coefficients:
             Estimate Std. Error t value Pr(>|t|)  
(Intercept)  ...        ....       ...      ...  
x1           ...        ....       ...      ...  
x2           ...        ....       ...      ...  
x1:x2        ...        ....       ...      ...  
---

These coefficients can be interpreted, in turn as:

Coefficient	Interpretation
(Intercept)	the estimated \(y\) when all predictors (\(x_1\) and \(x_2\)) are zero is [estimate]
x1	when \(x_2\) is zero, a 1 unit increase in \(x_1\) is associated with a [estimate] change in \(y\)
x2	when \(x_1\) is zero, a 1 unit increase in \(x_2\) is associated with a [estimate] change in \(y\).
x1:x2	as \(x_2\) increases by 1, the association between \(x_1\) and \(y\) changes by [estimate] or as \(x_1\) increases by 1, the association between \(x_2\) and \(y\) changes by [estimate]

What if there are other things (e.g., other predictors/covatiates) in the model too?

Note that the interaction x1:x2 changes how we interpret the individual coefficients for x1 and x2.

It does not change how we interpret coefficients for other predictors that might be in our model. For variables that aren’t involved in the interaction term, these are still held constant.

For example, suppose we also had another predictor \(c_1\) in our model:

lm(y ~ c1 + x1 + x2 + x1:x2)

Coefficient	Interpretation
(Intercept)	the estimated \(y\) when all predictors (\(c_1\), \(x_1\) and \(x_2\)) are zero is [estimate]
c1	a 1 unit increase in \(c_1\) is associated with a [estimate] increase in \(y\), holding constant all other variables in the model (\(x_1\) and \(x_2\))
x1	holding \(c_1\) constant, when \(x_2\) is zero, a 1 unit increase in \(x_1\) is associated with a [estimate] change in \(y\)
x2	holding \(c_1\) constant, when \(x_1\) is zero, a 1 unit increase in \(x_2\) is associated with a [estimate] change in \(y\).
x1:x2	holding \(c_1\) constant, as \(x_2\) increases by 1, the association between \(x_1\) and \(y\) changes by [estimate] or holding \(c_1\) constant, as \(x_1\) increases by 1, the association between \(x_2\) and \(y\) changes by [estimate]

Note

See S2 Week 1 Lectures, S2 Week 1 Lab, S2 Week 1 Lab, S2 Week 2 Lectures, S2 Week 2 Lab, S2 Week 3 Lectures, S2 Week 3 Lab, S2 Week 4 Lectures, and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Assumptions: Interactions

A few points to note when assessing the assumptions of an interaction model:

• When there is an interaction in the model, assessing linearity becomes difficult. In fact, crPlots() will not work. To assess, you can create a residuals-vs-fitted plot.
  
• Interaction terms often result in multicollinearity, because these terms are made up of the product of some ‘main effects’. Mean-centering the variables like we have here will reduce this source of structural multicollinearity (“structural” here refers to the fact that multicollinearity is due to our model specification, rather than the data itself).

Note

See S1 Week 9 Lab, S1 Week 9 Lectures, S2 Week 1 Lectures for further details, examples, and to revise these concepts further.

Mean Centering

By mean centering a continuous predictor, we change what 0 means/represents. Mean centering a variable involves subtracting the mean of that variable from every individual value. Normally, when we don’t have an interaction, this simply changes the intercept value (see 8A #centering-and-scaling-predictors).

If we have the interaction y ~ x1 + x2 + x1:x2, then mean centering x1 will make the coefficient for x2 now represent “the association between x2 and y for someone at the average of x1”. When working with models that contain interaction terms, it is generally a good idea to center your continuous predictor variables. This is because:

• centering helps to address issues of multicollinearity
• centering makes the model coefficients easier to interpret

Remember that the underlying model does not change, it is just extracting different information as we have changed what “zero” represents.

Note

See S2 Week 1 Lectures, S2 Week 2 Lectures, S2 Week 1 Lab, and S1 Week 10 Lab for further details, examples, and to revise these concepts further.

Side Contraints

Possible side-constraints on the parameters are:

Name	Constraint	Meaning of \(\beta_0\)	R
Sum to zero (Effects Coding)	\(\beta_1 + \beta_2 + \beta_3 = 0\)	\(\beta_0 = \mu\)	contr.sum
Reference group (Dummy Coding)	\(\beta_1 = 0\)	\(\beta_0 = \mu_1\)	contr.treatment

IMPORTANT

• By default R uses the reference group constraint. If your factor has \(g\) levels, your regression model will have \(g-1\) dummy variables (R creates them for you)

• We can switch back to the default reference group constraint by applying either of these:

# Option 1
contrasts(rest_spend$music) <- NULL
# Option 2
contrasts(rest_spend$music) <- "contr.treatment"

Note

See S1 Week 7 Lab, S1 Week 8 Lab, S1 Week 7 Lectures, S1 Week 8 Lectures, S2 Week 3 Lectures, S2 Week 3 Lab, S2 Week 4 Lectures, and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Interactions & Coding Constraints

When we have categorical predictors, our choice of contrasts coding changes the bits that we’re getting our of our model.

Suppose we have a 2x2 design (condition A and B, in groups 1 and 2):

Figure 5: Categorical x Categorical Interaction plot

When we are using the default contrasts coding (i.e., treatment or dummy) in R, then our coefficients for the individual predictors represent moving between the dots in Figure 5.

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)    
(Intercept)            1.9098     0.1759  10.855  < 2e-16 ***
conditionB             1.1841     0.2488   4.759 5.65e-06 ***
grouping2             -1.6508     0.2488  -6.635 1.09e-09 ***
conditionB:grouping2  -2.1627     0.3519  -6.146 1.15e-08 ***
---

• The intercept is the red circle in Figure 5.
  
• The coefficient for condition is the difference between the red circle and the red triangle in Figure 5.
  
• The coefficient for grouping is the difference between the red circle and the blue circle in Figure 5.
  
• The interaction coefficient is the difference from the slope of the red line to the slope of the blue line.

However, when we change to using effects (or sum-to-zero) coding, we’re switching where zero is in our model. So if we change to sum contrasts (here we’ve changed both predictors to using sum-to-zero coding), then we end up estimating the effect of each predictor averaged across the other.

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)    
(Intercept)           1.13577    0.08796  12.912  < 2e-16 ***
conditionB            0.05141    0.08796   0.584     0.56    
grouping2            -1.36607    0.08796 -15.530  < 2e-16 ***
conditionB:grouping2 -0.54066    0.08796  -6.146 1.15e-08 ***
---

• The intercept is the grey X in Figure 6.
  
• The coefficient for condition is the difference between the grey X and the grey triangle in Figure 6.
  
• The coefficient for grouping is the difference between the grey X and the blue line in Figure 6.
  
• The interaction coefficient is the difference from the slope of the grey line to slope of the blue line.

Figure 6: Visualisation of sum-to-zero for categorical x categorical interaction plot

It can get quite confusing when we start switching up the contrasts, but it’s all just because we’re changing what “zero” means, and what “moving 1” means:

Dummy/Treatment Coding

Effects/Sum-to-Zero Coding

Note

See S1 Week 7 Lab, S1 Week 8 Lab, S1 Week 7 Lectures, S1 Week 8 Lectures, S2 Week 3 Lectures, S2 Week 3 Lab, S2 Week 4 Lectures, and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Contrasts: Rules for Assigning Weights

• Rule 1: Weights are -1 \(\geq\) x \(\leq\) 1
• Rule 2: The group(s) in one chunk are given negative weights, the group(s) in the other get positive weights
• Rule 3: The sum of the weights of the comparison must be 0
• Rule 4: If a group is not involved in the comparison, weight is 0
• Rule 5: For a given comparison, weights assigned to group(s) are equal to 1 divided by the number of groups in that chunk.
• Rule 6: Restrict yourself to running \(k\) - 1 comparisons (where \(k\) = number of groups)
• Rule 7: Each contrast can only compare 2 chunks of variance
• Rule 8: Once a group singled out, it can not enter other contrasts

Note

See S1 Week 8 Lab, S1 Week 8 Lectures, S2 Week 4 Lectures, and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Multiple Comparisons: Why does the Number of Tests Matter?

If we are conducting all possible pairwise comparisons, we can calculate how many tests are being conducted via the following rule:

\[ _nC_r = \frac{n!}{r!(n-r)!} \\ \] \[ \begin{align} \\ & \text{Where:} \\ & n = \text{total number in the set} \\ & r = \text{number chosen} \\ & _nC_r = \text{number of combinations of r from n} \\ \end{align} \]

So, why does the number of tests matter? First, think back to “Type 1 errors” from DAPR1 - when we conduct an hypothesis test, and we set \(\alpha = .05\), we will reject the null hypothesis \(H_0\) when we find a \(p < .05\). Now remember what a \(p\)-value represents - it is the chance of observing a statistic at least as extreme as the one we do have, assuming the null hypothesis to be true. This means that if \(H_0\) is true, then we will still observe a \(p < .05\) 5% of the time. So our chance of making this error = the threshold (\(\alpha\)) at which below a p-value results in us rejecting \(H_0\).

But this error-rate applies to each statistical hypothesis we test. So if we conduct an experiment in which we plan on conducting lots of tests of different comparisons, the chance of an error being made increases substantially. Across the family of tests performed that chance will be much higher than 5%.²

Each test conducted at \(\alpha = .05\) has a .05 (or 5%) probability of Type I error (wrongly rejecting the null hypothesis). If we do 9 tests, that experiment-wise error rate is \(\alpha_{ew} \leq 9 \times .05\), where 9 is the number of comparisons made as part of the experiment.

Thus, if nine independent comparisons were made at the \(\alpha = .05\) level, the experiment-wise Type I error rate \(\alpha_{ew}\) would be at most \(9 \times .05 = 0.45\). That is, we could wrongly reject the null hypothesis on average 45 times out of 100. To make this more confusing, many of the tests in a family are not independent (see the lecture slides for the calculation of error rate for dependent tests).

Note

See S2 Week 4 Lectures and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Multiple Comparisons: When to use Which Correction

Bonferroni

• Use Bonferroni’s method when you are interested in a small number of planned contrasts (or pairwise comparisons).
• Bonferroni’s method is to divide alpha by the number of tests/confidence intervals.
• Assumes that all comparisons are independent of one another.
  
• It sacrifices slightly more power than Tukey’s method (discussed below), but it can be applied to any set of contrasts or linear combinations (i.e., it is useful in more situations than Tukey).
• It is usually better than Tukey if we want to do a small number of planned comparisons.

Šídák

• (A bit) more powerful than the Bonferroni method.
• Assumes that all comparisons are independent of one another.
• Less common than Bonferroni method, largely because it is more difficult to calculate (not a problem now we have computers).

Tukey

• It specifies an exact family significance level for comparing all pairs of treatment means.
  
• Use Tukey’s method when you are interested in all (or most) pairwise comparisons of means.

Scheffe

• It is the most conservative (least powerful) of all tests.
• It controls the family alpha level for testing all possible contrasts.
• It should be used if you have not planned contrasts in advance.
• For testing pairs of treatment means it is too conservative (you should use Bonferroni or Šídák).

In R, you can easily change which correction you are using via the adjust = argument.

Note

See S2 Week 4 Lectures and S2 Week 4 Lab for further details, examples, and to revise these concepts further.

Footnotes

1. A factorial ANOVA compares means across two or more independent variables (each with two or more levels) and their interaction.

2. what defines a ‘family’ of tests is debatable.