Sensitivity analysis

mwdata <- read_csv('https://uoepsy.github.io/data/wellbeing_rural.csv')

A sensitivity analysis allows us to assess the sensitivity of our results (i.e., parameter estimates, p-values, confidence intervals) to changes in our modelling approach (here, the removal of observations).

We can re-fit our model after excluding potentially influential observations, and compare these results to the original model.

ImportantRemove all influential points at once, or one by one?

The current example involves removing all potentially influential observations at the same time. Ideally, and to ensure a more thorough sensitivity analysis, you would remove each of these observations one at a time, assess the effects on the model by comparing to your original, reassessing the remaining pre-identified observations, and repeating the process if necessary.

## wellbeing model
wb_mdl1 <- lm(wellbeing ~ outdoor_time + social_int, data = mwdata) 
summary(wb_mdl1)

Call:
lm(formula = wellbeing ~ outdoor_time + social_int, data = mwdata)

Residuals:
    Min      1Q  Median      3Q     Max 
-15.761  -3.131  -0.421   3.313  18.841 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)   28.6202     1.4879   19.24  < 2e-16 ***
outdoor_time   0.1991     0.0506    3.93  0.00012 ***
social_int     0.3349     0.0893    3.75  0.00023 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 5.07 on 197 degrees of freedom
Multiple R-squared:  0.126, Adjusted R-squared:  0.118 
F-statistic: 14.3 on 2 and 197 DF,  p-value: 1.64e-06
## wellbeing model
wb_mdl2 <- lm(wellbeing ~ outdoor_time + social_int, data = mwdata[-c(16, 25, 50, 53, 56, 58, 59, 60, 62, 72, 73, 75, 76, 78, 79, 85, 101, 109, 125, 126, 127, 131, 149, 151, 159, 163, 165, 169, 173, 176, 179, 197), ])
summary(wb_mdl2)

Call:
lm(formula = wellbeing ~ outdoor_time + social_int, data = mwdata[-c(16, 
    25, 50, 53, 56, 58, 59, 60, 62, 72, 73, 75, 76, 78, 79, 85, 
    101, 109, 125, 126, 127, 131, 149, 151, 159, 163, 165, 169, 
    173, 176, 179, 197), ])

Residuals:
   Min     1Q Median     3Q    Max 
-8.770 -2.644 -0.607  2.859  9.661 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)   27.9131     1.4261   19.57  < 2e-16 ***
outdoor_time   0.1936     0.0490    3.95  0.00012 ***
social_int     0.3983     0.0896    4.44  1.6e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 4.04 on 165 degrees of freedom
Multiple R-squared:  0.177, Adjusted R-squared:  0.167 
F-statistic: 17.8 on 2 and 165 DF,  p-value: 1e-07
tab_model(wb_mdl1, wb_mdl2,
          dv.labels = c("Wellbeing (WEMWBS Scores)", "Wellbeing (WEMWBS Scores)"),
          pred.labels = c("outdoor_time" = "Outdoor Time (hours per week)",
                          "social_int" = "Social Interactions (number per week)"),
          title = "Regression Table for Wellbeing Models wb1 and wb2")
Regression Table for Wellbeing Models wb1 and wb2
  Wellbeing (WEMWBS Scores) Wellbeing (WEMWBS Scores)
Predictors Estimates CI p Estimates CI p
(Intercept) 28.62 25.69 – 31.55 <0.001 27.91 25.10 – 30.73 <0.001
Outdoor Time (hours per
week)		 0.20 0.10 – 0.30 <0.001 0.19 0.10 – 0.29 <0.001
Social Interactions
(number per week)		 0.33 0.16 – 0.51 <0.001 0.40 0.22 – 0.58 <0.001
Observations 200 168
R2 / R2 adjusted 0.126 / 0.118 0.177 / 0.167

We conducted a sensitivity analysis to assess how robust our conclusions were regarding outdoor time and the weekly number of social interactions in the presence of previously identified potentially influential observations. We re-fit the model, excluding these 28 observations (14% of our original sample), and compared these model results (wb_mdl2) to those of our original model (wb_mdl1).

There was little difference in the estimates from wb_mdl1 and wb_mdl2, and so we can conclude that after conducting a sensitivity analysis, there were no meaningful differences in our results, and hence our conclusions from our original model hold. Specifically:

  • The direction of all model estimates are the same in wb_mdl1 and wb_mdl2 (i.e., all positive)
  • There is no difference in statistical significance, and the p-values were of a similar magnitude (i.e., all < .001)
  • The estimate and confidence intervals for outdoor_time are very similar
  • There are some quantitative differences in the estimate and confidence intervals for social_int. The estimate differs slightly in magnitude by 0.07), but given that this remains positive and significant, we do not need to be too concerned about this.