For logistic regression models, we measure model fit using something called deviance.
Deviance is a measure of how different the model is from the data. It reflects the misfit, or badness of fit, of the model to the data. Think of deviance as the logistic regression equivalent of a standard linear model’s residual sums of squares.
The logistic regression model summary shows us two kinds of deviance: null deviance and residual deviance.
Let’s look at this using our senility symptoms example:
sendata <-read_csv("https://uoepsy.github.io/data/SenilityWAIS.csv")sen_mdl1 <-glm(senility ~ wais, family ="binomial", data = sendata)summary(sen_mdl1)
Call:
glm(formula = senility ~ wais, family = "binomial", data = sendata)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 2.404 1.192 2.02 0.0437 *
wais -0.324 0.114 -2.84 0.0045 **
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 61.806 on 53 degrees of freedom
Residual deviance: 51.017 on 52 degrees of freedom
AIC: 55.02
Number of Fisher Scoring iterations: 5
Null deviance is the deviance of an intercept-only version of this model (i.e., a version with no predictors).
Residual deviance is the deviance of the model we actually fit, the version with all the predictors.
Because deviance represents badness of fit, small deviance represents better fit.
The residual deviance should basically always be smaller than the null deviance, meaning that the full model will basically always do a better job of fitting the data than the null model. This is not a statistical result yet, though. To know whether the full model has a significantly better fit than the null model, we must compute the difference in deviance and see whether it’s significantly different from zero.
Comparing nested models using difference in deviance
The difference in deviance represents how much the full model improves model fit, compared to the null model. We compute difference in deviance as the null deviance minus residual deviance. For our senility example:
\[
61.806 - 51.017 = 10.79
\]
With a big enough sample size, differences in deviance are \(\chi^2\)-distributed, meaning that we can compare them to an appropriate \(\chi^2\) distribution to compute a p-value. (“Appropriate \(\chi^2\) distribution” means a \(\chi^2\) distribution with degrees of freedom equal to the difference in df between the models being compared—R will compute this for you automatically.)
We use the anova() function with test = 'Chisq' to compare the deviance of two logistic regression models.
# Create the model we'll compare our full model to # (doesn't have to be a null model – can be anything, as long as it's nested # within the full model)sen_mdl0 <-glm(senility ~1, family ="binomial", data = sendata)# See whether difference in deviance is significantly different from zero# by comparing to a Chi squared distributionanova(sen_mdl0, sen_mdl1, test ="Chisq")
We performed a deviance goodness-of-fit test to compare our fitted model to the null. At the 5% significance level, the addition of information about the participants’ WAIS score resulted in a significant decrease in model deviance (\(\chi^2(1) = 10.79, p = .001\)). Hence, we have strong evidence that the subjects’ WAIS score was a helpful predictor of whether or not participants will experience symptoms of senility.
Important: it’s possible to compare the full model to a different, non-null version of the same model, but the models being compared must be nested. In other words, one model must be a simplification of the other; all predictors of that simpler model need to be contained within the more complex one.