Report LMM analyses and results

In DAPR2, you learned to write an Analysis Plan section followed by a Results section. You might have noticed, though, that published academic articles typically go straight from a Methods section (where they discuss the study design, the study procedure, their participants, and the materials) into a Results section (where they report what they found). In DAPR3, we will train you to write something more like an article-style Results section, so that you’re better prepared for writing this section in your dissertations.

This flashcard doesn’t contain an example write-up, because we want you to learn to write your own report instead of using ours as a template. But below you’ll find guidelines and checklists for you to follow, so that you know you’ve included all the important information.

Checklist for a strong LMM-based Results section

Set the scene (so the reader knows what you’re talking about).

• Remind the reader briefly what the study aimed to test (e.g., what the predictions were).

Describe and show the sample data (so that the reader knows how generalisable your analysis might be).

• Introduce the variables the study is using to test this prediction, but avoid using R variable names; describe the variables in human-readable terms instead (e.g., what scale are they on? how were they measured?)
• Briefly describe the data set: how many observations? what does each observation represent?
• Describe how the data are structured, focusing on the grouping variables (e.g., how many people? how many observations per person? how many people per condition? …)
• If applicable, mention any data cleaning / outlier removal.

Visualise and summarise the results that address the RQ (so the reader knows the basic pattern of what you found).

• Include a plot of the data which includes all variables that are relevant for your story (and make sure to refer to it in the text).
  • Tip: If you transform any continuous predictors for analysis, use the transformed version of the predictor in the plot.
  • Tip: In the caption, briefly explain what every geom in the plot represents. Especially error bars/shaded error ribbons! Always tell the reader what they show (standard deviation? standard error? 95% CI?).
  • Tip: Academic writing styles vary, but Elizabeth likes it when figure captions give the reader a brief take-home message about what they should see in the plot.
• In a sentence or two, describe the overall pattern of results.
  • Tip: If you find yourself listing loads of numbers in the text, consider using tables or plots instead (and they’re better for your word count too!).
  • Tip: No inferential stats have been done yet, so we can’t make any claims just yet about whether the predictions were supported. But it is OK to say whether the results appear broadly in line with the prediction.

Introduce your statistical analysis (so the reader can interpret the numbers you will report).

• State that you’re running your analysis using R, and cite the developers (run citation() in the console to get the citation for your version of R).
• State the \(\alpha\) level that you’re using for all your analyses (in Psychology, the convention is 5%).
• Describe the fixed effects of your model in words, e.g.: “We fit a linear mixed model predicting Y as a function of X1, X2, and their interaction”.
• If fitting a logistic regression model, state this clearly.
• If categorical predictors, describe contrast coding, e.g.: “X1 has two levels, A (coded as –1) and B (coded as 1)”.
• If continuous predictors have been transformed, describe the transformation, e.g.: “X2 has been z-scored”.
• Describe the random effect structure of the maximal model.
  • One option using “random” terminology (more traditional): “The maximal model includes by-participant random intercepts, by-participant random slopes over X1, and by-stimulus random intercepts.”
  • Another option using “adjustment” terminology (less traditional but clearer): “The maximal model includes by-participant adjustments to the intercept and to the slope over X1, as well as by-stimulus intercept adjustments.”
• If you had to simplify the random effect structure, describe the order in which you removed parameters.
• Describe the random effect structure of your final model that converged without singularities (that is, describe the model that you’ll actually report below).
• State the model’s estimation method (REML or ML), how p-values were computed (if lmerTest, probably Satterthwaite’s method), and what optimiser you used (if not the default).
• For each model assumption, state whether you are satisfied that the assumption has been met.
• Include plots for each assumption check as figures in the Appendix section of the document (and refer to them in the main text).
• If applicable, check for multicollinearity and report VIF or GVIF.
• If you detected any high-influence observations and then conducted a sensitivity analysis, mention whether the pattern of results stayed the same even after removing each influential observation.
  • Tip: If you’ll report your results using the full regression table for your coefficients, then also include the full regression table for the sensitivity analysis in another Appendix section.

Report the results of your analysis (so the reader knows exactly what you found).

• State which model coefficient(s) are related to which prediction(s), and what direction/sign (positive or negative) you would expect the coefficients to have if the predictions were borne out.
• State the key coefficient estimates along with their 95% CI and p-value, e.g.: “The slope associating X1 and Y was significantly different from zero (\(b\) = 0.26, 95% CI [0.23, 0.29], \(p\) < 0.001)” or “The slope associating X1 and Y was not significantly different from zero (\(b\) = 0.26, 95% CI [–0.04, 0.56], \(p\) = 0.18)”.
  • Tip: You don’t need to report every single parameter estimate. Focus on the one(s) you identified as being relevant to your prediction(s).
  • Tip: Briefly link the direction and significance of the effect back to your earlier predictions—is the prediction supported by the parameter estimate?
• Include a table of all coefficients estimated by the model (and make sure to refer to it in the text). The function tab_model() from the package sjPlot is useful here, but note that tab_model() causes Rmd files to fail to knit to PDF, so you might want to manually recreate the table it produces.
  • Tip: if you DO report all coefficient estimates in the text, then there’s no need to include the coefficient table because the information is redundant.
• Plot the model-fitted values (and refer to the plot in the text).
• Report the magnitude of the SD of the group-level adjustments and compare it to the magnitude of the fixed effects. How does the intercept vary between groups? How does the effect of interest vary between groups? Interpret the variability in the group-level adjustments in the context of your data/analysis.
• Report the correlations between intercept and slope adjustments and interpret them in the context of your data/your analysis.
• Report any other information you consider important for your story!

Appendices might include:

• Descriptives tables or plots that are too big/disruptive to include in main text
• Figures that visualise assumption checks
• Sensitivity analysis regression table / any other useful details

General guidelines

• Your golden guiding principle should be: does the reader have all the information they would need to replicate your analysis in whatever statistical software they might be using?

• The goal of a Results section is to tell the reader what you found. The goal of the Discussion section is to tell the reader what it all means. The actual interpretation of the results belongs in the Discussion section, not the Results section.

• Follow the APA guidelines on how to report numbers and statistics.

• To make tables look extra professional, use horizontal lines only.

• If you have multiple research questions and multiple analyses, it’s usually most reader-friendly to run through this whole sequence with one analysis to address the first RQ, and then run through it again with the second analysis to address the second RQ, etc.

Elizabeth’s favourite books about writing

Completely optional: If you want to develop your academic writing skills even further, here are a couple books that Elizabeth learned a lot from and revisits often:

1. Style: Toward Clarity and Grace by Joseph M Williams
2. Stylish Academic Writing by Helen Sword