DAPR1 Workshops
  • 1: Exploratory Data Analysis
    • W1: Introductions
    • W2: Data & Variables
    • W3: Visualising Distributions
    • W4: Summarising Distributions I – Central Tendency
    • W5: Summarising Distributions II – Variability
  • 2: Introduction to Inference
    • W7: Z-scores & the Normal Distribution
    • W8: Sampling Distributions & Standard Error
  • 3: Common Statistical Tests
  • 4: The Move to Modelling

W8: Sampling Distributions & Standard Error

Pick someone new to be the driver this week. If everyone in the group has already had a week as the driver, start going round again!

The driver will type the code. The rest of you are navigators, you need to tell the driver where to go (i.e., what to type!).

Driver instructions:

  1. Log in to the bigger screen at your table
  2. Open RStudio Server. See the first drop-down in the FAQs page for instructions
  3. Open a new .R script, and load the tidyverse package.
  4. Listen to your navigators

Navigator instructions:

  1. Tell your driver where to go (i.e., what to type)!
Question 1

This week’s workshop is a little bit more conceptual in nature. The steps that you work through here are not steps that you would use in an actual data analysis scenario. Instead, the goal is to help you get more comfortable thinking about the precision of a summary statistic like the mean.

Concretely, you’ll create your own sampling distributions based on a population we’ve simulated. And you’ll see that the standard deviation of those sampling distributions comes pretty close to giving the same result as the standard error formula you saw in the lecture! This tells us that the standard error is a reasonable way to calculate the precision of an estimate.

If our silly questionnaire was filled in by every person in the world, we would have billions of outlook scores that tell us how pessimistic vs. optimistic everybody in the world is on a scale from –100 to 100. That would be our population distribution. In real life, this is of course impossible, but for the exercises this week, we are going to imagine that we know the mean and standard deviation of the population distribution for outlook: the mean is 15 and the SD is 25.

Here’s what this imaginary population distribution looks like. (The numbers along the y axis use scientific notation: for example, 8e+08 means “eight followed by eight zeros”, or 800,000,000. These counts are huge because there are over eight billion people in the world!)

Now we are going draw samples from this population. We can generate a pretend sample of outlook scores by using the function rnorm().

Copy this code into your R script and (re)run it a few times. Is your fake sample and its mean different from ours? Is your fake sample different every time you run the code?

fake_sample <- rnorm(
  n = 5,        # how many data points do we want to sample?
  mean = 15,    # what's the mean of the distribution we're sampling from?
  sd = 25       # what's the SD of the distribution we're sampling from?
)
fake_sample
[1] -25.6906  73.0784  13.0881  -5.9601   2.7541
mean(fake_sample)
[1] 11.454

Your numbers will look different to the ones below because of randomness in sampling.

One sample:

fake_sample <- rnorm(n = 5, mean = 15, sd = 25)
fake_sample
[1]  14.945  42.361  23.408  31.324 -11.803
mean(fake_sample)
[1] 20.047

Another sample:

fake_sample <- rnorm(n = 5, mean = 15, sd = 25)
fake_sample
[1] 21.6295 21.5960  2.1329 39.3774 -9.0878
mean(fake_sample)
[1] 15.13

Question 2

You have hopefully noticed that every time we take a new pretend sample, the actual numbers it contains are different, and so is its mean. This point is key: the mean of each sample is a little bit different. This raises our next question: how different are those means?

To find out, we are going to take a LOT of samples from the population, find the mean of each one, and look at how spread-out they are. We’ll use the function replicate(), which lets us do something in R lots of times. Specifically, we’re going to draw and summarise 1000 samples.

Copy and run the following code.

means20 <- replicate(1000, mean(rnorm(n = 20, mean = 15, sd = 25)))

means100 <- replicate(1000, mean(rnorm(n = 100, mean = 15, sd = 25)))

Now, do your best to answer the following questions about this code (and ask a tutor if you’re not sure!):

  1. What part(s) of this code draws a random sample from the population distribution?
  2. Do all samples contain the same number of data points?
  3. What part(s) of this code calculates the mean of the sample?
  4. What part(s) of this code draws and summarises a sample 1000 times?
  5. Which of these two objects, means20 or means100, has more variability among the means it contains?

  1. The rnorm() bit, specifically: rnorm(n = 20, mean = 15, sd = 25) generates a random sample of 20 data points from our population distribution, which is a normal distribution with mean 15 and standard deviation 25. And rnorm(n = 100, mean = 15, sd = 25) generates 100 data points from the same distribution.

  2. No: the first set of samples contains 20 data points each, while the second set of samples contains 100 data points each.

  3. The mean() bit, specifically: mean(rnorm(n = 20, mean = 15, sd = 25)) computes the mean of the 20 randomly sampled data points.

  4. The replicate() bit, specifically: replicate(1000, mean(rnorm(n = 20, mean = 15, sd = 25))) repeats the sampling-then-summarising process 1000 times, and it results in a vector of 1000 means.

  5. There’ll be more variability in means20 than in means100. Let’s just look at the first six values (yours will be a bit different from ours):

head(means20)   # these means are more spread out
[1] 14.2863 15.4354  8.8554  9.5261 15.5958 25.8270
head(means100)  # these means are closer together
[1] 14.080 18.080 15.671 13.189 15.698 16.003

This result should make some sense: you saw in the lecture that bigger samples result in less variability among the means.

Question 3

In this question, you will prepare the two sampling distributions means20 and means100 for plotting.

Currently, means20 and means100 are both vectors. But ggplot doesn’t like vectors. It only likes dataframes, and so we are going to make two dataframes: one for means20 and another for means100. To make a dataframe, we’ll use the function tibble().

In the following code, replace the ... with the name of the appropriate vector. Then run the code.

data20 <- tibble(
  estimate = ...
)

data100 <- tibble(
  estimate = ...
)

The result should be two dataframes that each contain one column called estimate. In that column are the values that make up each of our sampling distributions.

🗂️ See Create a dataframe flash card.

data20 <- tibble(
  estimate = means20
)

data100 <- tibble(
  estimate = means100
)

Question 4

First, use ggplot to make two plots, one for each of these sampling distributions. What sort of plot? That’s up to you!

🗂️ See Plot a variable flash card.

Once you’ve made both lots, let’s display them side by side in the same graphic. To do this, we’ll need the patchwork package (and you might need to install it first!).

🗂️ See Use R packages flash card.

To arrange the plots side by side, you’ll need to:

  1. assign each plot a name, e.g. plot1 <- ... and plot2 <- ..., and then
  2. use + to place them side by side (plot1 + plot2) or / to place one on top of the other (plot1 / plot2)

🗂️ See Patchwork flash card.

Each of the plots shows “means from 1000 samples we might see”, which is to say, they each show a sampling distribution of the mean. Rememebr that, because each sample is a random selection of people’s outlook scores, the means of the samples were different (some samples were full of more optimistic people, some with less optimistic people).

Now, do your best to answer the following questions (and ask a tutor if you’re not sure!):

  1. What differences do you see between the numbers in the plots you’ve made?
  2. Bonus challenge: Why are the numbers different?

library(patchwork)

p20 <- ggplot(data20, aes(x = estimate)) +
  geom_histogram()

p100 <- ggplot(data100, aes(x = estimate)) +
  geom_histogram()

p20 + p100

  1. On the left is the distribution of means from samples of size 20. These means range from just below 0 to just above 30 (we can tell from the numbers along the x axis). On the right is the distribution of means from samples of size 100. These means range from about 8 to just above 20—a narrower range.

  2. There’s more variability in the means of smaller samples, and less variability in the means of larger samples. This happens because bigger samples are more likely to include a bigger part of the population, so their means should be a bit closer to the mean of the full population.

Question 5

The goal of this final question is to show you that the standard deviation of the sampling distributions of the mean is pretty darn close to the standard error of the mean that we calculate using the following formula:

\[ SE_{\text{mean}} = \frac{\sigma}{\sqrt{n}} \]

You saw in the lecture that the point of the standard error is to characterise “how much a sample estimate will vary due to random sampling”. And that’s exactly what our sampling distributions show as well! So it makes sense that they give us similar numbers.

Let’s see it in action. Here’s how we’d calculate the standard error of the mean, using our sample sizes of 20 and 100, our population mean of 15, and our population SD of 25:

\[ \begin{align} SE_{\text{mean, n=20}} &= \frac{25}{\sqrt{20}} \\ \quad \\ &= 5.59 \\ \end{align} \]

\[ \begin{align} SE_{\text{mean, n=100}} &= \frac{25}{\sqrt{100}} \\ \quad \\ &= 2.5 \\ \end{align} \]

Now, compute the standard deviation of the means from samples of size 20, as well as for the means from samples of size 100.

They won’t match exactly, but they should be close. And that’s why the standard error formula is so cool and useful! It lets us directly calculate how much sample means are going to vary, without needing to go through this long effortful sampling process.

🗂️ See Measure Variability > Standard deviation flash card.

There are a couple ways to find the SDs of these sampling distributions.

The first is using your original vectors means20 and means100:

sd(means20)
[1] 5.5736
sd(means100)
[1] 2.469

The second is summarising the estimate column of data20 and data100:

data20 |>
  summarise(
    sd = sd(estimate)
  )
# A tibble: 1 × 1
     sd
  <dbl>
1  5.57
data100 |>
  summarise(
    sd = sd(estimate)
  )
# A tibble: 1 × 1
     sd
  <dbl>
1  2.47

Those numbers are indeed pretty similar to the SEs of 5.59 (for samples of size 20) and 2.5 (for samples of size 100)!

It’s really cool that statisticians figured out how to estimate the sampling variability using the standard error formula, because that save us from having to do all this sampling rigmarole. The standard error calculations above tell us how much the sample estimates will vary because of random sampling. So we can simply use the calculations to get at a theoretical probability distribution (the red curves below), rather than having to actually simulate stuff every single time.

Finishing up

Once you’ve finished, don’t forget to:

  1. Save your R script
  2. Post the R script to the group discussion space on Learn, to ensure that everyone has access to it! (For step-by-step instructions on how to do this, click here)