Hypothesis testing

import scipy as sp 
import numpy as np

A typical statement

A particular brand of tires claims that its deluxe tire averages at least 50,000 miles before it needs to be replaced. From past studies of this tire, the standard deviation is known to be 8,000. A survey of owners of that tire design is conducted. From the 28 tires surveyed, the mean lifespan was 46,500 miles with a standard deviation of 9,800 miles. Using 𝛼=0.05 , is the data highly inconsistent with the claim?

# What we know of the population
claim_pop_mean = 50_000
pop_std = 8000

# What we know of the sample
n = 28
sample_mean = 46_500
sample_std = 9800

# The chances of Type 1 error we are ready to accept
alpha = 0.05

The question can be formulated as: - “Compared to the mean of that population (50_000), how crazy is the sample mean (46_500) ? With an alpha of 0.05”

which becomes - “Using the sample deviation of the mean of that population, how far is the sample mean ? With an alpha of 0.05”

which becomes - “Is the sample mean further away than 1.64 times the standard error of that population ?”

# 1. How far is the sample_mean from the pop_mean ?
# H0 => pop_mean >= 50_000
# H1 => pop_mean < 50_000

population_standard_error = 8000 / np.sqrt(28) # "If you grab a random sample mean, how is it going to variate"
how_far_we_are_from_pop_mean = (46_500 - 50_000) / population_standard_error # How far is this specific sample mean from the population mean.

There are different ways to reject the null hypothesis.

We can look at wether we are smaller or not than 0.05.

how_far_we_are_in_z = sp.stats.norm.cdf(how_far_we_are_from_pop_mean) 
how_far_we_are_in_z

0.010305579572800304

In this case we are at 0.01, which means that in the distribution of sample means, we are so extreme that there is no way that the sample mean we observed actually came from the sample mean distribution that we built looking at the population.

Another way is to look at how far we go on the axis, not in term of percentage (like 0,05 being 5%) but in term of distance from the population mean. This would look like the following

how_far_we_are_from_pop_mean

-2.315032397181517

To know if this is a value too extreme or not, we can compare it to how far 0.05 is on the same axis:

- sp.stats.norm.ppf(0.95)

-1.6448536269514722

When you don’t have the population standard deviation

Realistically however, you often don’t have the population standard deviation. In this case, you need to estimate it from the sample.

Doing that is less accurate. In order to compensate a bit, we need to model the “spread of sample means” a bit differently.

Since normally we allow the sample mean to only go “so far” from the population mean. We will force it to be “even a bit further”. The way we do this is by using a “heavy tail” distribution for the sample mean. That way, the 0.05 mark will be further to the right or to the left, and we are forced to be a little bit more sure of ourselves before saying anything.

Let’s use the sample problem as above, but pretend that we don’t know that the population has a standard deviation of 8000. We are forced to use the 9800 that we discovered experimentally.

# What we know of the population
claim_pop_mean = 50_000
# --- pop_std = 8000 # we don't know this anymore ---

# What we know of the sample
n = 28
sample_mean = 46_500
sample_std = 9800

# The chances of Type 1 error we are ready to accept
alpha = 0.05

# 1. How far is the sample_mean from the pop_mean ?
# H0 => pop_mean >= 50_000
# H1 => pop_mean < 50_000

population_standard_error = 9800 / np.sqrt(28) # "If you grab a random sample mean, how is it going to variate"
how_far_we_are_from_pop_mean = (46_500 - 50_000) / population_standard_error # How far is this specific sample mean from the population mean. 

how_far_we_are_in_z = sp.stats.t.cdf(how_far_we_are_from_pop_mean, df=n-1) 
how_far_we_are_in_z

0.014225814767264972

We still reject the null hypothesis. But notice how much less confident we are ! Even if the standard deviation we sample was exactly 8000 (like the population one), we would still be less confident than if we received the standard deviation through a trustful source.

This is the whole point of this T student distribution !

Confidence interval

Confidence interval are only in the point of view of the sample we just took.

From that sample, let’s just add a standard error on each side and see how far this goes.

how_much_we_allow_on_unit_normal_distrib = sp.stats.norm.ppf(0.95)
sample_mean_standard_error = 9800 / np.sqrt(n)
how_much_we_allow_in_problem_domain = how_much_we_allow_on_unit_normal_distrib * sample_mean_standard_error
how_much_we_allow_in_problem_domain
[46_500 - how_much_we_allow_in_problem_domain, 46_500 + how_much_we_allow_in_problem_domain]

3046.311548011343