Source: Tanhane/Dunlop, Section 6.3, Chapter 7, Sections 9.1, 14.1
Tags: hypothesis testing, null hypothesis, alternative hypothesis, Type I error, Type II error, power function, p-value, significance level, Neyman-Pearson, rejection region, test statistic, critical value, UMP test, sample size determination, effect size, composite hypothesis, simple hypothesis
Hypothesis testing is a formal framework for deciding whether sample data provide enough evidence to support a research claim about a population parameter. You set up two competing hypotheses (null and alternative), collect data, compute a test statistic, and check whether it falls in a rejection region. The entire framework revolves around controlling two types of errors and balancing them through sample size and significance level choices.
Null hypothesis (H₀)
The default position, assumed true unless the data provide strong evidence against it. Represents the "status quo" or "no effect" claim.
Alternative hypothesis (H₁)
The research hypothesis, the assertion the researcher is seeking evidence to support. Sometimes called the research hypothesis.
Parameter space (Θ)
The set of all possible values for the parameter θ. Partitioned into Θ₀ (null region) and Θ₁ (alternative region), where Θ = Θ₀ ∪ Θ₁ and Θ₀ ∩ Θ₁ = ∅.
Test statistic T(X)
A function of the sample data and the parameter being tested, used to evaluate which hypothesis is more plausible. Similar to the pivot used in confidence interval construction.
Rejection region (R)
The set of values of T(X) that lead to rejecting H₀. Often takes the form R = {X : T(X) > C} where C is the critical value.
Critical value (C)
The boundary value that separates the rejection region from the non-rejection region.
Type I error (false positive)
Rejecting H₀ when H₀ is in fact true. Probability denoted α.
Type II error (false negative)
Failing to reject H₀ when H₀ is in fact false (i.e. H₁ is true). Probability denoted β.
Power of a test
The probability of correctly rejecting H₀ when H₁ is true. Equal to 1 − β. Higher power means better ability to detect a real effect.
Power function γ(θ*)
γ(θ*) = P(T(X) ∈ R | θ = θ*), the probability the test rejects H₀ as a function of the true parameter value θ*.
For θ* ∈ Θ₀: γ(θ*) gives the probability of a Type I error
For θ* ∈ Θ₁: γ(θ*) gives the probability of a correct decision (power)
For θ* ∈ Θ₁: 1 − γ(θ*) gives the probability of a Type II error
Size of a test (α)
The maximum probability of a Type I error over the null parameter space: α = sup{γ(θ) : θ ∈ Θ₀}. Also called the level of significance.
Significance probability (p-value)
The smallest value of α at which H₀ would be rejected given the observed data. Equivalently, the probability of observing a test statistic as extreme or more extreme than the one computed, assuming H₀ is true.
Neyman-Pearson philosophy
The standard approach: fix an upper bound α on the Type I error rate, then choose the test that minimises the Type II error rate (maximises power) subject to that constraint.
Simple hypothesis
A hypothesis that specifies a single value: θ = θ₀.
Composite hypothesis
A hypothesis that specifies a range of values: θ < θ₀, θ > θ₀, θ ≤ θ₀, θ ≥ θ₀, or θ ≠ θ₀.
One-sided hypotheses
H₀: θ ≤ θ₀ vs H₁: θ > θ₀, or H₀: θ ≥ θ₀ vs H₁: θ < θ₀.
Two-sided hypotheses
H₀: θ = θ₀ vs H₁: θ ≠ θ₀.
Uniformly most powerful (UMP) test
A test that has the largest possible power for all θ ∈ Θ₁ compared to any other test of the same size. Often difficult to find or may not exist.
Region of uncertainty
The interval between θ₀ and the critical value C. When the sample mean falls in this region, it exceeds θ₀ but does not provide enough evidence to reject H₀ under the Neyman-Pearson philosophy.
Effect size
A measure of practical significance, the magnitude of the departure from H₀. Used alongside statistical significance to assess whether a result matters in practice.
Fail to reject H₀ | Reject H₀ | |
|---|---|---|
H₀ true | Correct decision | Type I error (α) |
H₁ true | Type II error (β) | Correct decision (power = 1 − β) |
Identify the probability model and translate population statements into parameter ranges
Formulate H₁ as the research hypothesis; H₀ as its negation
Select a test statistic T(X) that measures plausibility of H₁
Specify significance level α (maximum P[Type I error])
Determine the rejection region using the sampling distribution of T(X)
Examine the power curve to ensure adequate protection against Type II errors at important parameter values
Compute the test statistic from data and determine whether it falls in the rejection region
Report the p-value alongside the decision
Where possible, provide a confidence interval for the parameter to assess practical significance
Left-sided (H₁: θ < θ₀)
R = {X : T(X) < C_α}
Right-sided (H₁: θ > θ₀)
R = {X : T(X) > C₁₋α}
Two-sided (H₁: θ ≠ θ₀)
R = {X : T(X) < C_{α/2} or T(X) > C_{1−α/2}}
For a fixed sample size n and parameter value in H₁:
Increasing α (from .001 to .05) increases power and decreases β
Decreasing α decreases power and increases β
For a fixed α and parameter value in H₁:
Increasing n increases power and decreases β
Decreasing n decreases power and increases β
The trade-off: for a finite sample size, you cannot simultaneously control both Type I and Type II error probabilities freely. Adjusting C to reduce one increases the other.
A p-value is the probability of obtaining a new data set, under H₀ conditions, that provides even stronger evidence for H₁ than the current data.
Decision rule using p-values:
If p-value ≤ α, reject H₀ (data supports H₁)
If p-value > α, fail to reject H₀ (data does not support H₁)
Critical misconceptions to avoid:
The p-value is not the probability that H₀ is true
A p-value does not measure the size of an effect or the importance of a result
By itself, a p-value does not provide a good measure of evidence regarding a model
p-values can indicate how incompatible the data are with a specified statistical model
p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone
Scientific conclusions should not be based only on whether a p-value passes a specific threshold
Proper inference requires full reporting and transparency
A p-value does not measure the size of an effect or the importance of a result
By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis
The ASA warns against "p-hacking" and "data dredging," practices that chase small p-values rather than pursuing sound scientific reasoning.
The arrangement of hypotheses affects the location of the region of uncertainty and can lead to different conclusions from the same data. The research hypothesis (the claim you want to establish) should be placed as H₁, with H₀ representing the status quo.
Example with the same data (n = 10, sample mean = 86.3, S = 3.2, α = .01):
Case 1 (H₁: μ > 84): critical value = 86.85, sample mean = 86.3 < 86.85, fail to reject
Case 2 (H₁: μ < 84): T = 2.27 > −2.821, fail to reject
Both fail to reject, but the "fail to reject" conclusions support opposite directions. The region of uncertainty shifts depending on how H₀ and H₁ are arranged.
A confidence interval provides an alternative way to perform hypothesis tests.
Two-sided test (H₁: θ ≠ θ₀): Construct a 100(1 − α)% CI for θ. Reject H₀ if θ₀ falls outside the CI.
One-sided test (H₁: θ > θ₀): Construct a 100(1 − α)% lower bound (θ̂_L, ∞). Reject H₀ if θ₀ < θ̂_L.
One-sided test (H₁: θ < θ₀): Construct a 100(1 − α)% upper bound (−∞, θ̂_U). Reject H₀ if θ₀ > θ̂_U.
This equivalence holds exactly when the test statistic and the pivot used for the CI are the same.
Power function (general):
γ(θ*) = P_{θ*}(T(X) ∈ R)
Size of test:
α = sup{γ(θ) : θ ∈ Θ₀}
Type II error probability:
β(θ) = 1 − γ(θ) for θ ∈ Θ₁
Power curves take three characteristic shapes depending on the alternative:
H₁: θ < θ₀ produces a decreasing curve (high power far left of θ₀)
H₁: θ > θ₀ produces an increasing curve (high power far right of θ₀)
H₁: θ ≠ θ₀ produces a U-shaped curve (power dips to α at θ₀, rises in both directions)
⚠️ You never "prove" H₀ or H₁. You only state which is more plausible given the data. There is always a possibility of Type I or Type II error.
⚠️ "Fail to reject H₀" is not the same as "accept H₀." Absence of evidence is not evidence of absence.
⚠️ The p-value is not the probability that H₀ is true. This is perhaps the most common misconception in statistics.
⚠️ The arrangement of H₀ and H₁ shifts the region of uncertainty and can change the conclusion. Always place the research claim in H₁.
⚠️ Increasing α increases power but also increases the risk of a Type I error. There is no free lunch with finite samples.
⚠️ Always examine the power curve to assess whether your sample size provides adequate protection against Type II errors at practically important parameter values.
Q: What is the difference between the size of a test and the p-value?
A: The size (α) is the pre-specified maximum allowable probability of Type I error, set before collecting data. The p-value is computed from the data and represents the smallest α at which H₀ would be rejected. If p-value ≤ α, you reject H₀.
Q: A researcher tests H₀: μ ≤ 50 vs H₁: μ > 50 and obtains a p-value of 0.03. What can she conclude at α = 0.05? At α = 0.01?
A: At α = 0.05, she rejects H₀ because 0.03 < 0.05. At α = 0.01, she fails to reject H₀ because 0.03 > 0.01.
Q: For a fixed sample size, what happens to the probability of Type II error as α decreases?
A: β increases. Reducing α (making it harder to reject H₀) means you require stronger evidence, so you are more likely to miss a true effect.
Q: Why does the Neyman-Pearson philosophy fix α first and then minimise β?
A: Because for finite samples, you cannot freely control both error probabilities simultaneously. The convention treats Type I error as the more serious error to control (since it leads to a false claim of a discovery), then seeks maximum power subject to that constraint.
Q: If a 95% confidence interval for μ is (52.3, 61.7), what is the result of testing H₀: μ = 50 vs H₁: μ ≠ 50 at α = 0.05?
A: Reject H₀, because 50 is not contained in the 95% CI.
Q: Is a Type I error always more consequential than a Type II error?
A: Not necessarily. In a medical screening test (H₀: patient does not have disease), a Type II error (missing a real disease) may be far more consequential than a Type I error (unnecessary further testing). The relative severity depends on the context.
hypothesis test, significance test, null hypothesis, alternative hypothesis, research hypothesis, Type I error, Type II error, false positive, false negative, alpha, beta, power, power function, power curve, p-value, significance probability, significance level, Neyman-Pearson, rejection region, critical value, test statistic, simple hypothesis, composite hypothesis, one-sided test, two-sided test, one-tailed, two-tailed, UMP test, uniformly most powerful, region of uncertainty, effect size, practical significance, statistical significance, ASA statement, p-hacking, data dredging, confidence interval test equivalence