Source: Mathematical Statistics, University of Central Florida
Tags: hypothesis testing, Type I error, Type II error, alpha, beta, null hypothesis, alternative hypothesis, decision matrix, reject null, retain null, false positive, false negative
Difficulty: Intermediate | Prerequisites: Basic probability, familiarity with sampling distributions and the idea of a null hypothesis.
Hypothesis testing is the framework researchers use to decide whether observed data provide enough evidence to reject a default assumption (the null hypothesis). Every test carries two risks: concluding there is an effect when there is none, or missing a real effect entirely. This section of the course formalises those risks as Type I and Type II errors and shows how they relate to each other through the overlapping distributions of the null and alternative hypotheses. If you understand the decision matrix and can label the regions on those two curves, everything else in power analysis follows from it.
When you run a hypothesis test, you either reject or retain the null, and the null is either true or false in reality. That gives four outcomes, two correct and two errors. Type I error (α) is a false alarm; Type II error (β) is a miss. Researchers typically worry more about Type I errors because they lead to publishing false findings.
Null hypothesis (H₀)
The default claim that there is no effect, no difference, or no relationship in the population. The hypothesis you are trying to find evidence against. Think of it as: the "nothing interesting is happening" assumption that stands unless the data say otherwise.
Alternative hypothesis (H₁ or Hₐ)
The research claim that some effect, difference, or relationship does exist. This is what the researcher hopes to support. Think of it as: the thing you actually suspect is true and are collecting data to demonstrate.
Null distribution
The sampling distribution of the test statistic assuming H₀ is true. This is the left-hand curve in the classic two-distribution diagram. In simple terms, this means: it shows where your sample results would land if the null were correct.
Alternative distribution
The sampling distribution of the test statistic assuming H₁ is true. This is the right-hand curve in the diagram. In simple terms, this means: it shows where your sample results would land if the effect is real.
Alpha (α)
The probability of rejecting H₀ when H₀ is actually true. Also called the significance level. Conventionally set at .05 or .01. Think of it as: your tolerance for false alarms.
Beta (β)
The probability of retaining (failing to reject) H₀ when H₀ is actually false. This is the miss rate. Think of it as: the chance you let a real effect slip through undetected.
Type I error
Rejecting H₀ when H₀ is true. The false positive. In simple terms, this means: you claim there is an effect when there really is not one.
Type II error
Retaining H₀ when H₀ is false. The false negative. In simple terms, this means: you fail to detect an effect that genuinely exists.
1 − α (confidence level)
The probability of correctly retaining H₀ when H₀ is true. A correct decision, sometimes called the specificity of the test.
1 − β (power)
The probability of correctly rejecting H₀ when H₀ is false. The ability of the test to detect a real effect when one exists.
Critical value / cutoff (α cutoff)
The threshold on the test statistic beyond which you reject H₀. The decision to retain the null falls on the left side of this cutoff (in a right-tailed test); rejection is on the right side.
In the standard diagram, you see two bell curves that overlap.
The left curve represents the null distribution (what the world looks like if H₀ is true).
The right curve represents the alternative distribution (what the world looks like if H₁ is true).
A vertical line marks the critical value (cutoff). This is set by your chosen α level.
Everything to the right of the cutoff in the null distribution is α, the rejection region.
Everything to the left of the cutoff in the null distribution is 1 − α.
Everything to the left of the cutoff in the alternative distribution is β, the region where you would fail to detect the effect.
Everything to the right of the cutoff in the alternative distribution is 1 − β (power), the region where you correctly detect the effect.
α sits in the null distribution (right tail, for a right-tailed test).
1 − α also sits in the null distribution (the large remaining area).
β sits in the alternative distribution (the portion that falls below the cutoff).
1 − β sits in the alternative distribution (the portion that exceeds the cutoff).
Because α and β are in different distributions, they do not have to add up to 100%. They are independent probabilities governed by different curves.
The four possible outcomes of any hypothesis test:
Null is true (in reality) | Null is false (in reality) | |
|---|---|---|
Reject null | Type I error (probability = α) | Correct decision (probability = 1 − β, i.e. power) |
Retain null | Correct decision (probability = 1 − α) | Type II error (probability = β) |
Given power = .8 and α = .01:
β = 1 − .8 = .20
1 − α = 1 − .01 = .99
Null is true | Null is false | |
|---|---|---|
Reject null | .01 (Type I error) | .80 (power) |
Retain null | .99 (correct retention) | .20 (Type II error) |
Retaining the null when the null is true is a correct decision. You did not raise a false alarm.
Rejecting the null when the null is false is also a correct decision. You detected a real effect. This correct detection is what we call power.
Researchers are typically most concerned about Type I error because it means publishing a false finding, one that claims an effect exists when it does not. False positives enter the literature, influence policy, waste replication resources, and can cause real harm (think: approving an ineffective drug). This is why α is set conservatively (usually .05 or .01) before data are collected. Type II errors are also costly (missing a real effect), but they are generally seen as less damaging to the broader scientific record because no false claim is made public.
Key relationships:
Power = 1 − β
Significance level = α
Confidence level = 1 − α
β = 1 − Power
Decision matrix formula (general):
Each column of the matrix sums to 1 within its own column:
Null true column: α + (1 − α) = 1
Null false column: (1 − β) + β = 1
In clinical trials, a Type I error would mean approving a treatment that does not work, exposing patients to side effects for no benefit. A Type II error would mean shelving a treatment that actually helps. Regulatory agencies (like the FDA) set strict α levels precisely because the cost of false positives in medicine is so high.
"α and β must add up to 1." They do not. α is a probability in the null distribution; β is a probability in the alternative distribution. They are from different curves entirely.
"Retaining the null proves the null is true." Retention only means you did not find sufficient evidence to reject. It is not confirmation.
"Type II error is less important because β is usually larger than α." The size of β does not indicate importance. Missing a real effect can be very costly depending on the context.
"The cutoff line separates the null and alternative distributions." The cutoff separates the rejection region from the retention region. Both distributions span the entire x-axis; they overlap.
⚠️ You will almost certainly be asked to fill in the four-cell decision matrix with specific α and power values. Practise doing this quickly.
⚠️ Know which distribution each probability (α, β, 1 − α, 1 − β) belongs to. This is a very common exam question.
⚠️ Be ready to explain why α and β do not sum to 1.
⚠️ "Which type of error concerns researchers most and why?" is a classic short-answer prompt.
1. True or False: α and β must sum to 1.00. False. They are probabilities from two different distributions.
2. Fill in the blank: The probability of correctly rejecting a false null hypothesis is called ______. Power (1 − β).
3. True or False: Type I error means failing to detect a real effect. False. That is Type II error. Type I error is a false positive.
4. Fill in the blank: β lives in the ______ distribution. Alternative distribution.
5. True or False: Retaining the null hypothesis proves it is true. False. It only means you lacked sufficient evidence to reject it.
Q: If we retain the null and our sample mean comes from the null is true distribution, what is this called?
A: A correct decision (true negative). The null is true and we correctly retained it. Its probability is 1 − α.
Q: If we reject the null and our sample mean comes from the null is false distribution, what is this called?
A: A correct decision (true positive), also known as power. Its probability is 1 − β.
Q: In the two-distribution diagram, which curve does α come from, and which does β come from?
A: α comes from the null distribution (H₀ true). β comes from the alternative distribution (H₁ true).
Q: Do α and β have to add up to 100%? Why or why not?
A: No. α and β are probabilities drawn from two separate distributions (null and alternative, respectively), so there is no reason they must sum to 1.
Q: Fill in the decision matrix if 1 − β = .8 and α = .01.
A: Reject null / Null true = .01 (Type I error). Reject null / Null false = .80 (power). Retain null / Null true = .99 (correct). Retain null / Null false = .20 (Type II error).
Q: Which type of error are researchers most concerned about, and why?
A: Type I error (false positive), because it leads to publishing claims about effects that do not exist, which can mislead the field and cause real-world harm.
This material connects directly to the next section on power analysis, where you learn how to manipulate α, sample size, and effect size to control β and power. It also underpins all inferential statistics you will encounter later in the course, from t-tests to ANOVA to regression, since every test carries these same four possible outcomes. Understanding the decision matrix is also essential for research methods and experimental design courses.
hypothesis testing, Type I error, Type II error, alpha level, beta, significance level, false positive, false negative, null hypothesis, alternative hypothesis, power, 1 minus beta, decision matrix, reject null, retain null, fail to reject, critical value, cutoff, confidence level, p-value, statistical inference, UCF mathematical statistics, HW 7