Course: PSY 301 Understanding Statistics, East Texas A&M Source: Moore & Notz, Statistics: Concepts and Controversies, 10th ed., Chapters 21, 22 Covers: Exam 4 (Final Exam) material
Difficulty: Intermediate to Advanced Prerequisites: Everything from Exams 1 through 3, especially the normal distribution (Chapter 13) and basic probability (Chapters 17, 18). Confidence intervals and hypothesis tests rely on z-scores, standard deviations, and the logic of probability. If those concepts are not solid, revisit them first.
Up to this point in the course, you have been describing data (descriptive statistics). Chapters 21 and 22 mark the shift to inferential statistics: drawing conclusions about a population based on sample data. Chapter 21 introduces the confidence interval, a range of plausible values for a population parameter. Chapter 22 introduces statistical significance testing, a method for deciding whether an observed result is likely due to chance or reflects a real effect. Together, these chapters answer the central question of statistics: "What can we learn about the whole population from the part we actually measured?"
A confidence interval gives a range of reasonable values for a population parameter, paired with a confidence level (usually 95%) that describes how often the method works in the long run. Statistical significance testing asks whether observed data are consistent with a specific claim (the null hypothesis). If the data would be very unlikely under the null hypothesis (p-value below a threshold, typically 0.05), we reject the null. Neither tool proves anything with certainty; both quantify uncertainty.
Statistical inference
The process of drawing conclusions about a population based on data from a sample. Think of it as: using partial information to make educated statements about the whole.
Parameter
A number that describes the population (e.g., the population mean μ or the population proportion p). In simple terms, parameters are fixed but usually unknown, which is why we estimate them.
Statistic
A number computed from sample data (e.g., the sample mean x̄ or the sample proportion p̂). Think of it as: the sample's best guess at the corresponding parameter.
Sampling distribution
The distribution of a statistic over all possible samples of the same size from the same population. In simple terms, if you could take thousands of random samples and compute the mean each time, the histogram of all those means would be the sampling distribution.
Sampling variability
The fact that different random samples produce different values of a statistic. Think of it as: natural fluctuation from sample to sample. It is not a mistake; it is an inherent feature of working with samples.
Margin of error
The amount added and subtracted from the sample statistic to create a confidence interval. In simple terms, it represents the "plus or minus" range that accounts for sampling variability.
Confidence interval
An interval of the form "estimate ± margin of error" that is designed to capture the true population parameter a specified percentage of the time. Think of it as: rather than a single guess, you give a range that you are reasonably confident contains the true value.
Confidence level (e.g., 95%)
The percentage of all possible samples for which the confidence interval method will capture the true parameter. In simple terms, "95% confident" means that if you repeated the sampling and interval-building process many times, about 95% of the resulting intervals would contain the true population value.
Null hypothesis (H₀)
A statement of "no effect" or "no difference," the claim being tested. Think of it as: the default position, the assumption that nothing interesting is happening, until the data persuade you otherwise.
Alternative hypothesis (Hₐ)
The claim you are trying to find evidence for, typically that there is an effect, a difference, or a relationship. In simple terms, this is what you suspect is true and hope the data will support.
P-value
The probability of obtaining a result as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true. Think of it as: how surprised you should be if the null hypothesis were correct. A small p-value means the data are hard to explain under the null.
Statistical significance
A result is statistically significant if it is unlikely to have occurred by chance alone, typically when the p-value falls below a chosen significance level (commonly 0.05). In simple terms, significant means "the data provide enough evidence to reject the null hypothesis," not "the result is important" in everyday language.
Significance level (α, alpha)
The threshold for the p-value below which you reject the null hypothesis. In simple terms, α = 0.05 means you are willing to accept a 5% risk of wrongly rejecting a true null hypothesis.
Type I error
Rejecting the null hypothesis when it is in fact true (a false alarm). Think of it as: concluding there is an effect when there is not one. The probability of a Type I error equals the significance level α.
Type II error
Failing to reject the null hypothesis when the alternative is in fact true (a miss). Think of it as: failing to detect a real effect. This is related to sample size and effect size, and is more likely with small samples.
Practical significance
Whether a statistically significant result is large enough to matter in the real world. In simple terms, a very large sample can make a trivially small effect statistically significant. Always ask "how big is the effect?" not just "is it significant?"
A confidence interval has two components:
A point estimate (the sample statistic, e.g., x̄ or p̂).
A margin of error that reflects sampling variability.
The general form is: estimate ± margin of error.
The margin of error depends on three things:
The confidence level you choose (higher confidence = wider interval).
The sample size n (larger samples produce smaller margins of error).
The variability in the population (more variability = wider interval).
For a population proportion with a large sample, the approximate 95% confidence interval is:
p̂ ± 2 × √(p̂(1 − p̂) / n)
(The "2" is a rounded version of the z-value 1.96 for 95% confidence.)
What "95% confident" means:
It does not mean there is a 95% chance the true parameter lies in this particular interval. The parameter is a fixed number; it either is in the interval or it is not.
It means: the method produces intervals that capture the true parameter 95% of the time across many samples.
This is a statement about the reliability of the procedure, not about one specific interval.
How to make the margin of error smaller:
Increase the sample size (the most common approach).
Accept a lower confidence level (e.g., 90% instead of 95%, though this is usually undesirable).
Reduce variability (often not under your control).
Confidence intervals require random samples. If the sample is biased (convenience, voluntary response), no confidence interval can rescue it.
The logic of significance testing:
Start with a null hypothesis (H₀) that represents no effect or no difference.
State the alternative hypothesis (Hₐ) that represents the effect you are looking for.
Collect data and compute a test statistic.
Find the p-value: the probability of results this extreme (or more) under H₀.
Compare the p-value to the significance level α.
If p-value ≤ α, reject H₀ in favour of Hₐ. If p-value > α, fail to reject H₀.
Interpreting the p-value:
The p-value is not the probability that H₀ is true. It is the probability of the observed data (or more extreme) assuming H₀ is true.
A small p-value (e.g., 0.01) means the observed result would be rare under H₀, which is evidence against H₀.
A large p-value (e.g., 0.45) means the observed result is perfectly consistent with H₀, so there is no reason to reject it.
"Fail to reject" is not the same as "accept." When the p-value is large, you have not proved H₀ is true; you simply have insufficient evidence to reject it. The distinction matters.
Errors in significance testing:
Type I error (false positive): rejecting H₀ when it is true. Probability equals α.
Type II error (false negative): failing to reject H₀ when Hₐ is true.
These two errors are in tension: reducing one typically increases the other unless you increase sample size.
Statistical significance vs. practical significance:
A large sample can make a tiny, meaningless difference statistically significant.
Always report the size of the effect alongside the significance result.
Example: a study finds that a new teaching method improves exam scores by 0.3 points on a 100-point scale with p = 0.02. Statistically significant, yes. Practically meaningful? Probably not.
Significance testing does not prove causation. Even a significant result from an observational study cannot establish cause and effect because confounding variables may be present. Randomised experiments are needed for causal claims.
95% confidence interval for a proportion:
p̂ ± 2 × √(p̂(1 − p̂) / n)
Where p̂ is the sample proportion and n is the sample size.
Margin of error (for a proportion, 95% confidence):
margin of error ≈ 2 × √(p̂(1 − p̂) / n)
Note: the factor "2" approximates the z-value of 1.96 for 95% confidence. For 90% confidence, use 1.645. For 99% confidence, use 2.576.
Relationship between margin of error and sample size:
To cut the margin of error in half, you must multiply the sample size by four. Margin of error decreases with the square root of n, not linearly.
Political polls always report a margin of error (e.g., "the candidate leads 52% to 48% with a margin of error of ±3 percentage points"). That margin of error is the confidence interval at work. If the margin of error overlaps the 50% mark, the race is too close to call statistically.
Clinical drug trials use significance testing to determine whether a new treatment performs better than a placebo. The FDA typically requires p < 0.05 before approving a new drug, but also examines the size of the clinical benefit (practical significance) to decide whether the improvement is meaningful for patients.
Students often say "there is a 95% probability that the true mean lies in this interval." This is incorrect. The true mean is fixed; it either is in the interval or it is not. The 95% refers to the long-run success rate of the method, not to any single interval.
Students confuse the p-value with the probability that the null hypothesis is true. The p-value is a probability about the data given the null, not a probability about the null given the data.
Students think "not significant" means "no effect." Failing to reject H₀ does not prove H₀ is true; it may simply mean the sample was too small to detect a real effect (Type II error).
Students equate statistical significance with importance. A result can be statistically significant but practically meaningless if the effect size is tiny.
⚠️ Know the formula for a 95% confidence interval for a proportion and be able to calculate one.
⚠️ Understand and be able to explain what "95% confident" means in terms of the long-run success rate of the method. Expect this as a written or multiple-choice question.
⚠️ Know the six-step logic of significance testing: H₀, Hₐ, test statistic, p-value, compare to α, conclusion.
⚠️ Be able to interpret a p-value correctly. This is one of the most commonly tested (and most commonly missed) concepts.
⚠️ Understand Type I and Type II errors, and be able to identify which is which from a scenario.
⚠️ Know the difference between statistical significance and practical significance. Expect a scenario question that asks you to evaluate both.
⚠️ The final exam is cumulative. All material from Exams 1 through 3 is fair game alongside these topics.
True or false: A 99% confidence interval is narrower than a 95% confidence interval for the same data.
Fill in the blank: The __________ hypothesis represents the "no effect" or "no difference" position.
True or false: A p-value of 0.03 means there is a 3% chance the null hypothesis is true.
Fill in the blank: Rejecting a true null hypothesis is called a Type __________ error.
True or false: To halve the margin of error, you need to quadruple the sample size.
Answers: 1. False (99% is wider). 2. Null. 3. False (it means the data would be this extreme only 3% of the time if H₀ were true). 4. I. 5. True.
Q: A poll of 900 voters finds that 54% support a ballot measure. Construct an approximate 95% confidence interval for the true proportion of supporters.
A: p̂ = 0.54, n = 900. Margin of error ≈ 2 × √(0.54 × 0.46 / 900) = 2 × √(0.000276) = 2 × 0.0166 ≈ 0.033. The 95% confidence interval is 0.54 ± 0.033, which is roughly 0.507 to 0.573, or about 50.7% to 57.3%.
Q: A researcher tests whether a new drug lowers blood pressure compared to a placebo. The null hypothesis is that the drug has no effect. The p-value is 0.08. At the α = 0.05 level, what is the conclusion?
A: Since 0.08 > 0.05, we fail to reject the null hypothesis. There is not enough evidence at the 5% significance level to conclude the drug lowers blood pressure. This does not prove the drug has no effect; the sample may simply have been too small.
Q: Explain the difference between a Type I error and a Type II error in the context of a criminal trial analogy.
A: In a trial, H₀ is "the defendant is innocent." A Type I error is convicting an innocent person (rejecting a true H₀). A Type II error is acquitting a guilty person (failing to reject a false H₀). The justice system sets a high bar for conviction (beyond reasonable doubt) to minimise Type I errors, at the cost of some Type II errors.
Q: A study with 50,000 participants finds that a diet supplement reduces weight by an average of 0.1 kg over 6 months, with p = 0.001. Is this result practically significant?
A: Although the result is statistically significant (p = 0.001 is well below 0.05), the effect size of 0.1 kg over 6 months is trivially small. This is not practically significant; the supplement makes virtually no real-world difference.
Q: Why does increasing the sample size make the margin of error smaller?
A: Larger samples provide more information about the population, which reduces sampling variability. Mathematically, the margin of error includes √(1/n), so as n increases, the margin of error decreases. The relationship is not linear: you must quadruple n to halve the margin of error.
Confidence intervals and significance tests are built on everything that came before. The normal distribution (Chapter 13) provides the curve from which z-values and areas are drawn. The concept of sampling variability (Chapters 3 and 13) explains why intervals are needed at all. Bias, reliability, and validity (Chapter 8) determine whether the inference is trustworthy in the first place: a significant result from a biased sample is still misleading.
statistical inference, parameter, statistic, sampling distribution, sampling variability, margin of error, confidence interval, confidence level, 95% confidence, null hypothesis, alternative hypothesis, p-value, statistical significance, significance level, alpha, Type I error, Type II error, false positive, false negative, practical significance, effect size, hypothesis testing, fail to reject, PSY 301, Moore and Notz, understanding statistics, final exam review