Source: Mathematical Statistics, University of Central Florida
Tags: statistical power, effect size, sample size, power tables, alpha, beta, one-tailed t-test, independent samples, power trade-offs, increasing sample size
Difficulty: Intermediate | Prerequisites: Hypothesis testing errors and the decision matrix (see Part 1 notes), basic understanding of sampling distributions.
Once you understand Type I and Type II errors, the natural next question is: how do you control them? Power analysis is the tool for that. It connects four quantities, alpha (α), power (1 − β), effect size, and sample size, so that knowing any three lets you solve for the fourth. This is how researchers decide how many participants they need before collecting data. It is also how you evaluate whether a study that failed to find an effect was actually capable of detecting one. Nearly every research proposal and ethics application requires a power analysis, so this is not just exam material; it is a professional skill.
Power is the probability that your test will detect an effect if one truly exists. It depends on three things you can influence: your significance level (α), your sample size (n), and the size of the effect you are looking for. Larger samples, larger effects, and more lenient α levels all increase power, but each trade-off has consequences.
Statistical power (1 − β)
The probability of correctly rejecting a false null hypothesis. Conventionally, researchers aim for power of at least .80. Think of it as: your test's sensitivity, its ability to pick up a real signal.
Effect size
A standardised measure of how large the difference or relationship is in the population. For a t-test, this is often Cohen's d. In simple terms, this means: how big is the thing you are trying to detect? A larger effect is easier to spot.
Sample size (n)
The number of observations or participants in the study. Larger n gives the sampling distribution a smaller standard error, making it easier to detect effects. Think of it as: the amount of information you are working with.
Power table
A reference table that cross-references effect size, sample size, α, and power so you can look up any one of them given the other three. Commonly used for t-tests, F-tests, and chi-square tests. In simple terms, this means: a lookup chart that tells you how many participants you need, or how much power you have, for a given scenario.
Cohen's d (effect size conventions)
A common standardised effect size for comparing two means. Small = 0.2, medium = 0.5, large = 0.8. These are rough benchmarks, not rules. Think of it as: a ruler for "how different are these two groups?" that does not depend on the original measurement scale.
Power, effect size, sample size, and α are all linked. Change one and at least one other must change too (holding the rest constant).
A power analysis lets you solve for whichever quantity is unknown, typically sample size during study planning.
Power = 1 − β, so as β increases, power decreases. They move in opposite directions.
A higher β means a higher probability of missing a real effect.
Increasing α (e.g., from .01 to .05) moves the cutoff to the left on the null distribution.
This makes the rejection region larger, so it becomes easier to reject the null.
In the alternative distribution, the area beyond the cutoff (power) gets larger, and the area below the cutoff (β) gets smaller.
So: increasing α increases power, but at the cost of a higher false-positive rate.
α increases (larger rejection region in the null distribution).
1 − α decreases.
β decreases (smaller miss region in the alternative distribution).
1 − β (power) increases.
It becomes easier to reject the null.
Power increases.
This is the fundamental α-power trade-off: making it easier to reject the null catches more real effects but also allows more false alarms.
If the effect size increases (the two distributions move further apart) while α stays constant:
β decreases, because less of the alternative distribution falls below the cutoff.
1 − β (power) increases.
Larger effects are simply easier to detect with the same test setup.
How Sam is right: increasing sample size reduces the standard error of the sampling distributions. The curves get narrower and their overlap shrinks, which increases power. With enough participants, you can detect even small effects.
How Sam is wrong: more participants are not always better. Very large samples can detect trivially small effects that have no practical importance, leading to "statistically significant" results that are meaningless in context. There are also ethical and practical considerations: collecting more data than needed wastes participant time, money, and resources. If an effect is large, a smaller sample may be perfectly adequate.
Finding sample size (given power and effect size):
Power = .8, effect size = .6: look up the intersection in the power table. For a one-tailed independent-samples t-test at α = .05, you need approximately n = 36 per group.
Power = .9, effect size = .4: a smaller effect with higher power demands more participants, approximately n = 133 per group.
(Exact values depend on the specific table provided in your course materials. The key principle: smaller effect sizes and higher desired power both require larger samples.)
Finding power (given effect size and sample size):
Effect size = .6, n = 60: with a medium-to-large effect and a decent sample, power will be high, likely above .95.
Effect size = .3, n = 200: a small effect with a large sample also yields high power, likely around .95 or above.
(Again, consult your specific course power tables for exact values. The logic: large n compensates for small effect size.)
Change | Effect on power | Side effect |
|---|---|---|
Increase α | Power increases | More false positives |
Decrease α | Power decreases | Fewer false positives |
Increase n | Power increases | Greater cost, risk of detecting trivial effects |
Increase effect size (in reality) | Power increases | Not under researcher's control |
Increase β | Power decreases | More missed real effects |
Core relationship:
Power = 1 − β
Standard error of the mean (why n matters):
SE = σ / √n
As n increases, SE shrinks, the sampling distributions become narrower, overlap decreases, and power rises.
Cohen's d:
d = (M₁ − M₂) / σ_pooled
Where M₁ and M₂ are the two group means and σ_pooled is the pooled standard deviation.
Effect size benchmarks (Cohen's conventions for d):
Small: d = 0.2
Medium: d = 0.5
Large: d = 0.8
Before running a clinical trial, researchers are required to do a power analysis to justify their sample size to ethics boards and funding agencies. An underpowered study wastes participants' time and may expose them to risk without a realistic chance of yielding useful results. An overpowered study wastes resources detecting clinically meaningless differences. Getting the sample size right is a practical, ethical decision, not just a statistical exercise.
"More participants are always better." Statistically, power does keep increasing with n, but practically, very large samples detect effects so tiny they have no real-world importance. Researchers need to balance power with practical significance.
"Increasing α is bad and should never be done." A stricter α reduces false positives but also reduces power. In some exploratory contexts, a more lenient α (e.g., .10) is a defensible choice. The right α depends on the costs of each error type in your specific situation.
"Effect size is something the researcher chooses." The researcher chooses the minimum effect size they want to be able to detect, but the true effect size in the population is fixed and unknown. Power analysis uses the planned detectable effect size, not a preferred outcome.
"Low power means the null is probably true." A non-significant result from an underpowered study tells you very little. The test may simply have been unable to detect the effect. You cannot interpret a failure to reject as evidence for the null unless power was high.
⚠️ Expect to be given two of the three inputs (power, effect size, sample size) and asked to find the third using a power table.
⚠️ "Explain how Sam is right and how Sam is wrong about more participants being better" is directly from this assignment and tests whether you understand both the statistical benefit and the practical limits of increasing n.
⚠️ Know the direction of every trade-off: what happens to power when α increases? When effect size increases? When n increases? These are quick, testable facts.
⚠️ Be able to explain what happens to α, β, 1 − α, and 1 − β when the cutoff moves left or right. Draw the two-distribution diagram and trace through it.
1. True or False: Increasing α from .01 to .05 decreases power. False. Increasing α increases power (but also increases the Type I error rate).
2. Fill in the blank: Power = 1 − ______. β
3. True or False: A very large sample size can detect effects that are too small to matter practically. True. This is one reason "more participants" is not always better.
4. Fill in the blank: If the effect size increases while α stays the same, β ______ and power ______. β decreases; power increases.
5. True or False: α and β are always inversely related, so decreasing α always increases β. True (holding sample size and effect size constant). Making it harder to reject the null reduces false positives but increases the chance of missing a real effect.
Q: Sam says "The more participants, the better!" Explain how he is right in one way and wrong in another.
A: He is right because increasing sample size reduces standard error, which narrows the sampling distributions, reduces their overlap, and increases power. He is wrong because excessively large samples can detect trivially small effects that have no practical importance, and collecting more data than necessary wastes time, money, and participant effort.
Q: If we move our cutoff (α) to the left, what happens to α, β, 1 − α, and 1 − β?
A: α increases (larger rejection region), 1 − α decreases, β decreases (less of the alternative distribution falls below the cutoff), and 1 − β (power) increases. It becomes easier to reject the null, and power increases.
Q: If effect size increases while α stays constant, what happens to β and 1 − β?
A: β decreases and power (1 − β) increases. The alternative distribution shifts further from the null, so less of it falls below the cutoff.
Q: Using power tables for a one-tailed t-test (α = .05) for independent samples, how would you determine the required sample size if power = .8 and effect size = .6?
A: Find the row for effect size .6 and the column for power .8 in the one-tailed independent-samples table at α = .05. The cell gives the required n per group (approximately 36 per group, depending on the specific table).
Q: How does increasing β affect power?
A: Power = 1 − β, so increasing β directly decreases power. A higher β means a greater probability of failing to detect a real effect.
Q: How does increasing α affect power?
A: Increasing α moves the critical cutoff toward the centre of the null distribution, expanding the rejection region. This means more of the alternative distribution exceeds the cutoff, so power (1 − β) increases. The cost is a higher false-positive rate.
Power analysis connects backwards to everything in hypothesis testing (you need to understand the decision framework before you can optimise it) and forwards to research design and methodology. When you study ANOVA later, you will see power tables for F-tests that work on the same principles but with different effect size metrics (Cohen's f instead of d). In regression, power depends on the number of predictors and expected R². The core logic, that power is a function of α, n, and effect size, remains the same across all parametric tests.
statistical power, power analysis, effect size, sample size, Cohen's d, power tables, alpha level, beta, one-tailed t-test, independent samples t-test, Type I error, Type II error, significance level, standard error, sampling distribution, underpowered study, overpowered study, research design, UCF mathematical statistics, HW 7, power trade-offs