Difficulty: Intermediate–Advanced | Prerequisites: Z-tests, p-values, hypothesis-testing framework (see companion notes)
Beyond just rejecting or not rejecting H₀, you need to understand the errors you can make, how likely you are to detect a real effect (power), and how many observations you need. Bayesian posterior probability adds another lens: given the data, what is the probability the null is actually true? This material ties those ideas together and tests whether common intuitions about them hold up to formal proof.
Type I error (false positive)
Rejecting H₀ when it is actually true. The probability of a Type I error is α, the significance level you chose. Think of it as a false alarm.
Type II error (false negative)
Failing to reject H₀ when it is actually false. The probability of a Type II error is denoted β. Think of it as a missed detection.
Power (1 − β)
The probability of correctly rejecting H₀ when H₁ is true. Higher power means you are more likely to catch a real effect. In simple terms, power is the sensitivity of your test.
Prior probability, P(H₀)
In a Bayesian framework, your belief about the probability that H₀ is true before seeing any data. This is subjective and set before the experiment.
Likelihood, P(D|H)
The probability of observing the data D given that a specific hypothesis H is true. For H₀, this is related to the p-value. For H₁, it depends on the specific alternative.
Posterior probability, P(H₀|D)
The probability that H₀ is true after seeing the data. Computed via Bayes' theorem: P(H₀|D) = P(D|H₀) × P(H₀) / P(D). This is what many people mistakenly think the p-value tells them.
Sample size determination
Choosing n large enough so the test has a desired level of power against a specific alternative. Requires specifying α, the target power (1 − β), and the effect size you want to detect.
Every hypothesis test can produce two kinds of mistake. Understanding the tradeoff between them is central to this problem set.
You reject H₀ when it is true. The rate is set by your significance level α. Choosing α = 0.05 means you accept a 5% chance of falsely rejecting a true null.
You fail to reject H₀ when it is false. β depends on several things: the true parameter value, the sample size, and α itself. You do not choose β directly; it falls out of your test design.
For a fixed sample size and true parameter value, decreasing α (making the test more conservative) pushes the rejection threshold further out, which makes it harder to reject. That means β increases. You can reduce both errors simultaneously only by increasing the sample size.
In medical testing, a Type I error means approving a drug that does not work. A Type II error means failing to approve a drug that does work. The consequences are different, which is why context determines how you balance α and β.
Power = 1 − β = the probability of rejecting H₀ when H₁ is true. You want power to be high (conventionally at least 0.80).
Step 1: Find the critical values on the statistic scale under H₀.
Step 2: Compute the probability of the test statistic falling in the rejection region when the true parameter is the alternative value.
Two-sided test, H₀: p = 0.7, α = 0.05, n = 200. True value: p = 0.63.
From the companion notes, the critical values on the p̂ scale are approximately 0.6365 and 0.7635.
Under p = 0.63, p̂ is approximately normal with mean 0.63 and SE = √(0.63 × 0.37 / 200) ≈ 0.0341.
β = P(0.6365 ≤ p̂ ≤ 0.7635 | p = 0.63)
Standardise the lower boundary: Z_lower = (0.6365 − 0.63) / 0.0341 ≈ 0.19
Standardise the upper boundary: Z_upper = (0.7635 − 0.63) / 0.0341 ≈ 3.92
β ≈ P(Z ≤ 3.92) − P(Z ≤ 0.19) ≈ 1.0000 − 0.5753 ≈ 0.4247
Power = 1 − 0.4247 ≈ 0.5753
Interpretation: with n = 200, you have only about a 58% chance of detecting that the true on-time rate is 0.63, not 0.70. That is below the usual 80% target.
With n = 800, the SE under H₀ shrinks: SE₀ = √(0.7 × 0.3 / 800) ≈ 0.0162.
New critical values: p̂_lower ≈ 0.7 − 1.96 × 0.0162 ≈ 0.6682, p̂_upper ≈ 0.7 + 1.96 × 0.0162 ≈ 0.7318.
Under p = 0.63, SE = √(0.63 × 0.37 / 800) ≈ 0.0171.
Z_lower = (0.6682 − 0.63) / 0.0171 ≈ 2.24. Z_upper = (0.7318 − 0.63) / 0.0171 ≈ 5.95.
β ≈ P(Z ≤ 5.95) − P(Z ≤ 2.24) ≈ 1.0000 − 0.9875 ≈ 0.0125.
Power ≈ 0.9875.
Quadrupling the sample size from 200 to 800 raised power from about 58% to about 99%. More data means smaller standard errors, tighter critical values, and far better detection.
If you want a test with a specific power against a specific alternative, you can solve for the required sample size before collecting data.
n ≥ [(z_α + z_β) × σ / (μ₁ − μ₀)]²
where z_α is the critical value for the significance level, z_β is the z-value corresponding to the desired power, σ is the population standard deviation, and (μ₁ − μ₀) is the effect size you want to detect.
A doctor tests H₀: μ = 20 vs H₁: μ > 20 (one-sided, α = 0.05). σ = 8. The doctor wants at least 80% power to detect μ₁ = 23.
z_α = z_{0.05} = 1.645
z_β = z_{0.20} = 0.8416 (for 80% power, β = 0.20)
Effect size: μ₁ − μ₀ = 23 − 20 = 3
n ≥ [(1.645 + 0.8416) × 8 / 3]²
n ≥ [(2.4866) × 8 / 3]²
n ≥ [6.631]²
n ≥ 43.97
Round up: the doctor needs at least 44 patients.
Always round up when computing sample size. Rounding down gives you less power than you asked for.
Frequentist hypothesis testing gives you a p-value: P(data this extreme | H₀ true). Bayesian analysis flips the conditional: P(H₀ true | data). These are fundamentally different quantities.
P(H₀|D) = P(D|H₀) × P(H₀) / P(D)
where P(D) = P(D|H₀) × P(H₀) + P(D|H₁) × P(H₁).
P(D|H₀) is the p-value (the probability of the observed data under the null). P(H₀) is your prior belief. P(D|H₁) is the probability of the data under the alternative.
From Q2a, the p-value for the tutoring programme is approximately 0.0062, so P(D|H₀) ≈ 0.0062.
Given: P(H₀) = 0.5, P(H₁) = 0.5, P(D|H₁) = 0.25.
P(D) = 0.0062 × 0.5 + 0.25 × 0.5 = 0.0031 + 0.125 = 0.1281
P(H₀|D) = (0.0062 × 0.5) / 0.1281 = 0.0031 / 0.1281 ≈ 0.0242
With equal priors, the posterior probability of H₀ is about 2.4%. The data strongly shift your belief away from the null.
Now suppose a strong prior that the programme will not work: P(H₀) = 0.9, P(H₁) = 0.1, P(D|H₁) = 0.25.
P(D) = 0.0062 × 0.9 + 0.25 × 0.1 = 0.00558 + 0.025 = 0.03058
P(H₀|D) = (0.0062 × 0.9) / 0.03058 = 0.00558 / 0.03058 ≈ 0.1824
Even with a very sceptical prior (90% belief the programme does not work), the data bring P(H₀|D) down to about 18%. Compare this to part (b): the prior matters. A stronger prior in favour of H₀ means you need more evidence to overturn it, and the posterior is higher (18% vs 2.4%), though still well below 50%.
This section addresses the theoretical questions in the practice set. These are the kinds of claims an exam might ask you to prove or disprove.
For a one-sided test H₁: μ > μ₀ with fixed n and fixed true μ₁ > μ₀:
The rejection region is X̄ ≥ μ₀ + z_α × SE. Decreasing α increases z_α, which pushes the critical value to the right. Under the true distribution (centred at μ₁), the probability of falling in the rejection region shrinks. That probability is the power, so power decreases and β = 1 − power increases.
The same logic applies to two-sided tests and tests for proportions. The statement is true.
This is false. A counterexample: set P(H₀) = 0.99 and P(D|H₁) = 0.03.
Suppose the p-value = P(D|H₀) = 0.04 < α = 0.05.
P(D) = 0.04 × 0.99 + 0.03 × 0.01 = 0.0396 + 0.0003 = 0.0399
P(H₀|D) = (0.04 × 0.99) / 0.0399 = 0.0396 / 0.0399 ≈ 0.9925
The p-value is below α, yet the posterior probability of H₀ is about 99%. A strong prior can keep P(H₀|D) well above 0.5 even when the frequentist test rejects. The p-value and the posterior answer different questions.
For a fixed α and fixed true parameter value different from the null:
Increasing n decreases the standard error SE = σ / √n. A smaller SE makes the sampling distribution under H₁ more concentrated. The critical value (on the X̄ scale) moves closer to μ₀ because the rejection threshold tightens. The probability that X̄ falls in the rejection region under H₁ increases. Power rises.
Formally, for a right-tailed test, power = P(Z > z_α − (μ₁ − μ₀)/(σ/√n)). As n increases, the term (μ₁ − μ₀)/(σ/√n) grows, so the argument of P(Z > ·) decreases, and the probability increases. The statement is true.
The Type I error rate is α, which you choose. The Type II error rate β depends on the true parameter, sample size, and α.
Counterexample: set α = 0.10 and suppose the true mean is very far from μ₀ with a large sample. β could be essentially 0 (power ≈ 1). Then α = 0.10 > β ≈ 0.
The statement makes no general claim that holds. α can be larger than, equal to, or smaller than β depending on the specifics.
Power (one-sided, right-tailed z-test for a mean)
Power = P(Z > z_α − (μ₁ − μ₀) / (σ / √n))
Type II error rate
β = 1 − Power
Sample size for desired power (one-sided z-test)
n ≥ [(z_α + z_β) × σ / (μ₁ − μ₀)]²
Bayes' theorem for two hypotheses
P(H₀|D) = P(D|H₀) × P(H₀) / [P(D|H₀) × P(H₀) + P(D|H₁) × P(H₁)]
Standard error under the alternative (proportion)
SE_alt = √[p_true × (1 − p_true) / n]
Key z-values for power calculations
80% power: z_β = z_{0.20} = 0.8416
90% power: z_β = z_{0.10} = 1.2816
95% power: z_β = z_{0.05} = 1.645
Students often think power is a fixed property of the test. It is not. Power depends on the true parameter value, n, α, and σ. A test can have high power against one alternative and low power against another.
Students confuse the p-value with the Bayesian posterior P(H₀|D). Q5b shows these can diverge dramatically when the prior is strong.
Students sometimes assume that a larger sample always makes α smaller. Sample size does not change α; you set α yourself. A larger sample reduces β and increases power.
Students forget to round up when computing minimum sample size. Rounding down gives strictly less than the target power.
⚠️ Power and sample-size calculations are high-value exam questions. Memorise the sample-size formula and practise plugging in z-values.
⚠️ Be ready to compute P(H₀|D) using Bayes' theorem with different priors. The exam may ask you to compare results across two priors, as in Q2b vs Q2c.
⚠️ Q5-style "prove or give a counterexample" questions test deep understanding. For false statements, a single numerical counterexample is enough. For true statements, sketch the algebra showing the direction of the effect.
⚠️ Know the difference between what the p-value measures and what the posterior measures. This is one of the most commonly tested conceptual distinctions in introductory statistics.
True or false: Power increases as the true parameter moves further from the null value.
True. A larger effect is easier to detect.
Fill in the blank: To compute P(H₀|D), you need the p-value, a ______, and the likelihood under H₁.
Prior probability, P(H₀).
True or false: Decreasing α from 0.05 to 0.01 always increases power.
False. Decreasing α makes the test more conservative, which decreases power (increases β).
Fill in the blank: When computing the minimum sample size, always round ______.
Up.
True or false: If the p-value is 0.03, there is a 97% chance the alternative hypothesis is true.
False. The p-value does not tell you the probability that either hypothesis is true. That requires Bayes' theorem and a prior.
Q: A researcher tests H₀: μ = 50 vs H₁: μ > 50 at α = 0.05 with σ = 10 and n = 64. The true mean is 53. What is the power of this test?
A: SE = 10/√64 = 1.25. Critical value on X̄ scale: 50 + 1.645 × 1.25 = 52.056. Under μ = 53: Z = (52.056 − 53)/1.25 = −0.755. Power = P(Z > −0.755) ≈ 0.775, or about 77.5%.
Q: In the question above, what sample size is needed for 90% power?
A: n ≥ [(1.645 + 1.2816) × 10 / 3]² = [(2.9266) × 10 / 3]² = [9.755]² ≈ 95.17. Round up: n = 96.
Q: Suppose P(H₀) = 0.7, P(D|H₀) = 0.02, P(D|H₁) = 0.15. Find P(H₀|D).
A: P(D) = 0.02 × 0.7 + 0.15 × 0.3 = 0.014 + 0.045 = 0.059. P(H₀|D) = 0.014 / 0.059 ≈ 0.237. Despite starting with a 70% prior for H₀, the data bring the posterior down to about 24%.
Q: True or false, with justification: for a two-sided test at α = 0.05, if n increases from 100 to 400, the critical values of p̂ move closer to p₀.
A: True. The critical values are p₀ ± 1.96 × √[p₀(1−p₀)/n]. As n increases, the SE shrinks, so the interval around p₀ tightens. The rejection region starts closer to the null, making it easier to reject.
Power analysis connects directly to study design: before running any experiment, researchers use these calculations to determine whether their sample will be large enough. The Bayesian material here is a bridge to Bayesian statistics more broadly, which reverses the direction of inference from "probability of data given hypothesis" to "probability of hypothesis given data." The tradeoff between α and β reappears whenever you study decision theory, classification (false positives vs false negatives in machine learning), or diagnostic testing in medicine.
power, Type I error, Type II error, alpha, beta, false positive, false negative, sample size determination, power analysis, Bayes theorem, posterior probability, prior probability, likelihood, p-value interpretation, hypothesis testing errors, effect size, z-test power, NYU Analytical Statistics, ASTATS, PS5, Timothy Roeper, MTA subway trains, tutoring programme GPA, urgent care wait time