Difficulty: Advanced | Prerequisites: Sampling distributions, confidence intervals, normal approximation to the binomial, basic probability and Bayes' theorem.
Hypothesis testing formalises the question "is this effect real or could it be noise?" Once you can run a test, the natural follow-up questions are: how often does the test miss a real effect (Type II error), how do you design a study powerful enough to catch it (sample size for power), and how do you update your beliefs when the test result comes in (Bayesian updating). Q4 on the practice midterm walks through all four layers using a single example, the coding bootcamp problem.
You set up a null hypothesis, find the sample statistic that would just barely reject it, then ask how likely you are to land beyond that threshold under the alternative. If the test fails to reject, Bayes' theorem lets you update your prior beliefs about which hypothesis is true. Increasing sample size raises power.
Null hypothesis (H0)
The default claim you test against. In the bootcamp problem: p_cb = 0.2 (the bootcamp does not help). You assume H0 is true unless the data provide strong enough evidence to reject it.
Alternative hypothesis (H1 or Ha)
The claim you are trying to find evidence for. Here: p_cb > 0.2 (one-sided, the bootcamp helps).
One-sided (one-tailed) test
A test where the rejection region sits entirely in one tail of the distribution. You reject only if the sample statistic is sufficiently large (or small) in one direction. Think of it as: you only care about detecting an effect in one direction.
Type I error (alpha)
Rejecting H0 when it is true. The significance level alpha is the maximum probability of a Type I error that you are willing to tolerate.
Type II error (beta)
Failing to reject H0 when it is false. This is the probability of missing a real effect.
Power
1 - beta. The probability of correctly rejecting H0 when a specific alternative is true. Higher power means a better chance of detecting a real effect.
Critical value (p-hat-star)
The threshold value of the test statistic above which you reject H0. For a one-sided proportion test, it is the smallest p-hat that leads to rejection.
Prior probability
Your belief about the probability of each hypothesis before seeing the data. In Q4(c): P(p_cb = 0.3) = 0.70 and P(p_cb = 0.2) = 0.30.
Posterior probability
Your updated belief after observing the data (or the test outcome). Computed via Bayes' theorem.
Without the bootcamp, 20% of people find a coding job (p = 0.2).
H0: p_cb = 0.2. H1: p_cb > 0.2 (one-sided).
n = 100, alpha = 0.025.
Under H0, p-hat is approximately normal: N(0.2, 0.2 * 0.8 / 100) = N(0.2, 0.0016). So SE = 0.04.
For a one-sided test at alpha = 0.025, the critical z-value is 1.96.
p-hat-star = 0.2 + 1.96 * 0.04 = 0.2 + 0.0784 = 0.2784.
Reject H0 if the observed p-hat exceeds 0.2784.
Under the alternative p_cb = 0.3, p-hat is approximately N(0.3, 0.3 * 0.7 / 100) = N(0.3, 0.0021). SE = sqrt(0.0021) ≈ 0.0458.
Type II error = P(fail to reject | p_cb = 0.3) = P(p-hat < 0.2784 | p_cb = 0.3).
z = (0.2784 - 0.3) / 0.0458 = -0.0216 / 0.0458 ≈ -0.47.
P(Z < -0.47) ≈ 0.319.
So beta ≈ 0.32, and power ≈ 0.68.
Prior: P(p_cb = 0.3) = 0.70, P(p_cb = 0.2) = 0.30.
Event: fail to reject H0 (FTR).
P(FTR | p_cb = 0.2) = 1 - alpha = 1 - 0.025 = 0.975 (under the null, you almost always fail to reject).
P(FTR | p_cb = 0.3) = beta ≈ 0.319 (the Type II error rate from part b).
P(FTR) = P(FTR | p_cb = 0.3) * 0.70 + P(FTR | p_cb = 0.2) * 0.30 = 0.319 * 0.70 + 0.975 * 0.30 = 0.2233 + 0.2925 = 0.5158.
Posterior: P(p_cb = 0.2 | FTR) = P(FTR | p_cb = 0.2) * P(p_cb = 0.2) / P(FTR) = 0.975 * 0.30 / 0.5158 ≈ 0.567.
Failing to reject raises your belief that p_cb = 0.2 from 30% to about 57%.
You want P(p-hat > p-hat-star | p_cb = 0.3) = 0.99.
The critical value under H0: p-hat-star = 0.2 + z_alpha * sqrt(0.2 * 0.8 / n) = 0.2 + 1.96 * sqrt(0.16/n).
Under H1: P(p-hat > p-hat-star) = P(Z > (p-hat-star - 0.3) / sqrt(0.21/n)) = 0.99.
This means (p-hat-star - 0.3) / sqrt(0.21/n) = -2.326 (the z-value for the 1st percentile).
Substituting and solving for n: set up the equation 0.2 + 1.96 * sqrt(0.16/n) - 0.3 = -2.326 * sqrt(0.21/n).
Simplify: -0.1 + 1.96 * sqrt(0.16/n) = -2.326 * sqrt(0.21/n).
Let u = 1/sqrt(n): -0.1 + 1.96 * 0.4 * u = -2.326 * sqrt(0.21) * u.
-0.1 + 0.784u = -1.066u.
-0.1 = -1.066u - 0.784u = -1.85u.
u = 0.1 / 1.85 = 0.054. So sqrt(n) = 1/0.054 ≈ 18.5, and n ≈ 342.
You need approximately 342 participants for 99% power.
Standard error for a proportion: SE = sqrt(p * (1 - p) / n)
Critical value for a one-sided test (upper tail): p-hat-star = p0 + z_alpha * sqrt(p0 * (1 - p0) / n)
Type II error (beta) at a specific alternative p1: beta = P(p-hat < p-hat-star | p = p1) = Phi((p-hat-star - p1) / sqrt(p1 * (1 - p1) / n))
Power: Power = 1 - beta
Bayesian update after observing test outcome: P(H0 | FTR) = P(FTR | H0) * P(H0) / [P(FTR | H0) * P(H0) + P(FTR | H1) * P(H1)] where P(FTR | H0) = 1 - alpha and P(FTR | H1) = beta
Sample size for desired power (one-sided proportion test): Solve: z_alpha * sqrt(p0 * q0 / n) + z_beta * sqrt(p1 * q1 / n) = p1 - p0 which gives n = [(z_alpha * sqrt(p0 * q0) + z_beta * sqrt(p1 * q1)) / (p1 - p0)] squared
Students confuse the SE used under H0 (based on p0) with the SE used under H1 (based on p1). The critical value uses the null SE; the power calculation uses the alternative SE.
Students think "failing to reject H0" means "H0 is true." It does not. It means the data were not extreme enough to rule H0 out at the chosen alpha level.
Students apply two-sided z-values (1.96) when the problem specifies a one-sided test at alpha = 0.025. For a one-sided test at alpha = 0.025, the critical z is also 1.96 (since 0.025 in one tail corresponds to z = 1.96). This coincidence causes confusion: the same z-value arises from a two-sided 5% test and a one-sided 2.5% test.
Students forget that Bayesian updating requires a prior. Without prior probabilities for H0 and H1, you cannot compute a posterior.
⚠️ Q4 is worth 28 points and builds sequentially: each part uses the answer from the one before. If you get the critical value wrong in (a), the rest cascades. Double-check p-hat-star.
⚠️ The Bayesian updating part (c) is the trickiest conceptually. You need to condition on the event "fail to reject," not on a specific data value. The likelihoods are (1 - alpha) under H0 and beta under H1.
⚠️ For the sample-size calculation (d), you will need to set up and solve an equation with two different standard errors (one under H0, one under H1). Show your algebra.
⚠️ One-sided test at alpha = 0.025: the z-critical is 1.96, the same number as a two-sided test at alpha = 0.05. The exam may test whether you understand why.
True or false: Power increases as the true effect size increases (all else equal).
Fill in the blank: If alpha = 0.025 in a one-sided test and you fail to reject, P(FTR | H0 is true) = ______.
True or false: Increasing the sample size reduces both Type I and Type II error rates.
Fill in the blank: After a Bayesian update, the posterior probability of H0 is always ______ than the prior if you fail to reject.
Answers: 1. True. 2. 0.975. 3. False (alpha is fixed by the researcher; increasing n reduces only Type II error). 4. Higher (failing to reject is evidence in favour of H0).
Q: H0: p_cb = 0.2, H1: p_cb > 0.2, n = 100, alpha = 0.025. Find the critical value p-hat-star.
A: SE under H0 = sqrt(0.2 * 0.8 / 100) = 0.04. z_(0.025) = 1.96. p-hat-star = 0.2 + 1.96 * 0.04 = 0.2784.
Q: If p_cb = 0.3, what is the probability of a Type II error?
A: SE under H1 = sqrt(0.3 * 0.7 / 100) ≈ 0.0458. z = (0.2784 - 0.3) / 0.0458 ≈ -0.47. beta = P(Z < -0.47) ≈ 0.319.
Q: Prior: P(p_cb = 0.3) = 0.70, P(p_cb = 0.2) = 0.30. You fail to reject H0. What is the posterior P(p_cb = 0.2)?
A: P(FTR | p_cb = 0.2) = 0.975. P(FTR | p_cb = 0.3) ≈ 0.319. P(FTR) = 0.319 * 0.70 + 0.975 * 0.30 ≈ 0.516. Posterior = 0.975 * 0.30 / 0.516 ≈ 0.567.
Q: How large must n be for 99% power when p_cb = 0.3 and alpha = 0.025?
A: n = [(z_alpha * sqrt(p0 * q0) + z_beta * sqrt(p1 * q1)) / (p1 - p0)] squared = [(1.96 * sqrt(0.16) + 2.326 * sqrt(0.21)) / 0.1] squared = [(1.96 * 0.4 + 2.326 * 0.458) / 0.1] squared = [(0.784 + 1.065) / 0.1] squared = [18.49] squared ≈ 342.
The structure of this problem, testing a proportion then updating with Bayes, mirrors the mixture-distribution question in the Confidence Intervals notes (Q1 parts c and d). Power analysis connects to sample-size planning in experimental design. The trade-off between alpha and beta is central to understanding why researchers pick conventional significance levels and why underpowered studies are a problem in published research.
hypothesis testing, null hypothesis, alternative hypothesis, one-sided test, one-tailed test, Type I error, Type II error, alpha, beta, power, power analysis, sample size calculation, critical value, rejection region, p-hat, proportion test, normal approximation to binomial, Bayesian updating, prior probability, posterior probability, Bayes' theorem, fail to reject, significance level