Difficulty: Intermediate to Advanced | Prerequisites: z-distribution, sampling distribution of a proportion, confidence intervals, Bayes' theorem
Hypothesis testing is the framework for making yes-or-no decisions from data: does a treatment work, or does it not? This material covers the full lifecycle of a test. You set up null and alternative hypotheses, find the critical value that controls your Type 1 error rate (false positives), compute the probability of a Type 2 error (false negatives) under a specific alternative, and determine how large a sample you need to achieve a desired level of power. The final piece ties in Bayes' theorem: after running a test and observing the outcome (reject or fail to reject), you can update your prior belief about which hypothesis is true. This is the most conceptually demanding material on the midterm.
A one-sided test has a critical value that marks the boundary between "fail to reject" and "reject." Type 2 error is the probability of missing a real effect; power is 1 minus that probability. You can solve for the sample size needed to hit a target power. After observing the test outcome, Bayes' rule updates your belief about the null.
Null hypothesis (H_0)
The default assumption, typically that there is no effect or no difference. In a proportion test, it specifies the value of the parameter under "no effect." Think of it as: the boring explanation you are trying to find evidence against.
Alternative hypothesis (H_a)
The claim you are trying to find evidence for. In a one-sided test, this specifies the direction of the effect (e.g., p > 0.2). In simple terms, this means: the interesting explanation you hope the data will support.
One-sided test (one-tailed test)
A test where the alternative hypothesis specifies a direction. The entire rejection region is in one tail of the distribution. Think of it as: you only care about detecting an effect in one direction.
Critical value (p-hat-star)
The threshold value of the test statistic (or sample proportion) beyond which you reject the null. In simple terms, this means: if your observed statistic exceeds this value, you reject H_0.
Type 1 error (alpha, false positive)
Rejecting the null hypothesis when it is actually true. The significance level alpha is the maximum probability of a Type 1 error that you are willing to tolerate. Think of it as: a false alarm.
Type 2 error (beta, false negative)
Failing to reject the null hypothesis when the alternative is actually true. In simple terms, this means: you missed a real effect because your data did not cross the critical value.
Statistical power
The probability of correctly rejecting the null when the alternative is true. Power = 1 - beta. Think of it as: the sensitivity of your test, its ability to detect a real effect.
Bayesian updating after a hypothesis test
Using Bayes' theorem to update your prior belief about whether H_0 is true after observing the test outcome (reject or fail to reject). In simple terms, this means: combining your prior beliefs with the test result to get an updated probability that the null is true.
Context: testing whether a coding bootcamp improves job-finding success beyond the baseline rate of 20%.
H_0: p_cb = 0.2 (the bootcamp has no effect).
H_a: p_cb > 0.2 (the bootcamp improves the rate).
This is a one-sided (right-tailed) test.
With n = 100 and alpha = 0.025 for a one-sided test, the z-value is 1.96 (the 0.975 quantile of the standard normal).
The variance of the sample proportion under H_0: sigma^2 = p_0(1 - p_0)/n = 0.2(0.8)/100 = 0.0016.
The critical value: p-hat-star = 0.2 + 1.96 * sqrt(0.0016) = 0.2 + 1.96 * 0.04 = 0.2784.
Interpretation: if the observed sample proportion exceeds 0.2784, reject H_0.
Suppose the true proportion is p = 0.3. A Type 2 error means failing to reject H_0 when p = 0.3 is the truth.
P(Type 2 error) = P(p-hat < 0.2784 | p = 0.3).
Under p = 0.3, the standard deviation of p-hat is sqrt(0.3 * 0.7 / 100) = 0.04583.
z = (0.2784 - 0.3) / 0.04583 = -0.47.
P(z < -0.47) = 0.3192.
So the probability of a Type 2 error is about 32%. Power = 1 - 0.3192 = 0.6808.
Prior beliefs: P(p_cb = 0.3) = 0.7 and P(p_cb = 0.2) = 0.3.
After running the test and failing to reject H_0, update using Bayes' rule.
P(fail to reject | p = 0.2) = 1 - alpha = 0.975 (the test correctly fails to reject a true null).
P(fail to reject | p = 0.3) = P(Type 2 error) = 0.3192.
P(fail to reject) = 0.975 * 0.3 + 0.3192 * 0.7 = 0.2925 + 0.2234 = 0.5159.
P(p = 0.2 | fail to reject) = (0.975 * 0.3) / 0.5159 = 0.5670.
Interpretation: even though you started with only a 30% belief that the bootcamp is useless, failing to reject raises that belief to about 57%.
Goal: 99% power when p = 0.3, using alpha = 0.025 one-sided.
Two conditions must be satisfied simultaneously for the same critical value p-hat-star.
Alpha condition: p-hat-star = 0.2 + 1.96 * sqrt(0.2 * 0.8 / n).
Power condition: we need P(z > -2.33) = 0.99, so p-hat-star = 0.3 - 2.33 * sqrt(0.3 * 0.7 / n).
Set these equal and solve for n:
0.2 + 1.96 * sqrt(0.16/n) = 0.3 - 2.33 * sqrt(0.21/n)
Solving gives n = 342.89, so you need at least 343 observations.
\hat{p}^* = p_0 + z_{\alpha} \cdot \sqrt{\frac{p_0(1 - p_0)}{n}} \quad \text{(critical value, one-sided)}\sigma_{\hat{p}}^2 = \frac{p(1-p)}{n} \quad \text{(variance of sample proportion)}P(\text{Type 2}) = P\!\left(z < \frac{\hat{p}^* - p_a}{\sqrt{p_a(1-p_a)/n}}\right)\text{Power} = 1 - P(\text{Type 2 error})\text{Sample size: set } p_0 + z_\alpha \sqrt{\frac{p_0(1-p_0)}{n}} = p_a - z_\beta \sqrt{\frac{p_a(1-p_a)}{n}} \text{ and solve for } nThis is exactly how clinical trials are designed: a pharmaceutical company decides on a significance level (typically 0.05), a clinically meaningful effect size, and a target power (typically 80% or 90%), then solves for the number of patients needed. The same logic applies to tech companies running A/B tests to decide whether a new feature improves conversion rates. The Bayesian updating piece appears in regulatory decision-making, where agencies combine prior evidence with new trial results.
Students often confuse the significance level alpha with the p-value. Alpha is the threshold you set before the test; the p-value is the probability of observing your data (or more extreme) under H_0.
A common error is using the wrong standard deviation when computing the Type 2 error. Under H_0 you use p_0 in the variance; under H_a you use p_a. The switch matters.
Students frequently think that failing to reject means the null is true. It does not. It means the evidence was insufficient to reject it. The Bayesian updating part of the exam drives this home: failing to reject merely updates your belief.
When solving for sample size, students often forget there are two simultaneous conditions (the alpha condition and the power condition) involving two different standard deviations (one under H_0, one under H_a).
⚠️ The critical value question is a calculation question with no room for conceptual hand-waving. Write out the formula, plug in, compute. Partial credit depends on showing each step.
⚠️ Type 2 error questions require switching from the null distribution to the alternative distribution. The standard deviation changes.
⚠️ The Bayesian updating question combines material from earlier in the course (Bayes' rule) with hypothesis testing. It appeared on this midterm and is a strong candidate for reappearing.
⚠️ Sample size problems require setting two expressions for p-hat-star equal. Make sure you use z_alpha with the H_0 variance and z_beta with the H_a variance. Do not mix them up.
True or False: In a one-sided test with alpha = 0.025, the z-value for the critical value is 1.96.
True (1.96 is the 0.975 quantile, leaving 0.025 in the upper tail).
Fill in the blank: Type 2 error is the probability of ___ the null when the alternative is ___.
Failing to reject; true.
True or False: Power = alpha.
False. Power = 1 - beta, where beta is the Type 2 error probability.
Fill in the blank: To increase power without changing alpha, you can increase ___.
The sample size.
True or False: Failing to reject H_0 proves H_0 is true.
False. It means the data were insufficient to reject it.
Q: A coding bootcamp claims to improve the job-finding rate above the baseline of 20%. With n = 100 and a one-sided test at alpha = 0.025, what is the critical value p-hat-star?
A: p-hat-star = 0.2 + 1.96 * sqrt(0.2 * 0.8 / 100) = 0.2 + 1.96 * 0.04 = 0.2784.
Q: If the true proportion is p = 0.3, what is the probability of a Type 2 error?
A: P(p-hat < 0.2784 | p = 0.3) = P(z < (0.2784 - 0.3)/sqrt(0.3*0.7/100)) = P(z < -0.47) = 0.3192.
Q: You believe there is a 70% chance p = 0.3 and a 30% chance p = 0.2. You run the test and fail to reject. What is the updated probability that p = 0.2?
A: P(p=0.2 | fail to reject) = P(fail to reject | p=0.2) * P(p=0.2) / P(fail to reject). P(fail to reject | p=0.2) = 0.975. P(fail to reject | p=0.3) = 0.3192. P(fail to reject) = 0.9750.3 + 0.31920.7 = 0.5159. Answer: 0.975*0.3 / 0.5159 = 0.5670.
Q: How large a sample do you need for 99% power when p = 0.3 and alpha = 0.025 (one-sided)?
A: Set 0.2 + 1.96sqrt(0.16/n) = 0.3 - 2.33sqrt(0.21/n) and solve. n = 342.89, so at least 343.
This material builds on confidence intervals (the critical value is the edge of a one-sided CI) and connects to the law of total probability and Bayes' rule from the first set of notes. Power analysis and sample size planning are prerequisites for experimental design, which is typically covered later in the course. The concept of trading off Type 1 and Type 2 errors also appears in machine learning (precision vs recall) and decision theory.
Hypothesis testing, null hypothesis, alternative hypothesis, one-sided test, one-tailed test, two-tailed test, critical value, rejection region, significance level, alpha, Type 1 error, false positive, Type 2 error, false negative, beta, statistical power, power analysis, sample size determination, effect size, proportion test, z-test for proportions, Bayesian hypothesis testing, posterior probability, prior probability, fail to reject