Difficulty: Intermediate | Prerequisites: Sampling distributions, z-scores, basic confidence intervals, one-sample t-test.
These notes cover the mechanics of hypothesis testing beyond the reject/fail-to-reject decision: computing critical cutoffs, finding the probability of a Type II error (beta), understanding power, and applying Bayes' theorem to interpret test outcomes. The material draws on Problems 2, 3, and 4 of the NYU Analytical Statistics Exam 2 practice set.
A hypothesis test gives you a binary decision (reject or fail to reject), but the story does not end there. You can compute the probability of failing to reject when the alternative is true (Type II error, beta), find the power of the test (1 - beta), and even use Bayes' theorem to find the posterior probability that you made an error. These notes walk through all of that for proportion tests, paired-sample tests, and z-tests with known variance.
Type I error (alpha)
Rejecting H0 when it is true. The significance level alpha is the maximum probability of a Type I error that you are willing to tolerate. In simple terms: a false alarm. You conclude there is an effect when there is none.
Type II error (beta)
Failing to reject H0 when it is false. The probability beta depends on the true parameter value, the sample size, and alpha. Think of it as: missing a real effect. The signal was there, but the test was not sensitive enough to catch it.
Power (1 - beta)
The probability of correctly rejecting H0 when it is false. Higher power means a better chance of detecting a real effect.
Critical cutoff / rejection region
The value(s) of the test statistic, or equivalently of the sample statistic, beyond which you reject H0. For a one-sided test at alpha = 0.0228, the z-cutoff is z = 2.00.
Bayesian posterior probability of error
Using Bayes' theorem, you can combine the prior probability that H0 is true with the likelihood of the observed test outcome to find the posterior probability that you made a Type II error (or a Type I error).
Paired samples / matched pairs
When each observation in one group is naturally paired with an observation in the other (e.g. pretest and post-test on the same student). The analysis is done on the differences D_i = X_2i - X_1i, reducing the problem to a one-sample test on the mean difference.
Measurement error
Random noise added to the true score by the measurement process. If scores are self-reported with error, the observed variance of the differences increases, which inflates the standard error and reduces power.
Efficiency of an estimator
The reciprocal of the variance of the estimator. A more efficient estimator has smaller variance and thus gives more precise estimates for a given sample size. When comparing two study designs, the one with smaller variance for the estimator of interest is more efficient.
The setup: current pass rate p_0 = 0.70. H0: p = 0.70, H1: p > 0.70. n = 336, alpha = 0.0228.
Under H0, the sampling distribution of p-hat is approximately normal:
Mean = p_0 = 0.70
SE = sqrt(p_0(1 - p_0)/n) = sqrt(0.70 * 0.30 / 336) = sqrt(0.000625) = 0.025
For a one-sided test at alpha = 0.0228, we need the z-value such that P(Z > z) = 0.0228. From the standard normal table, z = 2.00.
Critical cutoff in terms of p-hat:
p-hat* = p_0 + z * SE = 0.70 + 2.00 * 0.025 = 0.70 + 0.05 = 0.75
Reject H0 if p-hat >= 0.75. In terms of count: at least 0.75 * 336 = 252 students must pass.
Under p = 0.80, the sampling distribution of p-hat has:
Mean = 0.80
SE = sqrt(0.80 * 0.20 / 336) = sqrt(0.000476) ≈ 0.02182
Beta = P(fail to reject | p = 0.80) = P(p-hat < 0.75 | p = 0.80)
z = (0.75 - 0.80) / 0.02182 = -0.05 / 0.02182 ≈ -2.29
Beta = P(Z < -2.29) ≈ 0.0110
The probability of a Type II error is about 1.1%, so the power against p = 0.80 is about 98.9%.
Given:
Prior P(H0 true) = 0.75, Prior P(p = 0.80) = 0.25
We observe: fail to reject.
We need P(Type II error | fail to reject), which is P(p = 0.80 | fail to reject).
By Bayes' theorem:
P(p = 0.80 \mid \text{FTR}) = \frac{P(\text{FTR} \mid p = 0.80) \cdot P(p = 0.80)}{P(\text{FTR})}Compute each piece:
P(FTR | H0 true) = 1 - alpha = 1 - 0.0228 = 0.9772
P(FTR | p = 0.80) = beta = 0.0110
P(FTR) = P(FTR | H0) * P(H0) + P(FTR | p = 0.80) * P(p = 0.80) = 0.9772 * 0.75 + 0.0110 * 0.25 = 0.7329 + 0.00275 = 0.73565
Posterior:
P(p = 0.80 | FTR) = (0.0110 * 0.25) / 0.73565 = 0.00275 / 0.73565 ≈ 0.00374
So the posterior probability of having made a Type II error, given that you failed to reject, is about 0.37%. The failure to reject is very likely correct.
The setup: n = 625 students, D_i = S_2i - S_1i, Var(D_i) = 64, H0: mu_D = 0, H1: mu_D > 0, alpha = 0.0228.
The standard deviation of D-bar:
SD(D-bar) = sqrt(Var(D_i)/n) = sqrt(64/625) = 8/25 = 0.32
This asks for the probability that a symmetric interval of width 0.8 around D-bar contains mu_D. Equivalently:
P(|D-bar - mu_D| < 0.4) = P(-0.4/0.32 < Z < 0.4/0.32) = P(-1.25 < Z < 1.25)
= 2 * Phi(1.25) - 1 = 2(0.8944) - 1 = 0.7888
The probability is about 0.7888, or roughly 79%.
First, find the critical cutoff. Under H0, D-bar ~ N(0, 0.32^2). For a one-sided test at alpha = 0.0228:
z = 2.00, so D-bar* = 0 + 2.00 * 0.32 = 0.64
Reject H0 if D-bar >= 0.64.
Under mu_D = 0.8:
Beta = P(D-bar < 0.64 | mu_D = 0.8)
z = (0.64 - 0.80) / 0.32 = -0.16 / 0.32 = -0.50
Beta = P(Z < -0.50) = 0.3085
The probability of a Type II error is about 30.85%. Power = 1 - 0.3085 = 0.6915.
Self-reported scores: R_1i = S_1i + M_1i, R_2i = S_2i + M_2i.
The observed difference is:
R_2i - R_1i = (S_2i - S_1i) + (M_2i - M_1i) = D_i + (M_2i - M_1i)
Since M terms are independent of S terms:
Var(R_2i - R_1i) = Var(D_i) + Var(M_2i - M_1i)
Var(M_2i - M_1i) = Var(M_2i) + Var(M_1i) - 2Cov(M_1i, M_2i) = 25 + 25 - 2(7) = 36
Var(R_2i - R_1i) = 64 + 36 = 100
Standard deviation of the sample mean of observed differences:
SD = sqrt(100/625) = 10/25 = 0.40
New critical cutoff at alpha = 0.0228:
D-bar* = 0 + 2.00 * 0.40 = 0.80
Note how measurement error inflated the standard error from 0.32 to 0.40, pushing the critical cutoff from 0.64 to 0.80 and making it harder to detect a real effect.
With n = 256 and Var(D_i) = 64 (no measurement error):
Var(D-bar) = 64/256 = 0.25
With n = 625 and measurement error (from part c):
Var(D-bar) = 100/625 = 0.16
The n = 625 design with measurement error has smaller variance (0.16 < 0.25), so it is more efficient despite the noisy measurements. The larger sample size more than compensates for the added measurement variance.
Study 1: n = 625, Var(D-bar_1) = 100/625 = 0.16 (with measurement error). Study 2: n = 256, Var(D-bar_2) = 64/256 = 0.25 (perfect scores).
Both D-bar_1 and D-bar_2 are unbiased for mu_D. A classmate proposes the simple average of all 881 observations, but the observations have different variances (100 vs 64), so equal weighting is not optimal.
The more efficient unbiased estimator uses inverse-variance weighting:
\hat{\mu}_D = \frac{w_1 \bar{D}_1 + w_2 \bar{D}_2}{w_1 + w_2}, \quad w_i = \frac{1}{\text{Var}(\bar{D}_i)}w_1 = 1/0.16 = 6.25, w_2 = 1/0.25 = 4.00
Weights: w_1/(w_1 + w_2) = 6.25/10.25 ≈ 0.6098, w_2/(w_1 + w_2) = 4.00/10.25 ≈ 0.3902
Variance of the optimal estimator:
Var = 1/(w_1 + w_2) = 1/10.25 ≈ 0.09756
This is smaller than either study's variance alone (0.16 or 0.25) and smaller than what you would get from a simple equal-weighted average of the 881 observations.
The setup: claimed mean mu_0 = 16 oz, n = 36, x-bar = 15.6, sigma = 1.2 (known).
Standard error: SE = sigma / sqrt(n) = 1.2 / sqrt(36) = 1.2 / 6 = 0.20.
For a 95% CI with known sigma, use z_(0.025) = 1.96.
CI = x-bar +/- z * SE = 15.6 +/- 1.96 * 0.20 = 15.6 +/- 0.392
95% CI: (15.208, 15.992)
Note that 16 is not in this interval, which suggests the true mean is below the claimed 16 oz.
Test statistic:
z = (x-bar - mu_0) / SE = (15.6 - 16) / 0.20 = -0.4 / 0.20 = -2.00
For a left-tailed test at alpha = 0.05, the critical value is z = -1.645.
Since z = -2.00 < -1.645, we reject H0. There is sufficient evidence that the true mean fill volume is below 16 oz.
Reject H0 if x-bar < mu_0 + z_(alpha) * SE, where z_(alpha) = -1.645 for a left-tailed test.
x-bar* = 16 + (-1.645)(0.20) = 16 - 0.329 = 15.671
Reject H0 if x-bar < 15.671.
Since our observed x-bar = 15.6 < 15.671, this confirms the rejection.
Beta = P(fail to reject | mu = 15.4) = P(x-bar >= 15.671 | mu = 15.4)
Under mu = 15.4, x-bar ~ N(15.4, 0.20^2).
z = (15.671 - 15.4) / 0.20 = 0.271 / 0.20 = 1.355
Beta = P(Z >= 1.355) = 1 - Phi(1.355) ≈ 1 - 0.9123 = 0.0877
The probability of a Type II error is about 8.77%. Power = 1 - 0.0877 = 0.9123, so the test has about 91% power to detect a true mean of 15.4 oz.
Students often confuse alpha (the probability of a Type I error, set in advance) with the p-value (the probability of observing a result as extreme as yours, assuming H0 is true). Alpha is a threshold you choose; the p-value is computed from your data.
A common mistake is computing beta under H0. Beta is defined under a specific alternative value. You must shift the distribution to the alternative mean (or proportion) before computing the probability of falling in the non-rejection region.
Students sometimes think "fail to reject" means "H0 is true." It does not. It means the data do not provide sufficient evidence against H0 at the chosen significance level. The Bayesian posterior calculation in Problem 2(c) shows how to quantify the probability that H0 is actually true.
When computing the variance of differences with measurement error (Problem 3c), students sometimes forget to subtract the covariance term. Var(M_2 - M_1) = Var(M_2) + Var(M_1) - 2Cov(M_1, M_2), not just the sum of the variances.
⚠️ Type II error calculations appear on nearly every exam. The pattern: find the critical cutoff under H0, then compute the probability of falling in the non-rejection region under the specified alternative.
⚠️ The Bayesian posterior twist (Problem 2c) is a favourite exam question. Know how to set up Bayes' theorem with P(FTR | H0), P(FTR | H1), and the priors.
⚠️ Measurement error and efficiency comparisons (Problem 3c-e) test whether you understand how added noise affects variance, standard error, and power. The key insight: larger n can compensate for noisier measurements.
⚠️ Know the z-cutoff for alpha = 0.0228: it is z = 2.00. This is a recurring value in this practice set.
True or False: Power increases as the sample size increases (all else equal).
Answer: True. Larger n reduces the standard error, which shrinks the non-rejection region under the alternative.
Fill in the blank: Beta is computed under the ______ hypothesis, not under H0.
Answer: alternative.
True or False: If alpha = 0.0228 for a one-sided upper-tail test, the z-cutoff is 2.00.
Answer: True.
Fill in the blank: Measurement error ______ the variance of observed differences, which ______ the standard error.
Answer: increases, increases.
True or False: Inverse-variance weighting gives more weight to the estimator with larger variance.
Answer: False. It gives more weight to the estimator with smaller variance (higher precision).
Q: A test has alpha = 0.05 and the true parameter value gives beta = 0.20. What is the power?
A: Power = 1 - beta = 1 - 0.20 = 0.80.
Q: You are testing H0: mu = 50 vs H1: mu > 50 with sigma = 10, n = 100, alpha = 0.05. What is the critical cutoff in terms of x-bar?
A: SE = 10/sqrt(100) = 1. z_(0.05) = 1.645. Cutoff: x-bar* = 50 + 1.645(1) = 51.645. Reject if x-bar >= 51.645.
Q: In the same test, if mu = 52, what is beta?
A: Under mu = 52, z = (51.645 - 52)/1 = -0.355. Beta = P(Z < -0.355) ≈ 0.3613.
Q: Explain why the inverse-variance weighted estimator is more efficient than a simple average when combining two unbiased estimators with different variances.
A: The simple average weights both equally, but the estimator with larger variance contributes more noise. Inverse-variance weighting gives more weight to the more precise estimator, minimising the variance of the combined estimate. This is the minimum-variance unbiased linear combination.
Type II error and power connect directly to sample-size planning: in practice, you choose n so that the power is at least 0.80 (or 0.90) for a clinically or practically meaningful effect size. The Bayesian posterior calculation in Problem 2(c) is a bridge to Bayesian inference more broadly, where prior beliefs are updated with data. Measurement error and efficiency (Problem 3) connect to regression, where measurement error in the independent variable causes attenuation bias.
Related Terms / Search Tags
hypothesis testing, Type I error, Type II error, beta, power, significance level, alpha, critical cutoff, rejection region, one-sided test, z-test known variance, paired t-test, matched pairs, Bayesian posterior, prior probability, Bayes theorem, measurement error, self-reported scores, efficiency, inverse-variance weighting, minimum-variance estimator, confidence interval, NYU analytical statistics, ASTATS exam 2