Difficulty: Intermediate | Prerequisites: Basic probability, calculus (derivatives, integrals), familiarity with normal distribution.
Big Picture
This material covers the building blocks that every hypothesis test and parameter estimate rests on. Before you can run a t-test or a chi-squared test, you need to know what "significance" means, what a p-value tells you, what makes one estimator better than another, and how to construct an estimator from scratch. If the rest of the course is a toolbox, this is the manual that explains how the tools were built. You should already be comfortable with expected value, variance, and the standard normal distribution.
TL;DR
Statistical inference has two pillars: estimation (how do we guess a parameter from data?) and testing (how do we decide whether a hypothesis about that parameter is true?). Estimators are judged by bias, variance, consistency, and efficiency. Hypothesis tests follow a four-step procedure: choose a test statistic, find its distribution under the null, pick a rejection region, and compare.
Significance level (α)
The probability of a Type I error, i.e. rejecting the null hypothesis when it is in fact true. Common values: 0.05, 0.01.
In simple terms, this is the false-alarm rate you are willing to tolerate.
Power (1 − β)
The probability of correctly rejecting the null hypothesis when the alternative is true.
Think of it as: how good your test is at catching a real effect.
Type I error (α)
Rejecting H₀ when H₀ is true (a false positive).
Type II error (β)
Accepting H₀ when H_A is true (a false negative, or a miss).
Bias
Bias of an estimator θ̂: E[θ̂] − θ₀. An estimator is unbiased if E[θ̂] = θ₀.
In simple terms, bias is how far off your estimator is on average from the true value.
Standard error
The standard deviation of an estimator's sampling distribution: σ(θ̂) = √Var(θ̂).
Consistency
An estimator is consistent if it converges in probability to the true parameter as n → ∞.
Think of it as: with enough data, the estimator locks on to the right answer.
Efficiency
Estimator A is more efficient than estimator B if Var(A) < Var(B), both unbiased. The one with smaller variance gives you tighter estimates.
P-value
The smallest significance level α at which you would still reject H₀, given your observed test statistic.
In simple terms, the p-value answers: "How surprised should I be by this data if the null were true?"
Jensen's inequality
For a convex function f: E[f(X)] ≥ f(E[X]).
Useful for proving that certain transformations of estimators introduce bias.
Every hypothesis test in this course follows the same four-step recipe.
Step 1: Identify the test statistic
Choose a function of your sample that will measure how far the data are from what the null hypothesis predicts. Examples: Z, T, χ², F.
Step 2: Find the distribution under the null
Work out what distribution the test statistic follows if H₀ is true. This is what lets you calculate critical values and p-values.
Step 3: Determine the rejection region (tail direction)
Two-sided (H_A: μ ≠ μ₀): reject in both tails
Right-tailed (H_A: μ > μ₀): reject in the right tail
Left-tailed (H_A: μ < μ₀): reject in the left tail
Step 4: Find critical values and compare
Look up the quantile from the relevant distribution table (e.g. Φ⁻¹(0.95) = 1.645 for a one-sided Z-test at α = 0.05). If your test statistic falls in the rejection region, reject H₀.
Useful standard-normal quantiles
Φ⁻¹(0.95) = 1.645
Φ⁻¹(0.975) = 1.96
Choosing between tests
When two tests share the same α, prefer the one with higher power (1 − β). A test of the form "reject if g(T(x)) < g(k)" for some monotonic function g is identical to "reject if T(x) < k," so the monotonic transformation does not change the test.
The idea is straightforward: set population moments equal to sample moments and solve for the unknown parameters.
The mth population moment: E[Xᵐ] = μₘ(θ)
The mth sample moment: Âₘ = (1/n) Σ Xᵢᵐ
Equate Âₘ = μₘ(θ) for m = 1, ..., k and solve for θ
Example (Bernoulli)
For X ~ Bernoulli(p): E[X] = p. The sample moment is X̄ₙ. So p̂_MoM = X̄ₙ = (1/n) Σ Xᵢ.
Population expectation for indicators
Define Yᵢ = 1 if Xᵢ ≥ k, and 0 otherwise. Then E[Yᵢ] = P(Xᵢ ≥ k). For X ~ Uniform(0, θ): P(X ≥ k) = max{1 − k/θ, 0}. If k < θ, the MoM estimator becomes θ̂ₙ = nk / (n − Σ Yᵢ).
The likelihood function is the joint density of the sample, viewed as a function of the parameter:
L(θ) = f(X₁, ..., Xₙ | θ) = Π f(Xᵢ | θ)
The MLE is the value of θ that maximises L(θ), or equivalently the log-likelihood ℓ(θ) = log L(θ).
Recipe for finding the MLE
Write down the likelihood L(θ)
Take the log to get ℓ(θ)
Differentiate with respect to θ, set equal to zero
Solve for θ̂_MLE
Example (Bernoulli)
L(p) = Π pˣⁱ (1−p)¹⁻ˣⁱ = p^(Σxᵢ) (1−p)^(n−Σxᵢ)
ℓ(p) = (Σxᵢ) log(p) + (n − Σxᵢ) log(1−p)
Set dℓ/dp = 0: (Σxᵢ)/p − (n − Σxᵢ)/(1−p) = 0
Solving gives p̂_MLE = (1/n) Σxᵢ = X̄ₙ. In this case MoM and MLE agree.
Unbiased and normally distributed, Var(θ) known:
Confidence interval: θ̂ ± Φ⁻¹(1 − α/2) · √Var(θ̂)
Unbiased and normally distributed, Var(θ) unknown but S available:
Confidence interval: θ̂ ± t_{n−1}(1 − α/2) · √Var(θ̂), using S in place of σ.
Sample mean and variance
X̄ₙ = (1/n) Σ Xᵢ
S² = (1/(n−1)) Σ (Xᵢ − X̄ₙ)²
When Xᵢ ~ N(μ, σ²): X̄ₙ ~ N(μ, σ²/n)
Chi-squared from sample variance
(n−1) S² / σ² ~ χ²_{n−1} (when sampling from a normal population with unknown mean)
Key normal-distribution facts
If X₁, ..., Xₙ are independent with Xᵢ ~ N(0, 1): X₁² + ... + Xₙ² ~ χ²ₙ
If X ~ N(0, 1) independent of Y ~ χ²ₙ: X / √(Y/n) ~ tₙ (Student's t)
If X ~ χ²ₘ independent of Y ~ χ²ₙ: (X/m) / (Y/n) ~ F_{m,n}
If X ~ N(μ, σ²) and X̄ₙ, S² are the sample mean and variance from n observations: X̄ₙ ~ N(μ, σ²/n), independent of S²
These relationships are what connect raw data to the test statistics used in Z-tests, t-tests, χ²-tests, and F-tests.
Students often confuse the p-value with the probability that H₀ is true. It is not. The p-value is the probability of seeing data at least as extreme as yours, assuming H₀ is true.
"Failing to reject H₀" does not mean H₀ is true. It means the evidence is not strong enough to reject it at the chosen α.
Bias and variance are separate properties. An estimator can be unbiased but high-variance (noisy), or biased but low-variance (consistently wrong in the same direction).
Consistency does not imply unbiasedness for finite n. An estimator can be biased at every sample size and still be consistent if the bias vanishes as n grows.
⚠️ You will be asked to state the four-step testing procedure from memory. Know it cold.
⚠️ Expect a question requiring you to derive the MLE for a given distribution (Bernoulli, Uniform, or Exponential are common).
⚠️ Be ready to explain the difference between bias, consistency, and efficiency in your own words, with an example for each.
⚠️ Know Jensen's inequality and when it applies (proving bias of transformed estimators).
True or False: A consistent estimator is always unbiased.
False. Consistency is about convergence as n → ∞; an estimator can be biased at every finite n and still be consistent.
Fill in the blank: The p-value is the smallest ______ at which you would reject H₀.
significance level (α)
True or False: If two tests have the same α, you should prefer the one with higher power.
True.
Fill in the blank: The MLE is the parameter value that maximises the ______ function.
likelihood (or log-likelihood)
True or False: E[f(X)] = f(E[X]) for any function f.
False. This holds only when f is linear. For convex f, Jensen's inequality gives E[f(X)] ≥ f(E[X]).
Q: Describe the four-step hypothesis testing procedure.
A: (1) Identify the test statistic. (2) Find its distribution under the null. (3) Determine the rejection region (left, right, or both tails). (4) Find critical values and compare the test statistic to them.
Q: What is the difference between an efficient estimator and a consistent estimator?
A: Efficiency compares the variances of two unbiased estimators at a given sample size: the one with smaller variance is more efficient. Consistency is about what happens as n → ∞: a consistent estimator converges in probability to the true value regardless of its bias at any finite n.
Q: Derive the MLE for p in a Bernoulli(p) sample of size n.
A: Write L(p) = p^(Σxᵢ)(1−p)^(n−Σxᵢ). Take the log: ℓ(p) = (Σxᵢ) log p + (n−Σxᵢ) log(1−p). Set dℓ/dp = 0 and solve to get p̂ = X̄ₙ.
Q: A researcher reports a p-value of 0.03 and says "There is a 3% chance the null hypothesis is true." Is this correct?
A: No. The p-value is the probability of data this extreme or more, assuming H₀ is true. It does not give the probability that H₀ is true.
Q: If (n−1)S²/σ² ~ χ²_{n−1}, what distribution does S² follow when scaled by σ²/(n−1)?
A: S² follows σ²/(n−1) times a chi-squared with n−1 degrees of freedom. Equivalently, E[S²] = σ², confirming S² is an unbiased estimator of σ².
Type I error, Type II error, significance level, alpha, power, beta, bias, unbiased estimator, standard error, consistency, efficiency, Jensen's inequality, p-value, hypothesis testing procedure, Method of Moments, MoM, Maximum Likelihood Estimation, MLE, log-likelihood, sample mean, sample variance, chi-squared distribution, t-distribution, F-distribution, normal distribution, quantile, critical value, rejection region, Analytical Statistics NYU