Source: Principles of Statistics I, Texas A&M / Tamhane & Dunlop Ch. 5, 14.6, 15.1
Tags: bootstrap, nonparametric bootstrap, parametric bootstrap, resampling, sampling with replacement, empirical distribution function, edf, maximum likelihood estimator, MLE, Fisher information, invariance property, asymptotic normality of MLE, standard error of MLE, Weibull, exponential
When the population distribution is unknown or the sample is too small for asymptotic theory, the bootstrap approximates the sampling distribution by resampling with replacement from the observed data. The parametric bootstrap does the same but generates new samples from a fitted model instead of the raw data. Maximum likelihood estimators (MLEs) have a known asymptotic normal distribution whose variance involves the Fisher information number, and the parametric bootstrap is especially useful when that asymptotic formula is hard to compute.
Bootstrap (nonparametric bootstrap)
A resampling procedure that approximates the sampling distribution of a statistic by repeatedly drawing samples of size n, with replacement, from the observed data and computing the statistic from each resample.
Bootstrap sample
A single sample of size n drawn with replacement from the original data. Some observations will appear more than once; others will be absent entirely.
Empirical distribution function (edf)
F-hat(x) = (number of Xᵢ ≤ x) / n. The bootstrap treats this as the "population" and samples from it.
Parametric bootstrap
A variant where, instead of resampling from the raw data, you fit a parametric model (e.g. Weibull), estimate its parameters via MLE, and then generate new samples from the fitted distribution.
Maximum likelihood estimator (MLE)
The parameter value θ-hat that maximises the likelihood function L(θ) = Π f(yᵢ; θ) given the observed data.
Likelihood function L(θ)
The joint density of the observed data, viewed as a function of the parameters rather than the data.
Log-likelihood l(θ)
The natural logarithm of L(θ). Maximising l(θ) is equivalent to maximising L(θ) and is usually easier to work with.
Fisher information number I_n(θ)
I_n(θ) = E[−∂²l(θ)/∂θ²]. Measures how much information the data carry about the parameter. A larger information number means the MLE is more precise.
Invariance property of MLEs
If θ-hat is the MLE of θ, then for any function h(θ), the MLE of h(θ) is h(θ-hat). For example, if you have the MLE of (γ, α) for a Weibull, the MLE of the survival function S(t) = exp(−(t/α)^γ) is simply S-hat(t) = exp(−(t/α-hat)^γ-hat).
The bootstrap addresses the situation where you cannot derive the sampling distribution mathematically, the sample is too small for asymptotic results, and the population distribution is unknown (ruling out simulation).
Procedure
Start with the observed data X₁, …, Xₙ and compute the statistic θ-hat.
Draw a sample of size n with replacement from the observed data. Call this the bootstrap sample X₁*, …, Xₙ*.
Compute the statistic from the bootstrap sample: θ-hat*.
Repeat steps 2 and 3 a total of M times, yielding θ-hat₁*, θ-hat₂*, …, θ-hat_M*.
Use these M values to estimate the mean, standard deviation, percentiles, and shape of the sampling distribution of θ-hat.
Why it works
The observed data provide an estimate of the population distribution (the edf). Resampling from the edf mimics drawing new samples from the true population. The quality of the approximation depends on how well the edf approximates the true CDF, which improves with n.
Two levels of approximation
Level 1: How well the edf F-hat approximates the true CDF F. Controlled by n and the shape of F.
Level 2: How well the M bootstrap replicates approximate the sampling distribution based on F-hat. Controlled by M (more replicates is better).
Eleven engine parts had the following life lengths:
1600, 4100, 5700, 7600, 12400, 12900, 19300, 21500, 28000, 36300, 91400
The goal was to estimate the sampling distribution of the sample median.
Point estimate: θ-hat = X₍₆₎ = 12,900
Bootstrap with M = 200 resamples
Mean of bootstrap medians: 14,877.5
Standard deviation: 5,552.6
Quartiles: Q1 = 12,400, Median = 12,900, Q3 = 19,300
Bootstrap with M = 20,000 resamples
Mean: 14,924.1
Standard deviation: 5,933.7
The means and standard deviations were similar to the M = 200 case, but the extreme quantiles differed considerably (min went from 4,100 to 1,600; max went from 36,300 to 91,400).
With 20,000 resamples, rare events became visible. For instance, bootstrap samples where the median equalled 1,600 (the minimum data value) require at least 6 of 11 resampled values to be 1,600. The expected count of such samples is about 3.2 out of 20,000.
Key observation: With n = 11, the sampling distribution of the median is visibly discrete and far from the asymptotic normal shape. Small n means the bootstrap is useful but also limited in resolution.
M = 200 is enough for rough estimates of mean and standard deviation.
M = 10,000 to 20,000 is better for estimating tail percentiles.
The bootstrap does not create new information; it reorganises what is already in the sample. With very small n, the bootstrap distribution can look coarse.
Plots to examine: boxplot, kernel density estimate, quantile function, and normal probability plot of the M bootstrap values.
Asymptotic normality of MLEs
Under regularity conditions, for large n:
h(θ-hat) is approximately N(h(θ), [h'(θ)]² / I_n(θ))
where I_n(θ) = E[−∂²l(θ)/∂θ²] is the Fisher information.
In the simplest case where h(θ) = θ:
θ-hat is approximately N(θ, 1/I_n(θ))
The asymptotic standard deviation is σ_A = 1/√I_n(θ), estimated by replacing θ with θ-hat.
Invariance property
If θ-hat is the MLE of θ, then h(θ-hat) is the MLE of h(θ) for any continuous function h. This is extremely convenient: you compute the MLE of the basic parameters once and then plug them into whatever derived quantity you need.
If T₁, …, Tₙ are iid Exp(β), the MLE is β-hat = T-bar.
The log-likelihood is:
l(β) = −n log(β) − (1/β) Σ tᵢ
Taking second derivatives and computing the expectation:
I_n(β) = n/β²
So the asymptotic standard error is:
SE(β-hat) = β-hat/√n
For Weibull(γ, α) data, the MLEs are found numerically by solving:
α-hat = ((1/n) Σ tᵢ^γ-hat)^(1/γ-hat)
and a second equation involving Σ tᵢ^γ-hat × log(tᵢ).
The asymptotic standard errors are obtained by numerically computing the second partial derivatives of the log-likelihood. R's fitdistr() function does this automatically.
R output example (n = 25 device lifetimes):
γ-hat = 0.984, SE(γ-hat) = 0.151
α-hat = 11.485, SE(α-hat) = 2.466
When the form of the MLE is complex or n is too small for the asymptotic approximation, the parametric bootstrap provides an alternative.
Procedure
Fit a parametric model to the data and obtain the MLEs (e.g. γ-hat, α-hat for Weibull).
Generate M new samples of size n from the fitted distribution (e.g. Weibull(γ-hat, α-hat)).
For each generated sample, re-estimate the parameters and compute the quantity of interest.
Use the M values to estimate the sampling distribution.
Device lifetime example
The goal was to estimate S(20) = P[T > 20], the survival probability at t = 20.
MLE point estimate: S-hat(20) = exp(−(20/11.485)^0.984) = 0.178
Distribution-free estimate (for comparison): 5/25 = 0.20
Parametric bootstrap (M = 10,000 samples of size 25 from Weibull(0.984, 11.485)):
Mean of S-hat(20): 0.1727
SD of S-hat(20): 0.0626
Quartiles: Q1 = 0.129, Median = 0.171, Q3 = 0.214
The bootstrap standard errors for γ and α (0.175 and 2.462) were very close to the asymptotic standard errors (0.151 and 2.466), confirming that the asymptotic theory is reasonable even at n = 25 for this particular dataset.
The bootstrap estimate of SE(S-hat(20)) = 0.0626 would be considerably harder to obtain from asymptotic MLE theory, because S(20) is a nonlinear function of two parameters.
Asymptotic MLE theory: When n is large enough, the population model is known, and the derived quantity is a simple function of the parameters.
Parametric bootstrap: When the population model is known (or strongly suspected) but n is modest or the quantity of interest is a complex function of the parameters.
Nonparametric bootstrap: When the population model is unknown.
Fisher information: I_n(θ) = E[−∂²l(θ)/∂θ²]
Asymptotic variance of MLE: Var(θ-hat) ≈ 1/I_n(θ)
Asymptotic variance of h(θ-hat): Var(h(θ-hat)) ≈ [h'(θ)]²/I_n(θ)
Exponential MLE: β-hat = T-bar, SE = β-hat/√n
Weibull survival function MLE: S-hat(t) = exp(−(t/α-hat)^γ-hat)
Bootstrap mean: μ-hat = (1/M) Σ θ-hatᵢ*
Bootstrap SE: SE = √((1/M) Σ (θ-hatᵢ* − μ-hat)²)
⚠️ The bootstrap resamples with replacement from the observed data. "With replacement" is essential: without it, you would just get the same dataset every time.
⚠️ The nonparametric bootstrap treats the edf as the population. The parametric bootstrap treats the fitted model as the population. Know the difference.
⚠️ The invariance property of MLEs is a favourite exam topic. If θ-hat is the MLE of θ, then g(θ-hat) is the MLE of g(θ) for any function g.
⚠️ The Fisher information I_n(θ) involves the expected value of the negative second derivative of the log-likelihood. Be able to derive it for simple distributions (exponential, Bernoulli).
⚠️ The asymptotic normality of MLEs requires regularity conditions. It does not hold for all distributions or all sample sizes.
⚠️ The parametric bootstrap is particularly useful for complex derived quantities (like survival probabilities) where the delta method would be messy.
Q: Describe the nonparametric bootstrap procedure in five steps.
A: (1) Observe data and compute θ-hat. (2) Draw a sample of size n with replacement from the data. (3) Compute θ-hat* from that resample. (4) Repeat M times. (5) Use the M values of θ-hat* to estimate the sampling distribution.
Q: What are the two levels of approximation in the bootstrap?
A: First, the edf approximating the true CDF (controlled by sample size n). Second, the M bootstrap replicates approximating the bootstrap sampling distribution (controlled by M).
Q: State the invariance property of MLEs and give an example.
A: If θ-hat is the MLE of θ, then h(θ-hat) is the MLE of h(θ). For example, if α-hat and γ-hat are MLEs of Weibull parameters, the MLE of S(t) = exp(−(t/α)^γ) is S-hat(t) = exp(−(t/α-hat)^γ-hat).
Q: For an exponential(β) model with n observations, derive the Fisher information and the asymptotic standard error of the MLE.
A: The log-likelihood is l(β) = −n log(β) − (1/β)Σtᵢ. The second derivative is ∂²l/∂β² = n/β² − 2/β³ Σtᵢ. Taking the negative expectation: I_n(β) = n/β². So SE(β-hat) = 1/√(n/β²) = β/√n, estimated by β-hat/√n.
Q: When would you prefer the parametric bootstrap over the nonparametric bootstrap?
A: When you have a reasonable parametric model for the data (confirmed by, say, a reference plot and goodness-of-fit test) and you want to estimate the sampling distribution of a complex function of the parameters.
Q: In the engine parts example, why does the bootstrap distribution of the median look discrete rather than normal?
A: With n = 11, the sample median must equal one of the original data values (or an average of two adjacent values). The discrete nature of the data shows through, and n is too small for the CLT approximation to smooth the distribution into a bell curve.
bootstrap, nonparametric bootstrap, parametric bootstrap, resampling, sampling with replacement, empirical distribution function, edf, kernel density estimator, maximum likelihood estimator, MLE, likelihood function, log-likelihood, Fisher information, information number, invariance property, asymptotic normality, delta method, standard error, Weibull MLE, exponential MLE, survival function, R fitdistr, engine parts example, device lifetime, Cauchy distribution, regularity conditions, Tamhane Dunlop, Hinkley Davison, STAT Texas A&M, Handout 10