Source: Principles of Statistics I, Texas A&M / Davison & Hinkley; Hjorth

Tags: bootstrap CI, nonparametric bootstrap, parametric bootstrap, studentized bootstrap, Box-Cox transformation, simultaneous CI, Bonferroni correction, Wald CI, MLE standard error, confidence interval transformation

TL;DR

When exact or asymptotic CIs are not available, you can use bootstrap resampling to approximate the sampling distribution of the pivot. Transformations to normality can produce valid intervals for quantiles and prediction, but generally fail for population means. Simultaneous CIs widen individual intervals to maintain joint coverage, using the Bonferroni inequality when intervals are dependent. Wald CIs from MLE standard errors are a general-purpose asymptotic method for any parametric model.

Key Terms

Basic bootstrap CI

Resample B times with replacement from the original data. Use the empirical percentiles of θ̂* − θ̂_D to estimate the percentiles of θ̂ − θ, then invert.

Studentized bootstrap CI

Divides θ̂* − θ̂_D by an estimated standard error before taking percentiles, analogous to using a t-statistic instead of a z-statistic. More accurate when the variance of the pivot depends on unknown parameters.

Parametric bootstrap

Instead of resampling the original data, simulate new samples from the fitted parametric model (with MLE parameters plugged in). Used when you know the distributional family but cannot derive the sampling distribution analytically.

Box-Cox transformation

A transformation g(Y) = X that makes data approximately normal. Valid for constructing PIs and TIs (just invert the endpoints), but generally invalid for CIs on the population mean μ_Y.

Simultaneous CI

A set of CIs where the joint probability that all intervals simultaneously contain their respective parameters is at least 1 − α.

Bonferroni inequality

P[all k events] ≥ 1 − Σα_i. To get simultaneous 100γ% coverage from k intervals, set each individual α_i = (1 − γ)/k.

Wald CI from MLE

θ̂ ± Z_α/2 · SE(θ̂), where SE is the asymptotic standard error obtained from the Fisher information. A general-purpose large-sample method.

Core Content

Bootstrap Confidence Intervals

When to Use Bootstrap

• The cdf F is unknown and the sample size is too small for asymptotic methods

• The sampling distribution of the estimator is intractable even if F is known

• The parameter θ can be written as a function of the cdf: θ = g(F)

Basic Bootstrap Procedure

1. From data X₁, …, X_n, compute θ̂_D = g(F̂) where F̂ is the empirical cdf.

1. Draw B bootstrap samples of size n with replacement from the original data.

1. For each bootstrap sample i, compute θ̂*_i.

1. Form R*_i = θ̂_i − θ̂_D and sort: R₍₁₎ ≤ … ≤ R*_(B).

1. The approximate 100(1 − α)% CI for θ is:

(θ̂_D − R*_{((B+1)(1−α/2))}, θ̂_D − R*_{((B+1)(α/2))})

Or equivalently: (2θ̂_D − θ̂_{((B+1)(1−α/2))}, 2θ̂_D − θ̂_{((B+1)(α/2))})

Choose B so that (B + 1)α/2 is an integer. Typical choice: B = 9999 or B = 19999.

Light bulb example: 20 bulb lifetimes. θ = P[T ≥ 1200] = 1 − F(1200). θ̂_D = 13/20 = 0.65. Basic bootstrap 95% CI: (0.45, 0.85).

Median example: Same data, θ = median. θ̂_D = 1354. Bootstrap 95% CI: (912.5, 1718.5). Comparable to the nonparametric CI (Y₍₆₎, Y₍₁₅₎) = (965, 1825).

Studentized Bootstrap

When the distribution of θ̂ − θ depends on unknown parameters (e.g., its variance changes with θ), the basic bootstrap can lose accuracy.

Fix: Studentize the pivot. Let V̂ be an estimate of Var(θ̂). Then:

Z = (θ̂ − θ) / √V̂

Bootstrap this Z instead:

1. For each bootstrap sample, compute θ̂_i, V̂_i, and Z*_i = (θ̂_i − θ̂_D) / √V̂_i.

1. Sort Z* values.

1. 100(1 − α)% CI:

(θ̂_D − √V̂_D · Z*_{((B+1)(1−α/2))}, θ̂_D − √V̂_D · Z*_{((B+1)(α/2))})

Air conditioning example: n = 12 exponential failure times. β̂ = T̄ = 108.08. V̂ = T̄²/n = 973.5.

• Studentized bootstrap 95% CI: (61.5, 247.7)

• Exact chi-square CI: (65.9, 209.2)

• Surprisingly close for n = 12.

Parametric Bootstrap

When a parametric model is known but the sampling distribution of some statistic is intractable.

Procedure:

1. Fit the model to data; obtain MLE parameter estimates.

1. Simulate B samples of size n from the fitted parametric distribution (not by resampling the data).

1. For each simulated sample, re-estimate the parameters and compute the statistic of interest.

1. Use the B bootstrap statistics to construct the CI as before.

Logistic CV example: Data from a logistic(β, γ) distribution, n = 25. Parameter of interest: CV = πγ/(√3 · β). The MLE sampling distribution is intractable at this sample size.

For a scale-type parameter like CV, use the ratio pivot R = CV̂*/CV̂_D:

CI = (CV̂_D / R_{((B+1)(1−α/2))}, CV̂_D / R_{((B+1)(α/2))})

Result: CV̂ = 0.274, 95% CI = (0.202, 0.408).

Bootstrap Pitfalls

• Bootstrap is not assumption-free. The edf F̂ must adequately represent the true F.

• Error has two sources: statistical error (F̂ vs F) and simulation error (finite B).

• The pivot's distribution must not depend on unknown parameters, or else use the studentized version.

• Bootstrap can fail entirely in some settings. Example: for Uniform(0, θ), the MLE θ̂ = Y_(n) has P[pivot* = 0] → 1 − e⁻¹ ≈ 0.632, so the bootstrap pivot cannot converge to the correct exponential distribution.

Transformations to Normality

If X = g(Y) is approximately normal, you can build intervals for X and then invert.

Valid: Prediction Intervals and Tolerance Intervals

For an increasing g with inverse g⁻¹:

PI for Y_n+1: (g⁻¹(X̄ − t_α/2 S_X√((n+1)/n)), g⁻¹(X̄ + t_α/2 S_X√((n+1)/n)))

TI for Y: (g⁻¹(L*_P,γ), g⁻¹(U*_P,γ))

For a decreasing g, reverse the order of the endpoints.

This works because F_Y(y) = F_X(g(y)) for increasing g, so quantile relationships are preserved.

Invalid (in general): CIs for the Mean μ_Y

The problem: μ_X = E[g(Y)] ≠ g(μ_Y) in general (Jensen's inequality).

By Taylor expansion: μ_X ≈ g(μ_Y) + ½g″(μ_Y) · Var(Y). The approximation is only good when Var(Y) is small relative to g″.

Lognormal example: Y = e^W, W ~ N(μ, σ²). Let X = log(Y).

• μ_X = μ

• μ_Y = e^{μ + σ²/2}

• log(μ_Y) = μ + σ²/2 ≠ μ_X when σ² is non-negligible

So inverting a CI for μ_X gives a CI for μ_Ye^−σ²/2, not for μ_Y.

Valid: CIs for Quantiles Q_Y(u)

For increasing g: Q_X(u) = g(Q_Y(u)), so a CI (L_X, U_X) for Q_X(u) transforms to (g⁻¹(L_X), g⁻¹(U_X)) for Q_Y(u).

Simultaneous Confidence Intervals

When making inferences about k parameters jointly, individual CIs overstate confidence.

Independent CIs

If k individual 100(1 − α)% CIs are independent, the joint coverage is:

γ = (1 − α)^k

To achieve joint γ = 0.95 with k independent CIs, set 1 − α = 0.95^1/k.

Bonferroni Method (Dependent or Independent)

Works regardless of dependence structure. For k CIs with desired simultaneous level γ:

Set each individual α_i = (1 − γ) / k

For k = 4 and γ = 0.95: each individual CI uses α/2 = 0.05/(2 · 4) = 0.00625 instead of 0.025.

Lipid example: Four feeding regimens, individual 95% CIs clearly separate SMO from the rest. But simultaneous 95% CIs (using t_.00635 instead of t_.025) are wider and overlap. The conclusion of a significant difference no longer holds at the 95% simultaneous level.

Bonferroni can be conservative (actual coverage > γ) because it uses an inequality rather than the exact joint probability.

Wald CI Based on MLEs

A general-purpose asymptotic CI for any parameter θ in a parametric model.

If θ̂ is the MLE with asymptotic standard error SE(θ̂):

100(1 − α)% Wald CI: θ̂ ± Z_α/2 · SE(θ̂)

R/SAS output from MLE fitting routines provides SE(θ̂) directly from the Fisher information matrix.

Weibull example: n = 23 bearing failure data. Weibull fit gives:

• γ̂ = 2.101, SE = 0.329 → 95% CI for shape: (1.457, 2.745)

• α̂ = 81.83, SE = 8.597 → 95% CI for scale: (73.24, 90.43)

These are wide, reflecting n = 23. The Wald CI becomes more reliable as n grows.

The same approach works for Kaplan-Meier survival estimates: the output gives estimated mean, median, and their standard errors, from which you build CIs directly.

Why It Matters / Exam Flags

⚠️ The basic bootstrap CI formula subtracts the upper percentile from θ̂_D for the lower CI bound, and vice versa. The "flip" is easy to get wrong on exams.

⚠️ Bootstrap is not universally valid. It requires regularity conditions (the Uniform(0, θ) counterexample shows this).

⚠️ Transformations preserve quantile CIs but generally destroy mean CIs. Know which is which.

⚠️ For the lognormal, inverting a CI for μ_X = E[log Y] gives a CI for the geometric mean (μ_Ye^−σ²/2), not the arithmetic mean μ_Y.

⚠️ Simultaneous CIs are wider than individual CIs. Using individual CIs to compare multiple parameters inflates the Type I error rate.

⚠️ The Bonferroni method works for any dependence structure but can be very conservative.

⚠️ Wald CIs from MLEs use Z_α/2 (normal percentiles), not t-percentiles, because they are asymptotic.

Practice Q&A

Q: In a basic bootstrap with B = 9999 and α = 0.05, which ordered R values give the CI percentiles?*

A: The (B + 1)(α/2) = 250th and (B + 1)(1 − α/2) = 9750th ordered values of R* = θ̂* − θ̂_D.

Q: Why does the studentized bootstrap improve on the basic bootstrap for exponential data?

A: The pivot θ̂ − θ for exponential data has a variance that depends on the unknown parameter β (since Var(T̄) = β²/n). Dividing by an estimated standard error removes this dependence, making the bootstrap distribution a better approximation.

Q: A researcher builds individual 95% CIs for 5 independent parameters and finds that none overlap. Can she claim all 5 parameters differ at the 95% simultaneous level?

A: No. The simultaneous coverage is only (0.95)⁵ = 0.774. To claim simultaneous 95% confidence, she needs to use 1 − α = 0.95^1/5 = 0.9898, i.e., individual 98.98% CIs.

Q: For a Weibull MLE with γ̂ = 2.5 and SE(γ̂) = 0.4, give a 99% Wald CI for the shape parameter.

A: 2.5 ± 2.576(0.4) = 2.5 ± 1.03 = (1.47, 3.53).

Q: If Y has a lognormal distribution and X = log(Y) ~ N(μ, σ²), what parameter does exp(X̄ ± t_α/2S/√n) provide a CI for?

A: It provides a CI for e^μ = μ_Y · e^−σ²/2, which is the geometric mean, not the arithmetic mean μ_Y = e^μ+σ²/2.

Related Terms / Search Tags

bootstrap confidence interval, basic bootstrap, studentized bootstrap, parametric bootstrap, nonparametric bootstrap, bootstrap pivot, resampling, Box-Cox transformation CI, transformation to normality, Jensen's inequality CI, quantile transformation, simultaneous confidence intervals, Bonferroni correction, multiple comparisons, family-wise error rate, Wald confidence interval, MLE standard error, Fisher information, asymptotic CI, Weibull MLE, Kaplan-Meier CI, logistic distribution bootstrap, coefficient of variation CI