Source: Principles of Statistics I (Texas A&M), Tamhane/Dunlop Ch. 2 & 4
Tags: distribution-free summaries, sample mean, sample median, sample variance, sample standard deviation, trimmed mean, five number summary, IQR, interquartile range, MAD, mean absolute deviation, skewness, kurtosis, robust estimation, m-estimator, L-estimator, R-estimator, order statistics, unbiased estimator, outliers
When we do not know (or do not assume) a particular distribution, we can still estimate population summaries by replacing the population CDF with the empirical distribution function. This gives us familiar statistics like the sample mean, median, standard deviation, and quantiles. Some of these estimators are sensitive to outliers; robust alternatives such as the trimmed mean, MAD, and m-estimators reduce that sensitivity.
Distribution-free summary (nonparametric summary)
An estimator of a population parameter that does not require specifying the form of the underlying distribution. Obtained by substituting the empirical distribution function (EDF) for the population CDF.
Empirical distribution function (EDF), F̂(y)
F̂(y) = (1/n) × Σ I(Y_i ≤ y), where I is the indicator function. It is a step function that jumps by 1/n at each observed data value.
Sample mean, Ȳ
Ȳ = (1/n) × Σ Y_i. Unbiased estimator of the population mean μ for any distribution with finite mean.
Sample variance, S²
S² = [1/(n−1)] × Σ(Y_i − Ȳ)². Unbiased estimator of σ² for any distribution with finite variance. Note the divisor n−1, not n.
Sample standard deviation, S
S = √S². Although S² is unbiased for σ², S itself is a biased estimator of σ (i.e. E[S] ≠ σ).
Sample median, Q̂(0.5)
The middle value of the ordered data. If n is odd: Y_((n+1)/2). If n is even: the average of Y_(n/2) and Y_(n/2+1).
Sample quartiles, Q̂₁ and Q̂₃
Estimates of Q(0.25) and Q(0.75). In R, use quantile(y, 0.25) and quantile(y, 0.75). Some textbooks define them by splitting the data into two halves and taking medians of each half; for small n, results may differ slightly.
Interquartile range (IQR)
IQR = Q̂(0.75) − Q̂(0.25). Measures the spread of the central 50% of the data.
Semi-interquartile range (SIQR)
SIQR = IQR / 2.
Five number summary
{Minimum, Q₁, Median, Q₃, Maximum}. Produced in R by quantile(y) or summary(y) (which also includes the mean).
Range, R̂
R̂ = Y_(n) − Y_(1). The difference between the largest and smallest observations. Highly sensitive to outliers.
Sample skewness, β̂₁
β̂₁ = μ̂₃ / (σ̂)³, where μ̂₃ = (1/n) × Σ(Y_i − Ȳ)³. Measures asymmetry of the distribution. Positive skewness means a longer right tail.
Sample kurtosis, β̂₂
β̂₂ = μ̂₄ / (σ̂)⁴, where μ̂₄ = (1/n) × Σ(Y_i − Ȳ)⁴. Measures tail heaviness. A normal distribution has β₂ = 3. Some software reports excess kurtosis (β̂₂ − 3) instead.
α-Trimmed mean, μ̂_(α)
The mean of the data after removing the [nα] smallest and [nα] largest values. Reduces the influence of extreme observations. In R: mean(y, trim = α).
MAD (Median Absolute Deviation)
MAD = Median{|Y_i − Median(Y)|} / 0.6745. A robust estimator of dispersion. The constant 0.6745 makes MAD consistent for σ when the data is Normal.
m-estimator of location
A weighted mean where extreme data values receive smaller weights. Computed iteratively. Reduces the influence of outliers without fully deleting them (unlike the trimmed mean).
Robust estimator
An estimator that is relatively unaffected by outliers or small deviations due to rounding/grouping.
Unbiased estimator
An estimator whose expected value equals the parameter it estimates. E[θ̂] = θ.
The general approach: take the population definition of a summary, replace the population CDF F with the EDF F̂, and compute.
Population mean: μ = ∫ y dF(y) becomes μ̂ = Ȳ
Population k-th central moment: μ_k = ∫ (y − μ)^k dF(y) becomes μ̂_k = (1/n) × Σ(Y_i − Ȳ)^k
This gives:
Sample standard deviation (EDF version): σ̂ = √[(1/n) × Σ(Y_i − Ȳ)²] (divisor n)
Sample skewness: β̂₁ = [(1/n) × Σ(Y_i − Ȳ)³] / (σ̂)³
Sample kurtosis: β̂₂ = [(1/n) × Σ(Y_i − Ȳ)⁴] / (σ̂)⁴
Software packages typically use modified versions that are unbiased under normality:
Unbiased estimator of μ: Ȳ = (1/n) × Σ Y_i. This is unbiased for any distribution with E[Y] < ∞.
Unbiased estimator of σ²: S² = [1/(n−1)] × Σ(Y_i − Ȳ)². Unbiased for any distribution with σ² < ∞. However, S = √S² is biased for σ.
Modified skewness: β̂*₁ = [n² / ((n−1)(n−2))] × [μ̂₃ / (S²)^(3/2)]. Unbiased for β₁ when the data comes from a Normal distribution. Does not necessarily hold for non-Normal data.
Modified excess kurtosis: β̂*₂ − 3 = [(n+1)n² / ((n−1)(n−2)(n−3))] × [μ̂₄ / (S²)²] − 3(n−1)² / ((n−2)(n−3)). Unbiased for β₂ − 3 when the data is Normal.
Different software packages may compute these differently. Always check which formula your software uses.
Median: Q̂(0.5). In R: median(y) or quantile(y, 0.5).
Quartiles: Q̂(0.25), Q̂(0.75). In R: quantile(y, c(0.25, 0.5, 0.75)).
IQR: Q̂(0.75) − Q̂(0.25). In R: IQR(y).
Range: Y_(n) − Y_(1).
Five number summary: {Min, Q₁, Median, Q₃, Max}. In R: quantile(y) or fivenum(y).
Remove the [nα] smallest and [nα] largest values and take the average of whatever remains.
K(α) = n − [nα] − [nα + 1] + 1 is the number of data values retained.
When nα is an integer, exactly 100α% is trimmed from each tail. When nα is not an integer, slightly less than 100α% is trimmed.
Example: n = 30, α = 0.05, so nα = 1.5. We trim Y_(1) and Y_(30) entirely, and partially weight Y_(2) and Y_(29) with weight p = 0.5.
In R: mean(y, trim = 0.1) computes the 10% trimmed mean.
A more precise version (from the robust estimation literature) uses partial trimming weights p = 1 + [nα] − nα on the first and last untrimmed values.
MAD is a robust alternative to the standard deviation.
Computation steps:
Find the sample median Q̂_Y(0.5)
Compute W_i = |Y_i − Q̂_Y(0.5)| for each observation
Find the median of the W values: Q̂_W(0.5)
MAD = Q̂_W(0.5) / 0.6745
The 0.6745 scaling makes MAD a consistent estimator of σ under normality. In R: mad(y).
The m-estimator is a refinement of the trimmed mean. Instead of deleting extreme observations outright, it downweights them. The further a data value falls from the centre, the smaller its weight.
Definition: m̂ = [1 / Σw_i] × Σ w_i × Y_i
The weights depend on three quantities:
v: a robust measure of dispersion (typically MAD)
t: a tuning constant, usually between 1.345 and 1.5 (smaller t = more aggressive downweighting)
The distance of each observation from the current estimate of centre
Weight function:
w_j = 1 if m̂ − tv ≤ Y_j ≤ m̂ + tv (no downweighting)
w_j = tv / |Y_j − m̂| if Y_j falls outside that interval (weight decreases as distance increases)
Computation is iterative:
Start with an initial value (e.g. the sample median)
Compute weights based on current estimate
Compute new weighted mean
Repeat until convergence (typically 9 or so iterations suffice)
In R: this can be implemented manually or using existing robust estimation packages.
The ozone data comparison (Stamford and Yonkers) demonstrates robustness vividly. When five extreme outliers (1000, 1200, 1500, 2000, 2500) are added to the data:
Estimator | Yonkers (original) | Yonkers (with outliers) |
|---|---|---|
Mean | 54.69 | 106.50 |
Median | 49.0 | 50.0 |
5% Trimmed mean | 53.75 | 55.66 |
m-estimator | 52.89 | 54.61 |
Std Dev | 28.11 | 300.90 |
MAD | 30.39 | 31.13 |
IQR | 40.25 | 41.00 |
Range | 123 | 2491 |
The mean, standard deviation, and range are massively inflated. The median, trimmed mean, m-estimator, MAD, and IQR barely change. This is the core argument for robust estimators when outliers are present.
(Ȳ, S) provides a more complete picture and is more efficient when the data is roughly symmetric with few outliers (near-Normal shape).
(Median, MAD) is more robust but less efficient under normality.
Three broad classes of robust estimators exist:
R-estimators: based on ranks of the data (rank-based regressions)
M-estimators: minimise an objective function (generalised least squares)
L-estimators: linear combinations of the order statistics (trimmed means, median, trimean, etc.)
L-estimator comparisons (from variance tables):
For distributions close to Normal: the sample mean or a small-trim mean (5-10%) does best.
For moderate-tailed distributions (Logistic, One-Out, One-Wild): the MidMean (25% trim) or T(20%) performs well.
For heavy-tailed distributions (Slash, Cauchy, Double Exponential): higher-trim estimators like T(30%) or the median perform best. The untrimmed mean can have infinite variance.
The "best overall" estimator depends on which family of distributions you expect. The T(20%) to MidMean(25%) range represents a practical compromise for many real-world situations.
Sample mean: Ȳ = (1/n) × Σ Y_i
Sample variance (unbiased): S² = [1/(n−1)] × Σ(Y_i − Ȳ)²
Sample skewness (modified): β̂*₁ = [n² / ((n−1)(n−2))] × μ̂₃ / (S²)^(3/2)
MAD: Median{|Y_i − Median(Y)|} / 0.6745
m-estimator: m̂ = Σ(w_i × Y_i) / Σ(w_i), weights updated iteratively
Trimmed mean (R): mean(y, trim = α)
⚠️ The EDF-based σ̂ uses divisor n, while the unbiased S uses divisor n−1. Know which one is being asked for.
⚠️ S² is unbiased for σ², but S is biased for σ. This is a common exam trap.
⚠️ The modified skewness and kurtosis estimators are unbiased only under normality. For non-Normal data, that property does not hold.
⚠️ When nα is not an integer, the α-trimmed mean trims slightly less than α from each tail. Know the formula for K(α).
⚠️ Some software reports excess kurtosis (β̂₂ − 3) rather than kurtosis itself. For the Normal distribution, excess kurtosis = 0.
⚠️ The mean and standard deviation can be hugely distorted by even a few outliers. The median, MAD, and IQR are far more resistant. Exam questions often ask you to compare estimators with and without outliers.
⚠️ For the m-estimator, a smaller tuning constant t means more aggressive downweighting of extreme values.
Q: What is the difference between the EDF-based standard deviation (σ̂) and the sample standard deviation S?
A: σ̂ = √[(1/n) × Σ(Y_i − Ȳ)²] uses divisor n. S = √[(1/(n−1)) × Σ(Y_i − Ȳ)²] uses divisor n−1. S² is unbiased for σ² under any distribution with finite variance.
Q: Why is S a biased estimator of σ even though S² is unbiased for σ²?
A: Because the square root is a concave function, Jensen's inequality tells us E[√X] ≤ √E[X]. So E[S] = E[√S²] < √E[S²] = √σ² = σ.
Q: If n = 20 and α = 0.10, how many observations are trimmed from each tail for the α-trimmed mean?
A: nα = 2, which is an integer, so exactly 2 observations are trimmed from each end. K(α) = 20 − 2 − 3 + 1 = 16 values remain. Exactly 10% is trimmed from each tail.
Q: What does the constant 0.6745 accomplish in the MAD formula?
A: It makes MAD a consistent estimator of σ when the data comes from a Normal distribution. Specifically, 0.6745 is the 75th percentile of the standard Normal, so Median{|Z|} = 0.6745 for Z ~ N(0,1).
Q: Name three estimators of centre that are robust to outliers.
A: The sample median, the α-trimmed mean, and the m-estimator.
Q: In the ozone data example, the mean jumped from 54.69 to 106.50 when five outliers were added, but the median moved only from 49.0 to 50.0. Why?
A: The mean gives equal weight to every observation, so extreme values pull it strongly. The median depends only on the rank ordering near the centre of the data, so a few extreme additions barely shift it.
distribution-free, nonparametric, empirical distribution function, EDF, sample mean, sample variance, sample standard deviation, unbiased estimator, sample median, quartiles, five number summary, interquartile range, IQR, SIQR, range, trimmed mean, MAD, median absolute deviation, m-estimator, robust estimation, skewness, kurtosis, excess kurtosis, order statistics, L-estimator, R-estimator, outlier sensitivity, ozone data example, Stamford Yonkers, STAT principles of statistics, Texas A&M