Source: Tamhane/Dunlop Ch. 2, Sections 4.1, 4.2, 4.3.4
Tags: quantile function, sample quantile, sample median, fractional order statistics, Parzen quantile, smoothed quantile, inverse CDF, percentile estimator, Q-hat
The sample quantile function is the inverse of the empirical CDF. The raw version is a step function that has two known problems: it does not match the standard sample median definition for even n, and it assigns the sample min/max to Q(0) and Q(1), which is unrealistic. The continuous (smoothed) version fixes both issues by using fractional order statistics and linear interpolation. Different software packages (R, SAS, Excel, Minitab) use slightly different formulas, all unified by a two-parameter family.
Quantile function Q(u)
The inverse of the CDF: Q(u) = F-inverse(u) = inf{y : F(y) ≥ u} for 0 ≤ u ≤ 1. It returns the value of Y below which at least 100u% of the distribution lies.
Raw sample quantile Q-hat_R(u)
The inverse of the raw EDF. It is a piecewise constant (step) function of u, expressed in terms of order statistics: Q-hat_R(u) = Y_(j) for (j-1)/n < u ≤ j/n.
Continuous sample quantile Q-hat_C(u)
A piecewise linear, continuous version of the sample quantile defined using fractional order statistics. Also called the Parzen quantile.
Fractional order statistic Y_(k+r)
A linearly interpolated value between two adjacent order statistics: Y_(k+r) = (1 - r) × Y_(k) + r × Y_(k+1), where k is an integer and 0 < r < 1.
Sample median
The middle value of a sorted sample. For odd n: Y_((n+1)/2). For even n: the average of Y_(n/2) and Y_((n+2)/2).
The raw sample quantile is the formal inverse of the EDF:
Q-hat_R(u) = inf{y : F-hat(y) ≥ u}
In terms of order statistics: Q-hat_R(u) = Y_(j) for (j-1)/n < u ≤ j/n, where j = 1, 2, ..., n
Spelled out for a sample of size n:
Q-hat_R(u) = Y_(1) for 0 < u ≤ 1/n
Q-hat_R(u) = Y_(2) for 1/n < u ≤ 2/n
... and so on up to ...
Q-hat_R(u) = Y_(n) for (n-1)/n < u ≤ 1
The raw quantile function is a 90-degree rotation of the mirror image of the EDF:
Flat regions in F-hat become jumps in Q-hat_R
Jumps in F-hat become flat regions in Q-hat_R
F-hat is right-continuous; Q-hat_R is left-continuous
These mirror the same relationships that hold for the population CDF and quantile function.
Problem 1: disagreement with the standard sample median
For odd n, Q-hat_R(0.5) = Y_((n+1)/2), which matches the sample median. No issue.
For even n, Q-hat_R(0.5) = Y_(n/2), which does not match the sample median. The standard definition averages Y_(n/2) and Y_((n+2)/2).
Problem 2: unrealistic boundary estimates
Q-hat_R(0) = Y_(1) and Q-hat_R(1) = Y_(n)
The sample minimum and maximum are poor estimators of the population minimum and maximum, especially for small n
The step function shape is also unrealistic for a continuous population quantile function
For integer k (1 ≤ k ≤ n - 1) and 0 < r < 1:
Y_(k+r) = Y_(k) + r × [Y_(k+1) - Y_(k)] = (1 - r) × Y_(k) + r × Y_(k+1)
Example: Y_(2.4) = 0.6 × Y_(2) + 0.4 × Y_(3). The value sits 40% of the way from Y_(2) to Y_(3).
Defined as:
Q-hat_C(u) = Y_(nu + 0.5) for 1/(2n) ≤ u ≤ 1 - 1/(2n)
Q-hat_C(u) is undefined for 0 ≤ u < 1/(2n) and 1 - 1/(2n) < u ≤ 1
Why leave the tails undefined? It avoids the problem of assigning unrealistic estimates to Q(u) for extreme values of u.
The function connects the points Y_(j) = Q-hat_C((2j - 1)/(2n)) with straight lines:
Y_(1) is assigned to u = 1/(2n)
Y_(2) is assigned to u = 3/(2n)
Y_(n) is assigned to u = 1 - 1/(2n)
For odd n: Q-hat_C(0.5) = Y_((n+1)/2), since (n+1)/2 is an integer
For even n: Q-hat_C(0.5) = 0.5 × Y_(n/2) + 0.5 × Y_((n+2)/2), the standard average
This resolves Problem 1 from the raw quantile.
All major software packages use piecewise linear quantile functions. They differ in which value of u they assign to each order statistic Y_(j).
The unified formula:
Q-hat(u; a, b) = Y_((n - b)u + a)
Software / Method | a | b | Assignment: Y_(j) at u = | Notes |
|---|---|---|---|---|
Excel | 1 | 1 | (j - 1)/(n - 1) | Q-hat(0) = Y_(1), Q-hat(1) = Y_(n) – same boundary problem |
Minitab / Tamhane-Dunlop | 0 | -1 | j/(n + 1) | Undefined for u < 1/(n+1) and u > n/(n+1) |
SAS / SPSS default | 0.5 | 0 | (j - 0.5)/n | Parzen quantile, same as Q-hat_C |
R default | 1 | 1 | (j - 1)/(n - 1) | Same as Excel |
R type=5 | 0.5 | 0 | (j - 0.5)/n | Nearly Parzen, but defines Q-hat at the boundaries |
Tukey recommendation | 3/8 | 1/4 | (j - 3/8)/(n - 1/4) | Slightly different spacing |
To get the Parzen definition in R, specify quantile(x, type=5).
Raw sample quantile
Q-hat_R(u) = Y_(j) for (j-1)/n < u ≤ j/n, j = 1, ..., n
Fractional order statistic
Y_(k+r) = (1 - r) × Y_(k) + r × Y_(k+1)
Continuous (Parzen) sample quantile
Q-hat_C(u) = Y_(nu + 0.5) for 1/(2n) ≤ u ≤ 1 - 1/(2n)
Unified quantile definition
Q-hat(u; a, b) = Y_((n - b)u + a)
⚠️ Know both problems with the raw sample quantile (median mismatch for even n, boundary problem) and how the continuous version resolves them.
⚠️ Be comfortable computing fractional order statistics by hand. Example: "Find Q-hat_C(0.3) for a sample of size n = 10" requires computing Y_(10 × 0.3 + 0.5) = Y_(3.5) = 0.5 × Y_(3) + 0.5 × Y_(4).
⚠️ The unified formula Q-hat(u; a, b) = Y_((n-b)u + a) appears across multiple software packages. Know how to identify which package corresponds to which (a, b) pair.
⚠️ R's default quantile function uses the Excel definition (a = 1, b = 1), not the Parzen definition. To get Parzen, use type=5.
Q: For a sample of size n = 8, what is Q-hat_R(0.5)?
A: Since n is even, (j-1)/n < 0.5 ≤ j/n gives j = 4. So Q-hat_R(0.5) = Y_(4), which does not equal the sample median (Y_(4) + Y_(5))/2.
Q: What is the fractional order statistic Y_(3.7) in terms of Y_(3) and Y_(4)?
A: Y_(3.7) = 0.3 × Y_(3) + 0.7 × Y_(4).
Q: Using the Parzen definition with n = 10, at what value of u is the order statistic Y_(1) plotted?
A: At u = 1/(2 × 10) = 1/20 = 0.05.
Q: Why is Q-hat_C(u) left undefined for very small and very large values of u?
A: To avoid the unrealistic claim that the sample minimum and maximum are good estimates of the population minimum and maximum, especially for small samples.
Q: What values of a and b in the unified formula give the R default quantile?
A: a = 1 and b = 1, yielding Q-hat(u) = Y_((n-1)u + 1).
quantile function, sample quantile, inverse CDF, percentile estimator, fractional order statistics, Parzen quantile, continuous quantile, smoothed quantile, sample median, order statistics, R quantile type, Excel quantile, SAS quantile, SPSS quantile, Minitab quantile, Tukey quantile, Q-hat, piecewise linear quantile, Principles of Statistics I, Texas A&M, STAT, Tamhane Dunlop