Source: Principles of Statistics I (Texas A&M), Tamhane/Dunlop Ch. 2 & 4
Tags: numerical summaries, parametric estimation, graphical estimators, method of moments, MOM, maximum likelihood estimation, MLE, Weibull, exponential, normal, gamma, location-scale family, likelihood function, log-likelihood, Bayesian inference, prior distribution, posterior distribution, point estimators
When the underlying distribution family is known (e.g. Normal, Weibull, Gamma), we can estimate its parameters using graphical methods, Method of Moments (MOM), or Maximum Likelihood Estimation (MLE). Once the distribution parameters are estimated, plug them into the formulas for population summaries like mean and standard deviation to get parametric estimators of those quantities.
Parametric family
A collection of probability distributions indexed by one or more unknown parameters (e.g. Normal indexed by mean and variance).
Location-scale family
A family of distributions whose CDF can be written in terms of a location parameter (shift) and a scale parameter (stretch). Examples: Normal, Logistic, Cauchy. The Weibull is not a location-scale family in the original variable, but becomes one after a log transformation.
Point estimator
A single-value estimate of an unknown population parameter, computed from sample data.
Method of Moments estimator (MOM)
An estimator obtained by setting sample moments equal to population moments and solving for the unknown parameters.
Maximum Likelihood estimator (MLE)
The parameter value(s) that maximise the likelihood function, i.e. the joint probability of the observed data given the parameters.
Likelihood function, L(θ; y)
The joint pdf of the sample evaluated at the observed data, treated as a function of the parameter(s) θ rather than of the data.
Log-likelihood, l(θ; y)
The natural logarithm of the likelihood function. Maximising l is equivalent to maximising L and is usually easier because products become sums.
Gamma function, Γ(a)
Defined as the integral from 0 to infinity of y^(a−1) × e^(−y) dy. Extends the factorial to non-integer arguments: Γ(n) = (n−1)! for positive integers. Used in Weibull and Gamma distribution formulas. In R, computed with gamma(x).
Prior distribution (Bayesian)
A distribution for the unknown parameter θ that encodes the researcher's beliefs before data collection.
Posterior distribution (Bayesian)
The updated distribution of θ after combining the prior with the observed data via the likelihood.
The handout covers three parametric methods plus a brief introduction to Bayesian inference:
Graphical estimators of location-scale parameters (Method 1)
Method of Moments, MOM (Method 2)
Maximum Likelihood Estimation, MLE (Method 3)
Bayesian inference (alternative to MLE, treats parameters as random)
The general workflow is the same across methods: estimate the distribution's parameters first, then substitute those estimates into the formulas for population summaries (mean, standard deviation, etc.).
Throughout the handout, a single dataset anchors the methods: times to failure (in 100 hours) for 25 newly designed fuel pumps. The assumed distribution is Weibull with CDF:
F(y) = 1 − e^(−(y/α)^γ)
where α is the scale parameter and γ is the shape parameter, both unknown.
The population mean and standard deviation of the Weibull are:
μ = α × Γ(1 + 1/γ)
σ = √[ α² × ( Γ(1 + 2/γ) − Γ²(1 + 1/γ) ) ]
This approach works when the CDF belongs to a location-scale family, or can be transformed into one.
Core idea:
Let Q₀(u) be the quantile function of the standard (reference) member of the family.
Let Q̂(u) be the sample quantile for u_i = (i − 0.5)/n.
Plot Q̂(u_i) versus Q₀(u_i). If the points fall roughly on a straight line, read off the intercept and slope.
Intercept = θ̂₁ (location estimate), Slope = θ̂₂ (scale estimate).
Weibull adaptation:
The Weibull is not itself a location-scale family, but W = log(Y) has a distribution that is. After the log transformation:
Location parameter: θ₁ = log(α)
Scale parameter: θ₂ = 1/γ
Standard quantile function: Q₀(u) = log(−log(1 − u))
From the fuel pump example, the fitted line on the Weibull reference distribution plot was:
Q̂_W(u_i) = 2.414 + 0.952 × Q₀(u_i)
So θ̂₁ = 2.414 and θ̂₂ = 0.952, giving:
γ̂ = 1/0.952 ≈ 1.05 (close to 1, suggesting an Exponential distribution)
α̂ = e^(2.414) ≈ 11.18
Plugging into the Weibull mean and SD formulas: μ̂ ≈ 10.96, σ̂ ≈ 10.44.
For comparison, the distribution-free estimates from the same data: Ȳ = 11.57, S = 12.29.
Core idea:
Write the first k population moments m_i = E[Y^i] as functions of the k unknown parameters.
Compute the corresponding sample moments: m̂_i = (1/n) × Σ Y_j^i.
Set sample moments equal to population moments and solve the resulting k equations in k unknowns.
Example 1: Normal(θ₁, θ₂²)
m₁ = E[Y] = θ₁
m₂ = E[Y²] = θ₂² + θ₁²
Solving: θ̂₁ = Ȳ, and θ̂₂ = √[(1/n) × Σ(Y_i − Ȳ)²].
Note this uses the divisor n, not n−1.
Example 2: Gamma(α, β)
m₁ = αβ
m₂ = αβ² + (αβ)²
Solving: β̂ = (m̂₂ − Ȳ²)/Ȳ, and α̂ = Ȳ²/(m̂₂ − Ȳ²).
Example 3: Weibull(γ, α)
The moment equations involve the Gamma function and cannot be solved in closed form. Numerical methods are required.
Core idea:
For iid observations Y₁, ..., Yₙ with pdf f(y; θ), the likelihood function is: L(θ; y) = Π f(y_i; θ).
The MLE is the value θ̂ that maximises L (or equivalently, the log-likelihood l).
Take the derivative of l with respect to each parameter, set equal to zero, and solve. Verify you have a maximum using second-derivative tests.
MLE uses the full structure of the distribution, whereas MOM uses only the first few moments. This often makes MLE more efficient (lower variance) for a given sample size.
Example 1: Exponential(β)
L(β) = (1/β)ⁿ × e^(−(1/β) × Σt_i)
l(β) = −n × log(β) − (1/β) × Σt_i
Setting dl/dβ = 0 gives β̂ = T̄ (the sample mean).
The second derivative is negative at β̂, confirming a maximum.
For the fuel pump data: β̂ = T̄ = 11.57. In R, fitdistr(t, "exponential") returns the rate = 1/β̂.
Example 2: Normal(θ₁, θ₂²)
l(θ₁, θ₂) = −(n/2) × log(2π) − n × log(θ₂) − (1/(2θ₂²)) × Σ(y_i − θ₁)²
θ̂₁ = Ȳ (same as MOM)
θ̂₂ = √[(1/n) × Σ(y_i − Ȳ)²] (same as MOM)
To verify these are maxima, check second and mixed partial derivatives. Let A = ∂²l/∂θ₁², B = ∂²l/(∂θ₁∂θ₂), C = ∂²l/∂θ₂². If B² − AC < 0 and A + C < 0, you have a relative maximum.
Example 3: Gamma(α, β)
No closed-form solution because of the Gamma function and its derivative. Use numerical methods. In R: library(MASS); fitdistr(y, "gamma") returns Shape (α̂) and Rate (1/β̂).
Example 4: Weibull(γ, α)
Taking partial derivatives of the log-likelihood with respect to α and γ and setting them to zero yields two equations that must be solved numerically.
From the fuel pump data (R output): γ̂ = 0.984, α̂ = 11.485 (with standard errors 0.151 and 2.466 respectively). SAS proc lifereg gives identical results.
Using these MLEs: μ̂ = 11.57, σ̂ = 11.76. Compare with the distribution-free Ȳ = 11.57 and S = 12.28. The mean estimates match closely; the standard deviation estimates differ slightly.
Bayesian methods differ from MLE in a fundamental way: the unknown parameter θ is treated as a random variable with its own distribution, rather than a fixed unknown constant.
Workflow:
Specify a prior distribution g(θ) before collecting data (encodes prior knowledge).
Compute the likelihood from the data, as in MLE.
Combine them to get the posterior distribution: h(θ | data) ∝ Likelihood × Prior.
The mode of the posterior is the "estimate" of θ; probability intervals replace confidence intervals.
Key consideration: the validity of Bayesian results depends on the appropriateness of the chosen prior. When prior knowledge is limited, a flat (uninformative) prior is often used.
Weibull CDF: F(y) = 1 − e^(−(y/α)^γ)
Weibull mean: μ = α × Γ(1 + 1/γ)
Weibull SD: σ = √[ α² × (Γ(1 + 2/γ) − Γ²(1 + 1/γ)) ]
MOM general: Set m̂_i = (1/n) × Σ Y_j^i equal to the population moments expressed as functions of the parameters.
MLE general: Maximise L(θ; y) = Π f(y_i; θ), typically via the log-likelihood.
Exponential MLE: β̂ = T̄
Normal MLEs: θ̂₁ = Ȳ, θ̂₂ = √[(1/n) × Σ(Y_i − Ȳ)²]
Bayesian posterior: h(θ | data) = [f(data | θ) × g(θ)] / Σ[f(data | θ) × g(θ)]
⚠️ MOM and MLE give the same estimators for the Normal distribution, but this is a coincidence. They differ for most other distributions.
⚠️ The MLE of σ for the Normal uses divisor n, not n−1. The unbiased estimator S uses n−1. Know which one a question is asking for.
⚠️ The Weibull family is not a location-scale family in Y, but log(Y) does belong to a location-scale family. This transformation trick is commonly tested.
⚠️ For Gamma and Weibull MLEs, closed-form solutions do not exist. You need numerical methods or software (R fitdistr, SAS proc lifereg).
⚠️ When γ̂ ≈ 1 in a Weibull fit, the data can be modelled as Exponential (Weibull with γ = 1 is the Exponential).
⚠️ To confirm an MLE is a maximum (not a minimum or saddle point), you must check second derivatives. For two parameters, use the B² − AC test.
Q: What are the three parametric methods for estimating distribution parameters covered in Handout 06?
A: Graphical estimators (for location-scale families), Method of Moments (MOM), and Maximum Likelihood Estimation (MLE).
Q: For a Weibull distribution, why do we take W = log(Y) before constructing a graphical estimator?
A: Because the Weibull family in Y is not a location-scale family (γ is a shape parameter, not location or scale). After the log transformation, W has a distribution that is a location-scale family with location parameter log(α) and scale parameter 1/γ.
Q: What is the MLE of β for an Exponential(β) distribution?
A: β̂ = T̄, the sample mean of the observations.
Q: For a Normal(θ₁, θ₂²) distribution, what are the MOM estimators of θ₁ and θ₂?
A: θ̂₁ = Ȳ and θ̂₂ = √[(1/n) × Σ(Y_i − Ȳ)²]. Note the divisor is n, not n−1.
Q: Why might MLE be preferred over MOM?
A: MLE uses the full structure (pdf) of the distribution, whereas MOM uses only the first few moments. This typically makes MLE more statistically efficient (lower variance estimators).
Q: In the second-derivative test for two-parameter MLE, what conditions confirm a maximum?
A: Compute A = ∂²l/∂θ₁², B = ∂²l/(∂θ₁∂θ₂), C = ∂²l/∂θ₂² at the solution. If B² − AC < 0 and A + C < 0, the solution is a relative maximum.
Q: How does Bayesian estimation differ fundamentally from MLE?
A: In MLE, parameters are treated as fixed but unknown constants. In Bayesian estimation, parameters are treated as random variables with a prior distribution, which is updated using observed data to produce a posterior distribution.
parametric estimation, method of moments, MOM, maximum likelihood, MLE, likelihood function, log-likelihood, Weibull distribution, exponential distribution, normal distribution, gamma distribution, location-scale family, graphical estimators, reference distribution plot, quantile function, Bayesian inference, prior distribution, posterior distribution, point estimator, fitdistr R, proc lifereg SAS, gamma function, shape parameter, scale parameter, STAT principles of statistics, Texas A&M, Tamhane Dunlop