Source: Handout 01, Principles of Statistics I (Texas A&M University) | References: Snee (1983), Journal of Quality Technology; Hogg-Ledoleter (1992), Applied Statistics for Engineers and Physical Scientist; Rickards (2008), Washington Post
Tags: designed experiment, variance components, sources of variation, nested design, operator variation, specimen variation, combustion run, chemical analysis, Challenger disaster, O-ring failure, scatter plot, extrapolation, relevant data, value at risk, VaR, bell curve, fat tails, complexity theory, risk models
Three case studies illustrate different lessons. A chemical analysis experiment shows how to identify and measure sources of variation in a process (here, 94.8% of variation came from operators). The Challenger disaster shows the catastrophic consequences of plotting only partial data and extrapolating beyond observed ranges. A Washington Post article on "Value at Risk" models shows how assuming risk follows a bell curve led Wall Street and regulators to underestimate catastrophic events.
Variance components
The portions of total variability in a measurement that are attributable to each identifiable source (e.g., operator, specimen, equipment run, analysis procedure).
Nested design
An experimental structure where one factor is nested within another. In the DMZ experiment, specimens are nested within operators, runs are nested within specimens and operators, and analyses are nested within runs, specimens, and operators.
Scatter plot
A graph that plots one variable against another, with each data point represented as a dot. Essential for identifying relationships between variables.
Extrapolation
Making predictions outside the range of observed data. Extrapolation is dangerous because the relationship that holds within the data range may not hold outside it.
Value at Risk (VaR)
A risk management model that estimates, with a stated probability (typically 99%), the maximum loss a portfolio can suffer over a given period, based on historical patterns.
Fat tails
The phenomenon where extreme events occur more frequently than a normal (bell curve) distribution predicts. Modelled by adding upward bends to the tails of the distribution.
Complex dynamic system
A system in which many interdependent components interact in ways that produce emergent, unpredictable behaviour not evident from examining individual parts. Capital markets are an example.
Context. A chemical analysis measures the amount of DMZ in raw materials before injection into an industrial process. The analysis involves multiple operators, specimens, combustion furnace runs, and duplicate chemical analyses. The question: which of these sources contributes the most variation?
Experimental design.
3 operators, randomly selected
Each operator analysed 2 specimens (6 specimens total)
Each specimen was split into 3 units, each placed in a separate combustion furnace run (18 runs total)
Each run was titrated in duplicate (36 chemical analyses total)
This is a nested (hierarchical) design: Specimen is nested within Operator, Run is nested within Specimen and Operator, Chemical Analysis is nested within Run, Specimen, and Operator.
Four potential sources of variation:
Operator (O): systematic differences in how operators follow procedures
Specimen within Operator (S(O)): variation in raw material specimens analysed by the same operator
Combustion Run within Specimen and Operator (R(S,O)): variation from run to run in furnace conditions, same specimen and operator
Chemical Analysis within Run, Specimen, and Operator (A(R,S,O)): variation in the titration, same run/specimen/operator
Results. The variance component analysis revealed:
Operator: 94.80% of total variation
Specimen: 0.00%
Combustion Run: 3.83%
Chemical Analysis: 1.29%
Other sources: 0.08%
The dominant source is operator-to-operator differences. Operator 1 was consistent with high values, Operator 2 was consistently lower, and Operator 3 was consistently higher still. Within-operator variation was comparatively small.
Key observations from the chemical variation plot:
Operator 1: very consistent results across both specimens and all three runs
Operator 2: lower average than the other two; large variation across runs within specimens, and at least one duplicate analysis on Specimen 4 with substantial spread
Operator 3: good consistency between the two specimens, but some combustion runs on Specimen 5 had noticeably larger duplicate-analysis variation
Hypothetical comparison. If all variation were due to combustion runs alone (no operator, specimen, or analysis variation), every operator mean, every specimen mean, and every duplicate pair would be identical. Only run means would differ. The hypothetical data table in the handout illustrates this scenario, where all operator means equal 147.75.
Context. On 28 January 1986, the Challenger space shuttle launched at roughly 31°F. Engineers had warned that O-ring failures might be related to temperature, but the launch proceeded.
The data error. The night before launch, engineers plotted O-ring distress against temperature using only the seven launches where failures had occurred. That plot showed no clear relationship.
The missing data. They excluded the 17 launches with zero failures, all of which occurred at temperatures of 66°F or higher.
The complete picture. When all 24 previous launches are plotted:
0 distressed O-rings per launch: 17 launches, all at T >= 66°F
1 distressed O-ring: 5 launches, 55 <= T <= 70°F
2 distressed O-rings: 1 launch, T = 75°F
3 distressed O-rings: 1 launch, T = 52°F
At temperatures above 60°F, only 4 of 21 launches (19%) had O-ring distress. Below 60°F, 3 of 3 launches (100%) had O-ring distress.
The extrapolation problem. Challenger launched at 31°F. The lowest previous launch temperature was 51°F. Any prediction at 31°F was an extrapolation beyond the observed data, which is always dangerous.
Lessons:
Always plot all relevant data, including cases where the outcome did not occur
Scatter plots of one variable against another are essential for spotting relationships
Extrapolation beyond the range of observed data is risky and should be flagged
Ron Snee's principle: "In God We Trust; Others Must Have Data."
Context. A 2008 Washington Post article by James Rickards explained how reliance on "Value at Risk" models contributed to the financial crisis.
How VaR works. The model assigns risk factors to every security, aggregates across a portfolio, nets out cancelling risks, and uses historical patterns to predict (at 99% probability) the maximum possible loss. If capital exceeds this figure, the institution is considered sound.
The conceptual error. VaR assumes risk is randomly distributed and that events are independent (like coin tosses). Under this assumption, catastrophic events are so improbable as to be practically impossible.
Why it failed. Financial markets are not like coin tosses:
Events are not independent; what happens today affects what happens tomorrow
Risk is not shaped like a bell curve; extreme events occur far more often than a normal distribution predicts
Capital markets are complex dynamic systems with emergent properties, where aggregate risk exceeds the sum of individual risks
Historical experience does not bound the size of possible events; the system's scale is the only limit
The avalanche analogy. VaR practitioners focus on individual snowflakes (causes and effects). Complexity theorists study the mountain (the system's instability). Ski patrols throw dynamite down slopes to prevent instability from building, which is a form of systemic regulation.
Fat tails are not enough. Some VaR practitioners add "fat tails" to model extreme events. But in a complex system, events are not confined to historical extremes. Unprecedented scale produces unprecedented catastrophes.
⚠️ The DMZ experiment is a textbook example of nested (hierarchical) design and variance component analysis. Be able to list the four sources of variation and explain the nesting structure.
⚠️ The Challenger example is the classic cautionary tale about plotting only partial data and about the dangers of extrapolation. Expect exam questions on both points.
⚠️ Know what it means for 94.8% of variation to be attributable to operators: the measurement process is dominated by who does the analysis, not by the equipment or material.
⚠️ The VaR case study illustrates the consequences of wrong model assumptions (independence, normal distribution). Relevant to later topics on probability distributions and model assessment.
⚠️ "In God We Trust; Others Must Have Data" (Ron Snee) may appear as a quotation-identification question.
Q: In the DMZ chemical analysis experiment, what percentage of total variation was attributable to operators?
A: 94.80%.
Q: What does it mean for Specimen variation to be 0.00% in this experiment?
A: The two specimens analysed by each operator produced virtually identical average results, so specimen-to-specimen differences within an operator contributed no measurable variation to the total.
Q: What critical mistake did engineers make when analysing O-ring data the night before the Challenger launch?
A: They plotted only the seven launches where O-ring failures occurred and excluded the 17 launches with zero failures. Including all data would have revealed a clear relationship between low temperature and O-ring distress.
Q: Why is extrapolation particularly dangerous in the Challenger context?
A: The launch temperature (31°F) was far below the lowest previously observed launch temperature (51°F). There was no data in this range to support any prediction, and the relationship between temperature and failure may behave differently at extreme values.
Q: What is the fundamental conceptual error behind Value at Risk models?
A: The assumption that risk is randomly distributed and that each market event is independent of the next, like coin tosses. In reality, financial markets are complex dynamic systems where events are interdependent and extreme outcomes occur far more often than a normal distribution predicts.
Q: What does the avalanche analogy illustrate about VaR vs. complexity theory?
A: VaR focuses on individual triggers (snowflakes), while complexity theory focuses on the system's overall instability (the mountain). The lesson is that it is the system's structure, not any single event, that determines catastrophic risk.
designed experiment, variance components, nested design, hierarchical design, sources of variation, operator variation, specimen variation, combustion run variation, chemical analysis variation, process variation study, Snee 1983, Challenger disaster, O-ring failure, scatter plot, plotting all data, extrapolation danger, relevant data, value at risk, VaR, bell curve, normal distribution, fat tails, complex dynamic systems, complexity theory, emergent properties, financial crisis 2008, Long-Term Capital Management, independent events, risk management