Source: Moore, McCabe & Craig, Introduction to the Practice of Statistics, 10th Ed., Ch. 3–4
Tags: sampling, SRS, bias, experiment, observational study, probability, sample space, conditional probability, independence, Bayes, ECON 2500
Difficulty: Introductory to Intermediate | Prerequisites: Ch. 1-2 notes (distributions, scatterplots, correlation)
Chapter 3 is about where data come from, which determines what conclusions you can draw. Observational studies can show association; only experiments with randomisation can establish causation. Chapter 4 introduces probability, the mathematical language of uncertainty that underpins all of statistical inference. Without Chapter 3, you will not understand why study design matters. Without Chapter 4, confidence intervals and hypothesis tests (Chapters 6-8) will make no sense. These two chapters bridge the descriptive world of Chapters 1-2 and the inferential world of Chapters 5-9.
Chapter 3 covers how to collect data well: sampling designs (especially simple random samples), sources of bias, and the principles of experimental design (control, randomisation, replication). Chapter 4 builds the rules of probability, including conditional probability, independence, and basic counting, so that you can reason formally about random events.
Population
The entire group of individuals you want information about.
Sample
A subset of the population that you actually collect data from. In simple terms, the slice of the whole that you can measure.
Simple random sample (SRS)
A sample chosen so that every possible sample of size n has an equal chance of being selected. The gold standard for avoiding selection bias.
Bias
A systematic tendency to favour certain outcomes over others in the design or conduct of a study. Think of it as a built-in tilt that makes your results unrepresentative.
Voluntary response sample
A sample consisting of people who choose to participate on their own. These are almost always biased because people with strong opinions are overrepresented.
Convenience sample
A sample chosen because it is easy to reach, not because it is representative.
Observational study
A study that records information about individuals without attempting to influence the outcomes. You observe what happens naturally.
Experiment
A study that deliberately imposes a treatment on individuals to observe their responses. The key feature is that the researcher controls who gets what.
Randomisation
Using a chance process to assign subjects to treatments. This balances out the effects of lurking variables, both known and unknown.
Control group
A group that receives no treatment (or a placebo) to serve as a baseline for comparison.
Placebo
A dummy treatment that looks like the real treatment but has no active ingredient. Used to control for the placebo effect.
Double-blind
Neither the subjects nor those who evaluate the results know which treatment each subject received. Prevents both subject and evaluator bias.
Confounding
Two variables are confounded when their effects on the response cannot be separated from each other.
Sample space (S)
The set of all possible outcomes of a random phenomenon.
Event
A subset of the sample space, a collection of outcomes. In simple terms, any result or set of results you might care about.
Probability
The long-run proportion of times an outcome would occur if the random phenomenon were repeated many times.
Conditional probability
The probability of event A given that event B has already occurred, written P(A|B). In simple terms, "now that we know B happened, what are the chances of A?"
Independent events
Two events are independent if knowing that one occurred does not change the probability of the other. Formally, P(A|B) = P(A).
Simple random sample (SRS): Every individual and every group of n individuals has an equal chance of selection. Use a random number table or generator. This is the design that probability theory relies on.
Stratified random sample: Divide the population into homogeneous groups (strata, e.g. by income bracket), then take an SRS within each stratum. Gives more precise estimates than an SRS of the same total size.
Cluster sample: Divide the population into clusters (e.g. classrooms), randomly select some clusters, and sample everyone within the chosen clusters. Cheaper when individuals are naturally grouped.
Multistage sample: Combine methods, sampling in stages (e.g. first select provinces, then cities, then households).
Undercoverage: Some members of the population are left out of the sampling frame.
Nonresponse: Selected individuals cannot be contacted or refuse to participate.
Response bias: Answers are systematically inaccurate because of question wording, interviewer influence, or social desirability.
Voluntary response: Only people who feel strongly bother to respond, skewing results.
An observational study observes individuals and measures variables but does not attempt to influence responses. It can show association but cannot prove causation because of potential confounding.
An experiment deliberately applies treatments to subjects. With proper randomisation, experiments can establish causal relationships because randomisation balances confounding variables across groups.
Control: Compare the treatment group to a control group.
Randomisation: Use chance to assign subjects to groups, balancing out lurking variables.
Replication: Use enough subjects so that the effects of chance are small relative to the treatment effect.
Blocking: Group similar subjects together before randomisation (e.g. block by gender), then randomise within each block. The analogy to stratification in sampling.
Matched pairs: A special case of blocking with two treatments or a before-and-after comparison on the same individual.
The probability of any event A is between 0 and 1: 0 ≤ P(A) ≤ 1.
The probability of the entire sample space is 1: P(S) = 1.
Complement rule: P(A does not occur) = 1 - P(A). Written as P(Ac) = 1 - P(A).
Disjoint (mutually exclusive) events: Events that cannot happen at the same time. P(A or B) = P(A) + P(B).
General addition rule: For any two events, P(A or B) = P(A) + P(B) - P(A and B). You subtract the overlap to avoid double-counting.
Independent events: P(A and B) = P(A) x P(B). Knowing one happened does not change the probability of the other.
General multiplication rule: P(A and B) = P(A) x P(B|A). Always works, whether events are independent or not.
P(A|B) = P(A and B) / P(B)
This is the probability of A occurring given that B has occurred. Read it as "the probability of A, given B."
When you know P(B|A) and want P(A|B):
P(A|B) = P(B|A) x P(A) / P(B)
where P(B) can be expanded using the law of total probability:
P(B) = P(B|A) x P(A) + P(B|Ac) x P(Ac)
This is essential for "reversing" conditional probabilities, e.g. finding the probability someone has a disease given a positive test result.
A discrete random variable takes a countable number of values. Its probability distribution lists each value and its probability; the probabilities sum to 1.
A continuous random variable takes values in an interval. Probabilities are areas under a density curve.
Expected value (mean) of X: mu = E(X) = sum of [xi x P(xi)]. The long-run average if the experiment were repeated many times.
Variance of X: sigma-squared = sum of [(xi - mu) squared x P(xi)]. Measures how spread out the distribution is.
Standard deviation: sigma = square root of the variance.
E(a + bX) = a + b x E(X)
Var(a + bX) = b-squared x Var(X)
For independent random variables X and Y: E(X + Y) = E(X) + E(Y), and Var(X + Y) = Var(X) + Var(Y). Variances add for independent variables; standard deviations do not.
Complement rule: P(Ac) = 1 - P(A)
General addition rule: P(A or B) = P(A) + P(B) - P(A and B)
Disjoint addition rule: P(A or B) = P(A) + P(B) (only when A and B cannot both occur)
General multiplication rule: P(A and B) = P(A) x P(B|A)
Independence multiplication rule: P(A and B) = P(A) x P(B) (only when A and B are independent)
Conditional probability: P(A|B) = P(A and B) / P(B)
Bayes' rule: P(A|B) = [P(B|A) x P(A)] / [P(B|A) x P(A) + P(B|Ac) x P(Ac)]
Expected value of a discrete random variable: mu = sum of xi x P(xi)
Variance of a discrete random variable: sigma-squared = sum of (xi - mu) squared x P(xi)
Linear transformation rules: E(a + bX) = a + bE(X); Var(a + bX) = b-squared x Var(X)
Sum of independent random variables: E(X + Y) = E(X) + E(Y); Var(X + Y) = Var(X) + Var(Y)
Every government survey and economic indicator depends on sampling design (Chapter 3). Statistics Canada uses stratified, multistage sampling to produce the Labour Force Survey. Clinical trials for new drugs follow the experimental design principles here: randomisation, control groups, and blinding. Probability (Chapter 4) is the engine behind insurance pricing, financial risk models, and quality control in manufacturing. When an economist says "there is a 95% probability that the true unemployment rate lies between 5.1% and 5.7%," the reasoning traces back to these probability rules.
Students often think that a large sample fixes bad design. It does not. A biased sampling method produces biased results regardless of sample size. A voluntary response survey of 100,000 people is still biased.
Students confuse "random" with "haphazard." A random sample uses a precise chance mechanism (like a random number generator), not just grabbing whoever is nearby.
Students assume that independent events are the same as disjoint events. They are not. Disjoint events cannot happen together, so if one occurs, the other definitely did not, which makes them dependent (unless one has probability zero).
Students reverse conditional probabilities without thinking. P(positive test | disease) is not the same as P(disease | positive test). You need Bayes' rule to go from one to the other.
⚠️ Know the difference between an observational study and an experiment. Exam questions often describe a scenario and ask you to identify which it is and whether causal conclusions are justified.
⚠️ Be able to identify sources of bias (undercoverage, nonresponse, response bias, voluntary response) in a described study.
⚠️ Probability calculations are heavily tested. Expect questions combining the addition and multiplication rules, and at least one Bayes' rule problem.
⚠️ Know the difference between independent and disjoint. If two events are disjoint and both have positive probability, they cannot be independent.
⚠️ Expected value and variance of random variables appear in calculation questions. Memorise the rules for linear combinations and sums of independent variables.
True or false: A stratified random sample divides the population into groups and then takes a random sample from each group. Answer: True.
Fill in the blank: An experiment can establish causation because ___ balances out lurking variables across treatment groups. Answer: Randomisation.
True or false: If P(A) = 0.3 and P(B) = 0.4, and A and B are disjoint, then P(A and B) = 0.12. Answer: False. Disjoint means P(A and B) = 0. The value 0.12 would apply if they were independent.
Fill in the blank: P(Ac) = ___. Answer: 1 - P(A).
True or false: Var(X + Y) = Var(X) + Var(Y) always holds for any two random variables. Answer: False. It holds only when X and Y are independent.
Q: A researcher wants to study the effect of a new tutoring programme on student test scores. She lets students volunteer for the programme and then compares their scores to those who did not volunteer. Is this an experiment or an observational study? Can she conclude the programme caused any difference in scores?
A: This is an observational study because students chose whether to participate; there was no random assignment. She cannot conclude causation. Students who volunteer may differ from those who do not (e.g. in motivation), and this confounding makes it impossible to isolate the tutoring effect.
Q: A diagnostic test for a rare disease has a 99% sensitivity (P(positive | disease) = 0.99) and a 2% false positive rate (P(positive | no disease) = 0.02). The disease affects 1 in 1,000 people. If someone tests positive, what is the probability they have the disease?
A: Use Bayes' rule. P(disease | positive) = [0.99 x 0.001] / [0.99 x 0.001 + 0.02 x 0.999] = 0.00099 / (0.00099 + 0.01998) = 0.00099 / 0.02097 ≈ 0.047, or about 4.7%. Even with an excellent test, the low prevalence means most positive results are false positives.
Q: Let X be the number of sales a shop makes in a day with the following distribution: P(X = 0) = 0.1, P(X = 1) = 0.3, P(X = 2) = 0.4, P(X = 3) = 0.2. Find E(X) and Var(X).
A: E(X) = 0(0.1) + 1(0.3) + 2(0.4) + 3(0.2) = 0 + 0.3 + 0.8 + 0.6 = 1.7. Var(X) = (0 - 1.7) squared x 0.1 + (1 - 1.7) squared x 0.3 + (2 - 1.7) squared x 0.4 + (3 - 1.7) squared x 0.2 = 0.289 + 0.147 + 0.036 + 0.338 = 0.81.
Q: Name three sources of bias in survey sampling and give an example of each.
A: Undercoverage (an online survey excludes people without internet access), nonresponse (selected households refuse to participate), and response bias (a question worded in a leading way pushes respondents toward a particular answer).
Chapter 3's discussion of sampling feeds directly into Chapter 5 (Sampling Distributions), where you will see mathematically what happens when you draw repeated samples from a population. The probability rules from Chapter 4 are the foundation for every confidence interval and hypothesis test in Chapters 6-9. Bayes' rule, in particular, will help you think about what a test result means, which connects to the interpretation of p-values later in the course.
Related Terms / Search Tags
sampling, simple random sample, SRS, stratified sample, cluster sample, multistage sample, bias, undercoverage, nonresponse, response bias, voluntary response, observational study, experiment, randomisation, control group, placebo, double-blind, confounding, lurking variable, sample space, event, probability, complement, addition rule, multiplication rule, disjoint, mutually exclusive, independent events, conditional probability, Bayes rule, Bayes theorem, random variable, discrete, continuous, expected value, mean, variance, standard deviation, law of total probability, ECON 2500, introductory statistics for economists