Difficulty: Beginner | Prerequisites: Foundations of Statistics notes (populations, samples, variables)
This topic covers how data gets collected and why collection method matters. A beautifully computed statistic is worthless if the sample behind it is biased. You will also learn the difference between observational studies and experiments, and why only experiments can establish causation. These ideas appear on virtually every STAT101 exam, often as scenario-based questions.
How you collect data determines whether your conclusions are trustworthy. Probability sampling methods (like simple random sampling) reduce bias; non-probability methods (like convenience sampling) invite it. Only well-designed experiments with randomisation can establish that one variable causes changes in another.
Sampling bias
A systematic tendency of the sampling method to favour certain outcomes, producing an unrepresentative sample.
In simple terms, this means: your sample is lopsided in a predictable way, so your conclusions about the population will be off.
Convenience sample
A sample drawn from whoever is easiest to reach (e.g. surveying people on the street outside a shopping centre).
Think of it as: the lazy option. Quick and cheap, but almost always biased.
Voluntary response sample
A sample where participants self-select to respond (e.g. online polls, call-in surveys).
Think of it as: a sample of people with strong opinions, not a sample of the population.
Simple random sample (SRS)
A sampling method in which every individual in the population has an equal chance of being selected.
Think of it as: drawing names from a hat, or using a random number generator on a numbered list.
Stratified random sample
A method where the population is divided into homogeneous subgroups (strata) and a random sample is drawn from each stratum.
Think of it as: making sure each important subgroup (e.g. age bracket, department) is represented, then sampling randomly within each.
Cluster sample
A method where entire groups (clusters) are randomly selected, and every unit in the chosen clusters is studied.
Think of it as: picking whole classrooms at random and surveying every student in those classrooms.
Systematic sample
A method where every kth individual is selected from a list after a random starting point.
Think of it as: picking every 10th name on a roster, starting from a randomly chosen position.
Multi-stage sample
A method that combines several sampling techniques across stages.
Observational study
A study in which the researcher observes variables without controlling or manipulating them.
Think of it as: watching what happens naturally. You can spot associations, but you cannot prove causation.
Experiment
A study in which the researcher actively manipulates one or more explanatory variables and measures the effect on the response variable.
Think of it as: the only study design that can establish cause and effect, because you control who gets what treatment.
Confounding variable (confounder, lurking variable)
A variable that influences both the explanatory and the response variable, potentially creating a false appearance of a direct relationship between them.
Think of it as: the hidden third factor. Ice cream sales and drowning deaths both rise in summer, but temperature is the confounder, not ice cream.
Association
Two variables tend to occur together or are related in some way.
Causation
Changes in one variable directly cause changes in the other.
Randomisation
Randomly assigning experimental units to treatment groups, so that lurking variables are roughly balanced across groups.
Control group
The group in an experiment that receives no treatment (or a placebo), serving as a baseline for comparison.
Replication
Repeating treatments on enough subjects to reduce the role of chance variation in the results.
Blinding
Keeping subjects, researchers, or both unaware of which treatment is being administered, to reduce bias.
In simple terms, this means: if patients know they got the real drug, the placebo effect can skew results. Blinding prevents that.
Matched pairs design
An experimental design where pairs of similar units are matched on key characteristics, then one member of each pair is randomly assigned to each treatment.
Think of it as: comparing left and right hands of the same person, so individual differences cancel out.
Sampling bias occurs when the method of selecting participants systematically favours certain outcomes. A biased sample cannot reliably represent the population, so any generalisations drawn from it are suspect.
Two common sources of biased samples:
Convenience samples pick whoever is easiest to reach. Quick, but the people you can easily access are rarely representative of everyone.
Voluntary response samples let participants opt in. People with strong opinions (positive or negative) are overrepresented; the silent majority is missing.
Neither method is considered scientifically valid for generalisation.
Probability sampling gives each individual a known chance of selection, which is the foundation for reducing bias.
Simple random sample (SRS): assign each unit a unique number, then use a random number generator to select n numbers. Every individual has an equal chance.
Stratified random sample: divide the population into homogeneous strata (e.g. by age group or department), then take a random sample from each stratum. This ensures each subgroup is represented and often produces more precise estimates.
Cluster sample: randomly select entire clusters (e.g. schools, city blocks), then study every unit in the chosen clusters. Useful when a full population list is unavailable.
Systematic sample: choose a random starting point, then select every kth individual. Simple to execute, but watch for hidden patterns in the list.
Multi-stage sample: combine methods across stages (e.g. randomly select districts, then randomly select schools within those districts, then randomly select students within those schools).
Even with a good sampling method, bias can creep in through:
Question wording: leading or confusing questions push respondents toward particular answers.
Context: the environment or timing of data collection (e.g. surveying employees with their manager present).
Inaccurate responses: respondents may misreport or misunderstand questions.
Small changes in phrasing can produce significantly different results.
Association means two variables tend to move together.
Causation means changes in one variable directly produce changes in the other.
Correlation or association does not imply causation. A confounding variable may explain the apparent relationship.
Classic example: ice cream sales and drowning deaths are positively correlated, but ice cream does not cause drowning. Temperature (the confounder) drives both.
In an observational study, the researcher records data without intervening. Associations can be identified, but causation cannot be established confidently because confounders are uncontrolled.
In an experiment, the researcher manipulates the explanatory variable and randomly assigns subjects to treatment groups. Randomisation balances lurking variables across groups, allowing causal conclusions.
Three core principles:
Control - use a control group or placebo to isolate the treatment effect.
Replication - apply the treatment to enough subjects so that chance variation does not dominate the results.
Randomisation - assign treatments randomly to ensure groups are comparable.
Blinding (single or double) further reduces bias by preventing subjects or researchers from letting expectations influence the outcome.
Experimental units (subjects): the individuals assigned to treatments.
Factors: the explanatory variables being manipulated.
Treatments: the specific conditions or combinations of factor levels applied.
Nuisance variables: variables that influence the response but are not the focus of the study.
Response variables: the outcomes measured.
Confounding variables: uncontrolled variables whose effects are mixed with those of the factor.
Pairs of similar units are matched on key characteristics, then randomly assigned to different treatments within each pair. This increases precision by removing variation between pairs.
Example: comparing the grip strength of right and left hands by pairing the same individual's hands.
Clinical trials use randomised, double-blind, placebo-controlled experiments to test whether a new drug works. The randomisation ensures that patient differences (age, health) are balanced across groups; the blinding prevents expectations from distorting the results. Government surveys (like the census or labour-force surveys) use stratified or multi-stage sampling to ensure every demographic group is represented. When you read a news headline claiming "X causes Y," ask whether the evidence came from an experiment or an observational study, because only the former supports a causal claim.
Students often think a large sample automatically eliminates bias. It does not. A biased method (e.g. voluntary response) stays biased no matter how many people respond.
Students confuse stratified and cluster sampling. In stratified sampling, you sample from every stratum. In cluster sampling, you randomly select entire clusters and study all units within them.
Students assume that any correlation between two variables means one causes the other. It does not. Always consider confounding variables.
Students forget that observational studies cannot establish causation, even with a strong correlation. Only a properly randomised experiment can do that.
⚠️ Scenario questions will describe a study and ask you to identify the sampling method. Know the five probability methods cold.
⚠️ "Can we conclude causation?" is a classic exam question. The answer is yes only if the study is a randomised experiment. For observational studies, the answer is no.
⚠️ Expect questions asking you to identify confounding variables in a given scenario.
⚠️ Know the three principles of experimental design (control, replication, randomisation) and what each one accomplishes.
True or false: A voluntary response sample is a type of probability sampling. (False. Participants self-select, so it is non-probability sampling.)
Fill in the blank: In a ______ sample, the population is divided into strata and random samples are drawn from each. (Stratified random.)
True or false: An observational study can prove that smoking causes lung cancer. (False. Observational studies can show association, but only experiments can prove causation.)
Fill in the blank: The three principles of experimental design are control, replication, and ______. (Randomisation.)
True or false: In cluster sampling, you randomly select clusters and then randomly sample individuals within them. (False. You study all individuals within the selected clusters.)
Q: A university wants to survey student satisfaction. It emails a link to all 30,000 students and 1,200 respond. What type of sample is this, and what is the main concern?
A: This is a voluntary response sample. The main concern is bias: students with strong feelings (very satisfied or very dissatisfied) are more likely to respond, so the results may not represent the student body.
Q: A researcher finds that people who drink more coffee tend to have higher stress levels. Can the researcher conclude that coffee causes stress? Explain.
A: No. This is an observational study, so the researcher can identify an association but cannot establish causation. A confounding variable (e.g. workload) may explain both higher coffee consumption and higher stress.
Q: Describe the steps for conducting a stratified random sample of employees at a company with three departments.
A: (1) Divide the employee list into three strata, one per department. (2) Randomly select a specified number of employees from each department. (3) Combine the three sub-samples into one stratified sample.
Q: In a clinical trial, patients are randomly assigned to receive either a new drug or a placebo, and neither the patients nor the doctors know who receives which. Name the three experimental design principles in play and the type of blinding.
A: The three principles are control (placebo group), replication (multiple patients in each group), and randomisation (random assignment to groups). The blinding is double-blind (neither patients nor doctors know the assignments).
Q: A study shows that cities with more firefighters have more fire damage. Does this mean hiring firefighters causes damage? Identify a likely confounder.
A: No. The likely confounder is city size. Larger cities have both more firefighters and more fire damage. The association between firefighters and damage is spurious.
Sampling methods are the bridge between the population-vs-sample framework from the Foundations notes and the inference techniques (confidence intervals, hypothesis testing) that follow. The idea that a statistic varies from sample to sample leads directly into sampling distributions. The association-vs-causation distinction resurfaces whenever you interpret regression output or correlation coefficients later in the course.
Related Terms / Search Tags
sampling bias, convenience sample, voluntary response sample, simple random sample, SRS, stratified random sample, cluster sample, systematic sample, multi-stage sample, probability sampling, non-probability sampling, observational study, experiment, randomised controlled trial, RCT, confounding variable, lurking variable, confounder, association vs causation, correlation does not imply causation, control group, placebo, replication, randomisation, blinding, double-blind, matched pairs design, STAT101, study design, experimental design