Difficulty: Introductory | Prerequisites: Data Types, Parameters and Statistics notes
Good data collection is the foundation of every statistical conclusion. If the sample is biased or the experiment is poorly controlled, no amount of clever analysis will fix the results. This topic covers how samples are drawn from populations and how experiments are structured to isolate the effect of a treatment. You will need these concepts for interpreting study designs in later hypothesis-testing questions, and the exam tests them directly.
The four main sampling methods are simple random, stratified, cluster and systematic. Bias creeps in through flawed sampling, nonresponse or social pressure. Experiments use randomisation, blinding and blocking to isolate treatment effects from confounding variables.
Simple random sampling (SRS)
A sampling method in which every possible group of size n from the population has an equal probability of being selected. Think of it as drawing names from a hat: every combination is equally likely.
Stratified sampling
The population is divided into non-overlapping, homogeneous groups called strata (e.g. by political party, age bracket, income level), and a random sample is drawn from every stratum. In simple terms, you guarantee each subgroup is represented.
Cluster sampling
The population is divided into heterogeneous groups called clusters (e.g. schools, city blocks), a few clusters are randomly selected, and every member of each selected cluster is surveyed. The clusters not selected are not sampled at all. Think of it as picking whole groups at random.
Systematic sampling
A sampling method in which every k-th element is selected from an ordered list after a random start. Example: testing every 50th phone on the production line.
Voluntary response bias
Bias that arises when participants self-select into a survey. People with strong opinions (positive or negative) are overrepresented, making the sample unreliable.
Nonresponse bias
Bias that occurs when a substantial portion of selected individuals do not respond, and those who do not respond differ systematically from those who do.
Social acceptability bias (social desirability bias)
The tendency of respondents to answer questions in a way that will be viewed favourably by others, rather than answering truthfully. This is the bias at play when people underreport drinking or overreport exercise.
Double-blind experiment
An experiment in which neither the subjects nor the investigators know who is receiving the treatment and who is receiving the placebo. This prevents both participant behaviour and researcher measurement from being influenced by knowledge of assignment.
Single-blind experiment
An experiment in which the subjects do not know their assignment, but the investigators do.
Randomised block design
An experimental design in which subjects are first grouped into blocks of similar units (based on a known confounding variable), and then random assignment to treatment and control occurs within each block. In simple terms, you neutralise a known source of variation before randomising.
Confounding variable
A variable that is related to both the treatment and the outcome, making it difficult to determine whether the treatment itself caused the observed effect.
Simple random sampling: every subset of size n is equally likely. Gold standard for unbiased representation, but impractical when the population is very large or has important subgroups you want to ensure are included.
Stratified sampling: divide into homogeneous strata, then sample randomly within each. Ensures every subgroup contributes to the sample. Use when you care about precision across known subgroups (e.g. polling by party affiliation: 145 Democrats, 154 Republicans, 19 Independents).
Cluster sampling: divide into heterogeneous clusters, randomly select a few clusters, survey everyone in them. Cheaper and faster when the population is geographically spread, but less precise because you miss entire clusters.
Systematic sampling: pick every k-th item after a random start. Simple to execute on a production line or ordered list (e.g. every 50th phone). Risk: if the list has a hidden periodic pattern that aligns with k, the sample can be biased.
Sampling bias: the method of selecting participants systematically favours certain outcomes.
Voluntary response bias: only people who feel strongly enough bother to respond, skewing results.
Nonresponse bias: selected individuals who do not participate differ from those who do.
Social acceptability bias: respondents give answers they believe are socially approved rather than truthful.
Randomisation: randomly assign subjects to treatment and control groups to balance out unknown confounders.
Blinding: conceal the assignment from subjects (single-blind) or from both subjects and investigators (double-blind) to prevent bias in behaviour and measurement.
Blocking: group subjects by a known confounding variable before randomising within each block. This isolates the treatment effect from the confounding factor.
Suppose a pharmaceutical company is testing a blood pressure drug on 100 participants and suspects age matters. The steps are:
Identify the blocking variable: age (under 40 vs 40 and older).
Divide participants into two blocks based on that variable.
Within each block, randomly assign half to the treatment and half to the control.
Compare outcomes within each block, then combine.
This is better than simple randomisation because it prevents the possibility of accidentally loading one group with older (or younger) participants, which would confound the treatment effect with the age effect.
Students often confuse stratified and cluster sampling. The key difference: in stratified sampling you sample from every group; in cluster sampling you sample entire groups and skip the rest.
Students assume strata and clusters have the same internal composition. Strata are homogeneous within (members are similar to each other); clusters are heterogeneous within (each cluster mirrors the population).
Students sometimes think "double-blind" means two treatments. It means two parties (subjects and investigators) are both kept unaware of who receives what.
Students forget that voluntary response samples are unreliable. The self-selection mechanism introduces bias regardless of sample size.
The exam presents real-world scenarios and asks you to identify the sampling method. Look for the distinguishing feature: every k-th item = systematic, groups by characteristic then sample each = stratified, randomly pick whole groups = cluster.
Bias-type questions describe a survey situation and ask which bias applies. Social acceptability bias is tested when respondents might be embarrassed by the truth.
Experimental design appears in both multiple-choice and long-answer questions. The long-answer version typically asks you to describe how to set up a randomised block design for a given scenario and explain why it is superior to simple randomisation.
True or false: voluntary response samples are generally highly reliable. (False)
True or false: in a cluster sample, every member of selected groups is surveyed. (True)
True or false: randomised blocking reduces the effect of known confounding variables. (True)
Fill in the blank: testing every 50th item on a production line is ____ sampling. (Systematic)
True or false: in a simple random sample, every possible group of size n has the same probability of being selected. (True)
Q: A pollster randomly samples 145 Democrats, 154 Republicans and 19 Independents. What sampling method is this?
A: Stratified sampling. The population is divided into strata by political affiliation, and a random sample is drawn from each stratum.
Q: Which type of bias occurs when people are reluctant to admit to behaviour that may reflect negatively on them?
A: Social acceptability bias (also called social desirability bias).
Q: In an experiment, neither the investigators nor the subjects know who receives the treatment. What is this design called?
A: A double-blind experiment.
Q: A pharmaceutical company suspects that age (under 40 vs 40+) may influence the results of a drug trial with 100 participants. Describe how to implement a randomised block design.
A: Divide the 100 participants into two blocks by age. Within each block, randomly assign half to the treatment group and half to the control group. This ensures that age-related variation is balanced across treatment and control, preventing age from confounding the results. Simple randomisation could, by chance, put most older participants in one group.
Q: An electronics manufacturer tests every 50th phone off the line. What sampling method is this?
A: Systematic sampling.
Sampling methods connect directly to the parameter-vs-statistic distinction: a statistic is only as good as the sample that produced it. These concepts also feed into confidence intervals and hypothesis testing, where the assumption of a random sample underlies the validity of every inference. Blocking connects to ANOVA and factorial designs in later coursework.
Related Terms / Search Tags
simple random sample, SRS, stratified sampling, cluster sampling, systematic sampling, voluntary response bias, nonresponse bias, social desirability bias, sampling bias, double-blind, single-blind, randomised block design, blocking variable, confounding variable, experimental design, placebo, control group, NYU analytical statistics midterm