Source: Principles of Statistics I, Texas A&M / Tamhane & Dunlop Ch. 3
Tags: sampling, sample vs census, sampling error, non-sampling error, bias, measurement bias, self-selection bias, response bias, non-response, probability sampling, simple random sample, SRS, systematic sampling, stratified sampling, cluster sampling, sampling frame, convenience sample, judgmental sample
Choosing how to sample from a population is one of the most consequential decisions in any study. Probability sampling methods (SRS, systematic, stratified, cluster) allow objective assessment of accuracy, while non-probability methods (convenience, judgmental) do not. Beyond sampling error, researchers must watch for non-sampling errors such as measurement bias, self-selection, and non-response, which persist regardless of sample size.
Sample
A small portion of the population selected for inspection or study.
Census
An inspection or survey of the entire population.
Sampling frame
A complete list of all N units in the population. There is a one-to-one correspondence between the numbers 1, 2, ..., N and the units in the frame.
Sampling error
The difference between the information contained in a sample and the information contained in the full population. This error exists because only a fraction of the population is observed. It can theoretically be eliminated by increasing sample size to cover the whole population.
Non-sampling error (bias, systematic error)
Consistent deviations of sample estimates from the true population values that persist even if the sample size is increased, or even in a full census. These are the truly problematic errors.
Measurement bias
A measuring device or survey instrument that consistently records values either too high or too low. Poorly worded questions or leading interviewer body language can cause this.
Self-selection bias (voluntary response bias)
Occurs when the people who choose to participate differ systematically from those who do not. Younger people, higher-income and lower-income individuals tend to participate at lower rates; politically active people participate at higher rates.
Response bias
Untruthful answers caused by very personal questions or questions requiring recall of distant events. ("Did you inhale?" "Have you ever cheated on your income tax?")
Non-response
When selected individuals do not respond at all, whether by refusing the phone call, not returning a survey, or (in agriculture/wildlife studies) missing data from external causes.
Probability sampling
A sampling method where every possible sample has a known probability of being selected, and selection uses a random process. This allows objective assessment of the accuracy of inferences.
Simple random sample (SRS)
The most basic probability sampling method. A sample of n units is drawn from a population of N such that every possible combination of n units (there are "N choose n" of them) has an equal chance of being selected.
Systematic random sampling (1-in-k sampling)
Select one unit at random from the first k units in a list, then select every kth unit thereafter. Each unit has selection probability n/N, but unlike SRS, not all possible samples of size n are equally likely.
Stratified random sampling
Divide the population into L non-overlapping groups (strata), then take independent simple random samples from each stratum. Useful when strata differ from each other but are internally homogeneous.
Cluster random sampling
Divide the population into clusters (groups of elements). Randomly select some clusters, then survey all or some elements within the chosen clusters. Useful when a complete list of individual elements is unavailable but a list of clusters is.
Convenience sample
A non-probability sample based on whatever data is readily available. Examples: the next 50 people who walk into a store, using a class of undergraduates for a research project.
Judgmental sample
A non-probability sample where an expert selects members considered "typical" or "representative." Extremely subjective and does not permit scientific assessment of accuracy.
Reduced cost and less time required.
More information per subject, because less effort is spread across fewer units.
Potentially greater accuracy, since technicians can be better trained and measurements more carefully taken. Subjects may even be missed in a census.
A census may be impossible when the population is mobile.
Measurement may require destroying the units being inspected (e.g. testing light bulb lifespan).
When designing or reading a study, always ask:
What is the population of interest to the researcher?
What is the method by which the sample was selected?
If the sample is not properly drawn from the population of interest, the conclusions may not generalise. For example, if the population of interest is "all diesel VW cars less than 10 years old" but the sample is "all VW cars on used car lots in Houston," the sample may not represent the population.
Sampling error can always be reduced by increasing sample size. In theory, it vanishes with a census.
Non-sampling errors (biases) persist even in a census. These are the dangerous ones. Six major types follow.
Measurement bias: an instrument or questionnaire that consistently pushes responses in one direction. Improperly worded questions, unclear instructions, or interviewer body language can all contribute.
Self-selection bias: the people who participate are systematically different from those who do not. Younger people and those at income extremes participate less; people with strong opinions participate more. End-of-semester course evaluations often reflect this: students who loved or hated the course respond, while indifferent students do not.
Selection method bias: random digit dialling is compromised because many people screen calls with caller ID or answering machines and do not pick up for unknown numbers.
Response bias: personal or sensitive questions invite untruthful answers. Recall-dependent questions invite inaccurate answers.
Timing of poll: a political poll conducted too far from an election may not reflect late-breaking information that changes voters' minds.
Non-response: when selected units simply do not respond. In agriculture, a predator raiding a bird's nest or a deer herd eating a corn field create missing data. Strategies for dealing with non-response include designing surveys to minimise it (follow-up calls, payment incentives), randomly sampling non-respondents to make inferences about the rest, using statistical models to predict missing responses, or ignoring it entirely, which is common in practice but very bad.
The 2016 U.S. presidential election polls illustrate several sampling problems at once:
Cell phone adoption made traditional landline-based random sampling increasingly unreliable.
Online polling methods introduced self-selection bias and were considered lower quality.
State-level polls had smaller sample sizes and used less rigorous methods than national polls, reducing statistical power.
"Likely voter" models based on past election turnout patterns failed to capture shifts in who actually voted.
Possible "shy Trump" voters introduced response bias; some supporters may not have admitted their preference to a live interviewer.
Late-deciding voters broke disproportionately for one candidate, and polls conducted too early missed this movement.
Aggregation models amplified shared biases across polls rather than smoothing them out.
These illustrate measurement bias, self-selection bias, response bias, timing effects, and non-response all operating simultaneously.
Simple Random Sampling (SRS)
Every possible sample of size n from the population of N has the same probability of being selected.
Selection is done using a random number generator. In R: sample(N, n) selects n integers from 1 to N without replacement.
R commands for random sampling:
sample(n) – random permutation of integers 1 to n
sample(x) – random permutation of elements in vector x
sample(x, n) – random sample of n items from x, without replacement
sample(x, n, replace=T) – random sample with replacement
sample(x, n, replace=T, p) – sampling with unequal probabilities
Systematic Random Sampling
From a sequential list, randomly select a starting point from the first k units, then take every kth unit.
k = N/n, so the sample size is n.
Each unit has selection probability n/N, same as SRS. However, not all possible samples of size n are equally likely, because only k distinct samples can be drawn.
Example: from N = 1,000 units, to sample n = 10, set k = 100. Randomly select a number between 1 and 100 (say 23). The sample is units 23, 123, 223, 323, ..., 923.
Works well when the list has no cyclic patterns. Breaks down when the population has a periodic structure that aligns with the sampling interval, because every sampled unit could come from the same phase of the cycle.
Stratified Random Sampling
Divide the population into L non-overlapping strata with N₁, N₂, ..., N_L units (summing to N).
Draw independent SRSs of sizes n₁, n₂, ..., n_L from the respective strata.
Reasons to stratify: precise estimates within each stratum; administrative convenience; different parts of the population may require different sampling approaches; possible gain in overall precision when there are large differences between strata but homogeneity within them.
Example: to estimate the percentage of Texans with health insurance, stratify counties into rural, small towns, medium cities, and large metropolitan areas, then randomly sample from each.
Cluster Random Sampling
The population consists of N clusters (primary sampling units, PSUs), each containing M₁, M₂, ..., M_N elements (secondary sampling units, SSUs).
In a single-stage cluster sample, randomly select n clusters and survey every element within each selected cluster.
In a multi-stage cluster sample, randomly select n clusters, then randomly sample some elements within each selected cluster.
Example: to study bicycle ownership in a community of 10,000 households, divide the community into 500 blocks of ~20 households each. Randomly select 15 blocks and survey every household in those blocks.
Cluster sampling is generally less precise than SRS (elements within a cluster tend to be similar), but it is much easier administratively when a list of individual elements is unavailable. Getting a list of city blocks in Houston is far simpler than getting a list of every household.
Stratified sampling tends to produce smaller variance in estimates, guarantees representation from every stratum, allows estimation of means for each stratum, and requires a sampling frame for each stratum.
Cluster sampling is useful when a frame exists for clusters but not for individual elements, when elements need to be interviewed or measured in person, and when population elements are widely separated or naturally grouped (households, schools).
Real studies often combine methods:
Multistage cluster sample: the EPA selects 10 states at random, then samples lakes within those states, then measures discharge points within those lakes. States are clusters of lakes; lakes contain discharge points.
Stratified multistage cluster sample: a researcher divides northern California into 20 regions (strata), randomly selects 10 oak trees per region (clusters), then samples 8 seedlings per tree (elements). Regions are strata; oak trees are clusters of seedlings.
⚠️ Distinguish sampling error (reducible by increasing n) from non-sampling error (persists even in a census). Exams love this distinction.
⚠️ Know all six types of non-sampling error by name with examples: measurement bias, self-selection bias, selection method bias, response bias, timing, non-response.
⚠️ For probability sampling, know the four methods (SRS, systematic, stratified, cluster) and when each is appropriate.
⚠️ Systematic sampling fails when the population has a cyclic pattern that matches the sampling interval.
⚠️ Stratified sampling beats SRS in precision when strata are internally homogeneous but differ from each other.
⚠️ Cluster sampling is less precise than SRS but is used when listing all individual elements is impractical.
⚠️ Be ready to identify combined designs (stratified + cluster, multistage cluster) from scenario descriptions.
⚠️ Convenience samples and judgmental samples are non-probability methods and do not allow scientific assessment of accuracy.
Q: A researcher surveys every 50th car coming off an assembly line. What sampling method is this?
A: Systematic random sampling (1-in-50 systematic sample).
Q: Why might a systematic sample give distorted results on a production line?
A: If the production process has a cyclic pattern whose period aligns with the sampling interval, every sampled unit could come from the same phase of the cycle, giving a non-representative sample.
Q: A polling firm contacts households by randomly dialling landline numbers. What type of non-sampling error is most likely?
A: Selection method bias (and self-selection bias). People who only use mobile phones are excluded entirely, and those who screen calls with caller ID will not be reached.
Q: What is the key advantage of probability sampling over non-probability sampling?
A: Probability sampling allows an objective, quantifiable assessment of the accuracy of inferences made about the population.
Q: A study divides a state into urban, suburban, and rural zones, then draws a random sample from each zone. What sampling method is this?
A: Stratified random sampling. The zones are the strata.
Q: An education researcher randomly selects 20 schools from a district and tests every student in those schools. What sampling method is this?
A: Single-stage cluster sampling. Schools are clusters; students are elements.
Q: A non-response rate of 40% is observed in a mail survey. Why is ignoring the non-response a bad idea?
A: Non-respondents may differ systematically from respondents (e.g. they may have no strong opinion, or may be demographically different). Ignoring them introduces bias that does not decrease with larger samples.
sampling, census, sample vs census, sampling frame, sampling error, non-sampling error, bias, measurement bias, self-selection bias, voluntary response bias, response bias, non-response, missing data, probability sampling, non-probability sampling, convenience sample, judgmental sample, simple random sample, SRS, systematic sampling, 1-in-k sample, stratified random sampling, strata, cluster random sampling, PSU, SSU, primary sampling unit, secondary sampling unit, single-stage cluster, multi-stage cluster, multistage sampling, R sample function, polling errors, 2016 polls, statistical power, survey design