Difficulty: Beginner | Prerequisites: Section 1.1 (Overview of Statistics)
Knowing what statistics is and how to classify data is only useful if you also know how to collect data properly. A brilliant analysis of badly collected data still gives you rubbish conclusions. This section covers the steps for designing a statistical study, the difference between observational and experimental studies, the main data-collection methods (simulation, survey), and the five sampling techniques you need to recognise and choose between. This material is tested heavily because poor study design is the most common source of bad statistics in the real world.
Statistical studies are either observational (you watch and record) or experimental (you apply a treatment and measure the response). Data can be gathered via simulation, survey, or direct measurement. When sampling, the five key methods are simple random, stratified, cluster, systematic, and convenience. Each has trade-offs in cost, effort, and the quality of conclusions you can draw.
Observational study
A study in which a researcher observes and measures characteristics of interest without influencing or manipulating the subjects. Think of it as watching from the sidelines. You record what happens but do not change anything.
Experimental study (experiment)
A study in which a researcher deliberately applies a treatment to subjects before observing the responses. In simple terms, you change something on purpose to see what effect it has.
Treatment group
The group of subjects in an experiment that receives the treatment being studied.
Control group
The group of subjects in an experiment that does not receive the treatment, serving as a baseline for comparison.
Experimental units
The subjects in both the treatment and control groups.
Census
A count or measure of an entire population. In simple terms, you survey everyone, not just a subset.
Sampling
A count or measure of part of a population. More commonly used than a census in statistical studies because it is cheaper and faster.
Simple random sample
A sampling method in which every possible member of the population has an equal chance of being selected. Think of it as drawing names from a hat: nobody has a better or worse shot at being picked.
Stratified sample
A sampling method in which the population is divided into subsets (strata) by a shared characteristic, and a random sample is then drawn from each stratum in proportion to its size. In simple terms, you split the group into categories first (e.g. by year of study), then sample randomly within each category.
Cluster sample
A sampling method in which the population is divided into groups (clusters), and then all members of one or more randomly selected clusters are surveyed. Think of it as picking entire groups at random rather than individuals. Example: randomly select 10 hospitals and interview every nurse in those hospitals.
Systematic sample
A sampling method in which members of the population are selected at regular intervals from a randomly determined starting point. In simple terms, you pick every kth item. Example: check every other egg carton for broken eggs.
Convenience sample
A sampling method in which members are selected because they are easy to reach. This method is the weakest because it often produces biased, non-representative data.
Simulation
A data-collection method that uses a mathematical or physical model to reproduce the conditions of a situation or process, often with computers. Useful when real experiments are impractical or dangerous.
Survey
An investigation of one or more characteristics of a population, carried out by asking people questions via interview, internet, phone, or mail.
A well-designed study follows these steps:
Identify the variable(s) of interest and the population of the study.
Develop a detailed plan for collecting data. If using a sample, ensure it is representative of the population.
Collect the data.
Describe the data using descriptive statistics.
Interpret the data and make decisions about the population using inferential statistics.
Identify any possible errors.
Observational: the researcher does not attempt to influence responses. Example: calling 3,040 US adults and asking them to rate economic conditions. The researcher records opinions but does not change them.
Experimental: the researcher applies a treatment. Example: giving 70 patients vitamin D3 daily and 70 patients a placebo for one year, then comparing respiratory infection rates. The researcher deliberately intervened.
The critical difference: can you establish cause and effect? Experiments can (if well designed); observational studies generally cannot, they can only show associations.
A study took a random sample of 1,500 adults in Orlando and asked about bedtime habits. The data showed that people who drank tea before bedtime went to sleep earlier.
This is an observational study: the researchers asked questions and recorded answers; they did not assign anyone to drink tea.
Population: all adults in Orlando.
Sample: the 1,500 adults surveyed.
A possible inference: adults in Orlando who drink tea before bed may tend to go to sleep earlier. (Note: you cannot conclude tea causes earlier sleep from an observational design.)
Simulation: car manufacturers use crash-test dummies to simulate accident impacts rather than testing on real people. Saves time, money, and avoids ethical issues.
Survey: asking female physicians whether financial stability was the primary reason for their career choice. Surveys must be carefully worded to avoid biased results.
Simple random sample
Every member has an equal chance.
Example: a polling agency randomly selects 300 registered voters in a small city and asks if they favour a particular candidate. Every registered voter had the same chance of selection.
Stratified sample
Divide the population into strata, then sample from each.
Example: 100 freshmen, 85 sophomores, 80 juniors, 75 seniors, and 50 graduate students are chosen at random from their respective classifications and asked about drinking habits. Each classification is a stratum.
Cluster sample
Divide the population into clusters, then randomly select entire clusters.
Example: to study nursing conditions in Florida, the Governor randomly selects 10 hospitals and interviews every nurse at each one. Each hospital is a cluster.
Systematic sample
Select every kth member from a starting point.
Example: a grocery manager checks every other egg carton for broken eggs. The interval is 2.
Convenience sample
Select whoever is easiest to reach.
This is the least reliable method and is prone to bias because the sample is unlikely to represent the population.
Clinical drug trials are the classic experimental study: one group receives the drug (treatment), one receives a placebo (control), and researchers compare outcomes. Political polling typically uses stratified or cluster sampling to ensure different demographics are represented. Quality-control inspections in factories almost always use systematic sampling (check every nth item off the production line).
Students often think observational studies are "less valid" than experiments. They serve different purposes. Many important research questions (e.g. the health effects of smoking) were first established through observational studies because randomly assigning people to smoke would be unethical.
Students confuse stratified and cluster sampling. In stratified, you sample some members from every subgroup. In cluster, you sample all members from some subgroups. The direction of "some" and "all" flips.
Students sometimes forget that convenience sampling is a real method with a name, and they may not recognise it as the weakest option on an exam. If the question describes selecting people because they were easy to reach, it is convenience sampling.
Calling a study an "experiment" when no treatment was applied is a common exam error. If the researchers only asked questions or observed behaviour, it is observational, regardless of how complex the study was.
⚠️ "Is this an observational study or an experiment?" is one of the most frequently tested questions in Chapter 1. Look for whether a treatment was applied. No treatment = observational.
⚠️ You will be asked to identify which sampling method was used. The tell-tale signs: equal chance for everyone (simple random), divided into subgroups first (stratified or cluster), regular interval (systematic), easy access (convenience).
⚠️ Know the distinction between stratified and cluster sampling cold. Stratified = sample from each subgroup. Cluster = select entire subgroups.
⚠️ Be prepared to identify population, sample, and possible inferences from a described study scenario.
True or False: An observational study applies a treatment to subjects.
Fill in the blank: In a ______ sample, the population is divided into strata and members are randomly selected from each one.
True or False: Convenience sampling is the most reliable sampling method.
Fill in the blank: The group in an experiment that does not receive the treatment is called the ______ group.
True or False: A systematic sample selects members at regular intervals.
Answers: 1. False (that describes an experiment). 2. Stratified. 3. False (it is the least reliable). 4. Control. 5. True.
Q: Researchers study the effect of vitamin D3 supplementation among patients with antibody deficiency. 70 patients receive 4000 IU of vitamin D3 daily for a year. Another group of 70 patients receives a placebo daily for one year. Is this an observational study or an experiment?
A: It is an experiment. The researchers applied a treatment (vitamin D3) to one group and a placebo to the other, then observed the results.
Q: Researchers call 3,040 US adults and ask them to rate current economic conditions and whether the economy is getting better or worse. Is this an observational study or an experiment?
A: It is an observational study. The researchers did not attempt to influence the respondents' views; they only asked questions and recorded answers.
Q: 100 freshmen, 85 sophomores, 80 juniors, 75 seniors, and 50 graduate students are chosen at random from their classification and asked about drinking habits. What sampling method is this?
A: Stratified sampling. The population was divided into strata (year of study), and random samples were drawn from each stratum.
Q: A grocery manager selects every other egg carton to check for broken eggs. What sampling method is this?
A: Systematic sampling. Members are selected at a regular interval (every 2nd carton).
Q: The Governor randomly selects 10 Florida hospitals and interviews every nurse in each hospital. What sampling method is this?
A: Cluster sampling. The population was divided into clusters (hospitals), and entire clusters were randomly selected for inclusion.
Study design and sampling methods connect to the idea of bias and error, which you will encounter when studying confidence intervals and margin of error in later chapters. Understanding the difference between observational and experimental studies is also essential for Chapter 7+ topics on hypothesis testing, where the type of study determines whether you can conclude causation or only association.
observational study, experimental study, treatment group, control group, experimental units, census vs sampling, simple random sample, stratified sample, cluster sample, systematic sample, convenience sample, simulation statistics, survey design, sampling techniques statistics, data collection methods, principles of statistics chapter 1 section 1.3, how to design a statistical study