Difficulty: Introductory | Prerequisites: Understand sampling methods (see Sampling Methods study notes).
Sampling tells you how to select people or things to study. Experimental design tells you how to assign treatments and measure outcomes so you can draw cause-and-effect conclusions. Without proper design, you cannot tell whether a difference in outcomes is caused by the treatment or by something else entirely.
This section covers the vocabulary of experiments (treatments, outcomes, subjects, confounders) and the techniques that make experiments trustworthy (randomisation, blinding, blocking). These concepts appear in almost every introductory statistics exam.
You should be comfortable with sampling methods before reading this. If a question asks about how subjects were selected for an experiment, that is sampling. If it asks about how treatments were assigned or outcomes measured, that is experimental design.
A well-designed experiment randomly assigns treatments to subjects, controls for confounders, and measures outcomes. Randomisation lets you conclude that differences are caused by the treatment. Double-blinding removes bias from both subjects and investigators. Blocking accounts for known sources of variation before randomisation.
Randomised experiment
An experiment in which treatments are assigned randomly to experimental units.
Think of it as: flipping a coin to decide who gets the drug and who gets the placebo. The random assignment is what separates this from an observational study.
Double-blind experiment
An experiment in which neither the investigators nor the subjects know how the treatments have been assigned.
In simple terms, nobody involved in running or receiving the experiment knows who is in which group until the data are collected. This prevents both the placebo effect and investigator bias.
Treatment
The specific condition or intervention applied to an experimental unit.
Think of it as: the thing being tested. A new drug, a fertiliser, a teaching method.
Outcome
What is measured on each experimental unit after the treatment is applied.
In simple terms, the result you are looking for. Did the patient improve? Did the plant grow taller?
Subject
The term used for experimental units when they are people.
Think of it as: "participant" in everyday language.
Confounder (confounding variable)
A variable that is associated with both the treatment and the outcome, making it difficult to determine whether the treatment itself caused the observed effect.
In simple terms, a confounder muddies the water. If treatment group members also happen to be younger, you cannot tell whether the outcome is due to the treatment or to age.
Bias (in experiments)
A systematic tendency to overestimate or underestimate the true value.
Think of it as: the experiment is consistently off in one direction, not just randomly wrong.
Randomised blocking
Subjects are first sorted into blocks (groups that share a characteristic, such as sex or age bracket), and then treatments are randomly assigned within each block.
In simple terms, you account for a known source of variation before you randomise. Sorting subjects by sex and then randomly assigning treatments within each sex group is randomised blocking.
In a randomised experiment, treatments are assigned randomly to experimental units. This is the gold standard for establishing causation.
If there are large differences in outcomes among treatment groups in a randomised experiment, we can conclude the differences are due to the treatments. Random assignment ensures that confounders are, on average, evenly distributed across groups.
Without randomisation, observed differences might be caused by pre-existing differences between groups rather than the treatment.
In a double-blind experiment, neither the investigators nor the subjects know which treatment each subject received.
This prevents two sources of bias: the placebo effect (subjects behave differently when they know they are getting the real treatment) and investigator bias (researchers unconsciously treat groups differently or interpret results differently).
Common exam trap: "double-blind" does not mean subjects are literally blindfolded or that investigators and subjects are hidden from each other's views. It means neither party knows the treatment assignment.
A confounder is a variable associated with both the treatment and the outcome. It makes it difficult to determine whether the observed effect is due to the treatment.
Example: if you are testing a new teaching method but the treatment group also has smaller class sizes, class size is a confounder.
Randomisation helps reduce confounding by distributing confounders roughly equally across treatment groups.
Randomised blocking first divides subjects into groups (blocks) based on a characteristic known to affect the outcome, then randomly assigns treatments within each block.
Example: subjects are categorised by sex, and then each subject is randomly assigned a treatment within their sex group. This ensures that differences in outcomes are not driven by differences in the proportion of males and females across treatment groups.
Blocking is not the same as stratified sampling. Blocking is about treatment assignment within an experiment. Stratified sampling is about selecting people from a population.
In an observational study, the researcher observes subjects without assigning treatments. No random assignment means you cannot draw cause-and-effect conclusions as confidently.
In an experiment, the researcher actively assigns treatments. With randomisation, cause-and-effect conclusions are supported.
An experiment that tends to systematically overestimate or underestimate the true value is said to be biased.
Bias can come from poor randomisation, lack of blinding, confounders, or flawed measurement.
Students think "double-blind" means subjects are physically blindfolded or hidden from view. It does not. It means neither the subjects nor the investigators know the treatment assignment.
Students confuse a randomised experiment (treatments assigned randomly) with random sampling (subjects chosen randomly from a population). Both use randomness, but for different purposes.
Students assume that any experiment with random elements is a randomised experiment. The defining feature is random assignment of treatments, not random selection of participants or random outcomes.
Students mix up blocking and stratified sampling. Blocking is about how treatments are assigned within an experiment. Stratified sampling is about how subjects are selected from a population.
⚠️ The definition of a randomised experiment is a favourite exam question. The correct answer is: treatments are assigned randomly to experimental units. Not "investigators chosen at random" or "outcomes are random."
⚠️ Double-blind: expect the exam to offer distractors about physical blindfolding or hiding identities. The correct definition is that neither investigators nor subjects know the treatment assignments.
⚠️ If a randomised experiment shows large differences between groups, the conclusion is that the differences are due to the treatments. This is a direct consequence of randomisation.
⚠️ When experimental units are people, they are called subjects. This is a vocabulary question that appears frequently.
⚠️ A confounder makes it difficult to determine whether an outcome is due to the treatment. The exam will test whether you can identify the confounder in a scenario.
True or false: In a double-blind experiment, the subjects are blindfolded. (False. Neither subjects nor investigators know the treatment assignments.)
Fill in the blank: In a randomised experiment, ______ are assigned randomly to experimental units. (Treatments.)
True or false: A confounder makes it easier to determine the effect of the treatment. (False. A confounder makes it harder.)
Fill in the blank: When experimental units are people, they are called ______. (Subjects.)
True or false: In randomised blocking, subjects are sorted into groups based on a known characteristic before treatments are assigned. (True.)
Q: Which of the following best describes a double-blind experiment? (a) Subjects are blindfolded, (b) Neither investigators nor subjects know treatment assignments, (c) Investigators are chosen at random, (d) Neither party knows the other's name.
A: (b). Double-blind means neither investigators nor subjects know how treatments were assigned.
Q: Which best describes a randomised experiment? (a) Treatments assigned randomly to units, (b) Investigators chosen at random, (c) Units selected at random, (d) Outcomes are random.
A: (a). The defining feature is random assignment of treatments to experimental units.
Q: In a randomised experiment with large differences between treatment groups, the differences are due to what?
A: The treatments. Randomisation ensures that other variables are balanced across groups on average, so large differences point to the treatment as the cause.
Q: Subjects are categorised by sex and then randomly assigned a treatment within each category. What is this called?
A: Randomised blocking. Subjects are first sorted into blocks (here, by sex) and then randomised within each block.
Q: What makes it difficult to determine whether an experimental outcome is due to the applied treatment?
A: A confounder. It is a variable linked to both the treatment and the outcome, clouding the causal picture.
Q: An experiment that tends to overestimate or underestimate the true value is said to be what?
A: Biased.
This connects to hypothesis testing later in the course. The logic of randomised experiments is the foundation for p-values and significance tests: if the treatment has no effect, how likely is the observed difference? That question only makes sense if the experiment was properly randomised.
This also connects to sampling methods. A well-designed experiment still needs a properly selected sample if you want the results to generalise beyond the study participants. Randomised assignment within the experiment handles internal validity; good sampling handles external validity.
Randomised experiment, randomized experiment, double-blind, single-blind, treatment, outcome, response variable, subject, participant, experimental unit, confounder, confounding variable, lurking variable, bias in experiments, randomised blocking, blocking, observational study vs experiment, causation, Navidi Chapter 1, elementary statistics experimental design