Difficulty: Introductory | Prerequisites: Part 1 of these study notes (Association, Causation, and Confounding). You should understand what confounding variables are and the difference between observational studies and experiments before reading this.
Once you know that only experiments (not observational studies) can establish causation, the next question is: what makes a well-designed experiment? This part covers the three principles every good experiment follows (control, randomisation, replication), the role of placebos and blinding, and the two main types of randomised experiments you need to know for the exam. These concepts recur throughout the course whenever you evaluate a study's conclusions.
A well-designed experiment uses three principles: control (compare treatments), randomisation (assign units by chance to neutralise lurking variables), and replication (use enough subjects to reduce chance variation). Placebos and double-blinding prevent the placebo effect and interpretation bias from contaminating results. Completely randomised experiments assign all units at random; matched pairs experiments pair similar units (or use the same unit twice) to reduce noise.
See Part 1 for full definitions. The terms most relevant to this part:
Control (Principle 1)
Comparing two or more treatments to isolate the effect of interest. The comparison can be against another treatment, no treatment, a placebo, or a combination.
Randomisation (Principle 2)
Allocating experimental units to treatments by chance. This makes treatment groups probabilistically alike on all lurking variables, avoiding systematic bias.
Replication (Principle 3)
Applying each treatment to enough subjects so that results are not driven by chance variation or errors in a single unit.
Placebo
A fake treatment designed to look like the real one. Only useful if participants do not know whether they are receiving the active treatment or the placebo.
Double-blind
Neither the subjects nor the experimenter knows who received which treatment until the study is complete.
Completely randomised experiment
All units are assigned at random to all treatments. Straightforward, but variation among units can mask the treatment effect.
Matched pairs experiment
Pairs of similar subjects are formed, and within each pair, treatments are randomly assigned. Alternatively, the same subject receives both treatments in random order. Always compares exactly two treatments. Can produce more precise results than a completely randomised design.
Every well-designed experiment follows these three principles:
Principle 1: Control (Comparison)
Experiments are comparative by nature. You always compare the response to a treatment against something else: another treatment, no treatment at all (a control), a placebo, or some combination. Without comparison, you cannot tell whether the treatment did anything. The gastric freezing example from the lecture illustrates this perfectly: the initial study had no control group, so the observed improvement could not be separated from the placebo effect.
Principle 2: Randomisation
The best way to remove bias is to assign units to treatments by chance. Randomisation makes the treatment groups probabilistically alike on all lurking variables, including ones you have not thought of. In the fertiliser example, assigning the 12 easternmost plots to Fertiliser A and the rest to B would confound the treatment with soil quality. Random assignment neutralises that problem.
How to randomise in practice:
Label each unit with a number.
Use a random number generator (or a table of random digits) to select which units go into each group.
The selected units form one treatment group; the remaining units form the other.
Principle 3: Replication
Apply each treatment to enough subjects to reduce the role of chance. With very small groups, random variation alone can produce differences that look meaningful. Replication ensures that results are not due to uncontrolled factors or manipulation errors.
The gastric freezing case study (Example 1.7):
This is the lecture's primary illustration of why control matters.
Initial design: patients received the gastric freezing treatment and many improved. No control group was used. The study concluded the treatment worked, and it was widely adopted for several years.
Problem: patients may have improved because of the placebo effect, not the freezing. The treatment effect was confounded with the placebo effect.
Improved design: a later experiment divided patients into two groups. One received the actual treatment; the other received an identical procedure with body-temperature liquid instead of cold (the placebo). Results: 34% improved in the treatment group, 38% in the placebo group. Gastric freezing was no better than a placebo.
Why placebos work:
A placebo is only useful when participants cannot tell whether they are receiving the real or the sham treatment. Many patients respond to any treatment because of trust in the doctor, expectation of a cure, or simply because many conditions improve on their own over time.
Double-blinding:
The strongest experimental design keeps both the subjects and the researchers in the dark about who received which treatment. This prevents subjects from behaving differently based on what they think they received (placebo effect) and prevents researchers from interpreting outcomes differently based on what they know about the subject's assignment (interpretation bias).
Completely randomised (randomised comparative) experiment:
All experimental units are allocated at random among all treatments. This is the simplest design.
Example: 30 rats are divided at random into two groups of 15. Group 1 receives a new diet; Group 2 receives a standard diet. Weight gain is compared after 28 days.
Strength: randomisation makes the groups similar before treatment.
Limitation: if there is a lot of natural variation among the units (e.g. location matters in a field experiment), that variation can obscure the treatment effect.
Matched pairs experiment:
Compares exactly two treatments. There are two versions:
Version 1: pair subjects who are closely matched on relevant characteristics (same gender, age, weight, etc.). Within each pair, randomly assign one to Treatment A and the other to Treatment B.
Version 2: use each subject as their own pair. Give both treatments to the same subject at different times, in random order.
Key points about matched pairs:
Always compares exactly two treatments.
The order of treatments must be randomised (e.g. by coin toss) to prevent order effects.
Can produce more precise results than a completely randomised design because individual differences are controlled.
The beetle board example (Example 1.9):
Researchers want to compare whether yellow or green sticky boards attract more cereal leaf beetles.
Completely randomised design: randomly assign half the poles in a field to yellow boards and half to green. Problem: locations vary in beetle density (near alfalfa = more beetles), and this variation can mask the colour effect.
Matched pairs design: mount both colours on every pole (the "pair" is the same location). Randomly choose which colour goes on top. Compare beetle counts on yellow vs green at each pole. This is more efficient because location-to-location variation is eliminated.
The twins/mouthwash example (Exercise 1.11):
20 sets of twins were used to test whether a fluoride mouthwash reduces cavities. One twin in each set used the mouthwash; the other did not. This is a matched pairs experiment: the twin pair controls for genetics, diet, brushing habits, and other factors that twins share.
Students often think "control group" means a group that the researcher controls. It means a group that receives no treatment or a placebo, serving as a baseline for comparison.
Students confuse "factor" with "treatment." A factor is the explanatory variable being tested. A treatment is a specific combination of factor levels. When there is only one factor with two levels, there are two treatments, and the terms can feel interchangeable, but they are distinct concepts.
Students sometimes think that matched pairs experiments are always better than completely randomised experiments. Matched pairs are more precise when pairing is done well, but they are limited to comparing exactly two treatments and require identifying the right matching criteria.
Students assume that double-blinding means the pills have no labels. Double-blinding means that neither the subjects nor the person evaluating outcomes knows who received which treatment. Removing labels from pills is one way to help achieve this, but the definition is about knowledge, not packaging.
Be able to name and define all three principles of experiments (control, randomisation, replication). Exam questions often describe a study and ask which principle is missing or violated.
The gastric freezing example is exam-favourite material. Know why the initial design was flawed and how the improved design fixed it.
Expect questions asking you to identify the factor, levels, treatments, experimental units, and response variable in a described experiment. The corn/chick diet example (3 corn varieties x 3 protein levels = 9 treatments) tests whether you can handle multi-factor designs.
Know the difference between a completely randomised experiment and a matched pairs experiment, and be able to design both for a given scenario.
Understand what "double-blind" means and why it matters. The depression medication example (Exercise 1.11) is likely exam material: the factor is which treatment the volunteers receive; the study is double-blind if neither the volunteers nor the psychiatrist knew who received which drug.
True or False: An experiment without a control group can still establish causation.
False. Without comparison, the treatment effect is confounded with lurking variables and the placebo effect.
Fill in the blank: An experiment is __________ if it systematically favours certain outcomes.
Biased.
True or False: A matched pairs experiment can compare three treatments.
False. Matched pairs experiments always compare exactly two treatments.
Fill in the blank: In a double-blind experiment, neither the __________ nor the __________ knows who received which treatment.
Subjects, experimenter (in any order).
True or False: Replication means repeating the entire study from scratch.
False. Replication means applying each treatment to enough subjects within a study to reduce chance variation.
Q: In an experiment comparing two corn varieties at three protein levels, how many treatments are there? Identify the factors and their levels.
A: There are 9 treatments (3 corn varieties x 3 protein levels). The factors are corn variety (3 levels: opaque-2, floury-2, normal) and protein level (3 levels: 12%, 16%, 20%).
Q: Pine seedlings are planted in a greenhouse in either full light or shade (5% of normal). At the end of the study, the trees are dried and weighed. Identify the experimental units, treatments, factor, and response variable.
A: Experimental units: pine seedlings. Treatments: full light and shade (5% of normal). Factor: light level (2 levels). Response variable: dry weight of the trees.
Q: 100 volunteers with severe depression are divided at random: 50 receive a new drug, 50 receive an existing drug. A psychiatrist evaluates improvement after 4 weeks. What is the factor in this study?
A: The factor is which treatment the volunteers receive (the drug type). The response variable is the extent to which depression was reduced.
Q: Describe how you would design a matched pairs experiment to test whether the right hand is stronger than the left in right-handed people, using 15 subjects.
A: Each subject serves as their own matched pair (both hands belong to the same person). For each subject, flip a coin to decide which hand is tested first. Measure grip strength for both hands. Compare the right-hand and left-hand measurements within each subject. The random coin toss controls for any effect of testing order.
Q: In the green tea and prostate cancer study, 60 men received either green tea extract or a placebo in a double-blind design. After one year, 1 of 30 in the green tea group developed cancer vs 9 of 30 in the placebo group. Are the three principles of experiments satisfied? Can we draw a causal conclusion?
A: Yes, all three principles are satisfied: control (placebo comparison), randomisation (random assignment to groups), and replication (30 subjects per group). Because this is a randomised, controlled, double-blind experiment with a statistically significant difference, we can conclude that green tea extract helped prevent prostate cancer in this population.
The three principles here lay the groundwork for everything in the inference portion of the course. When you later learn about confidence intervals and hypothesis tests, you will be applying the logic of "is this difference too large to be due to chance?" that the concept of statistical significance introduces here. Matched pairs designs will return when you study paired t-tests. Multi-factor experiments connect to two-way ANOVA if your course covers it.
three principles of experiments, control group, comparison group, placebo, placebo effect, double-blind, double-blinded experiment, single-blind, completely randomised design, CRD, randomised comparative experiment, matched pairs design, matched pairs experiment, crossover design, factor, levels, treatment, experimental units, subjects, bias in experiments, randomisation methods, random number generator, random assignment, replication in experiments, gastric freezing, STA 270, introductory statistics, experimental design principles