Source: Research Methods, Statistics, and Applications, Ch. 1–2
Tags: validity, internal validity, external validity, experimenter bias, demand characteristics, confounds, placebo effect, double-blind, population, sample, research bias, psychology
Difficulty: Introductory | Prerequisites: Research Methods and Designs notes (the companion document in this set)
Knowing the research designs is only half the picture. This material covers how to judge whether a study's conclusions are trustworthy. Validity asks "do these results mean what the researchers say they mean?" while bias covers the many ways a study can go sideways without anyone noticing. Sampling determines who the findings apply to. Together, these concepts give you the critical lens you need to evaluate any published study.
Validity comes in two flavours: internal (did the independent variable cause the change?) and external (do the results generalise to the real world?). Bias can creep in from the experimenter's side or the participant's side, and the double-blind procedure is the standard defence. Your sample needs to represent your population, or your conclusions do not travel.
Validity
The soundness of the conclusions drawn from an experiment. A valid study measures what it claims to measure and supports the inferences the researchers draw. In simple terms, validity is about whether a study's results are believable and meaningful.
Internal validity
The degree to which changes in the dependent variable can be attributed to manipulation of the independent variable, rather than to some other factor. Think of it as: "Can I be sure the treatment caused the effect?"
External validity
The degree to which an experiment's findings generalise to real-world settings and populations beyond the lab. Think of it as: "Do these results apply outside this specific study?"
Experimenter bias
A threat to validity in which the experimenter's expectations influence the outcome of the research. In simple terms, the researcher unconsciously steers the results toward what they hope to find.
Demand characteristics
Cues in the experimental setting that communicate to participants how the experimenter wants them to behave. In simple terms, participants pick up on hints about what the "right" answer is and adjust their behaviour accordingly.
Confounds
Systematic biases or extraneous variables that undesirably influence the dependent variable, making it impossible to isolate the effect of the independent variable. Think of it as noise that muddies the signal.
Research participant bias
A situation in which the behaviour of research participants is influenced by their own expectations and thoughts about the study. In simple terms, participants change how they act because they know they are being studied.
Placebo effect
A phenomenon in which participants' expectations alone produce a particular outcome, even without any real treatment. Think of it as the mind creating the result the person expects.
Placebo
A harmless, inert substance or procedure that has no physiological effect. Used as a comparison condition so researchers can separate the effect of the actual treatment from the effect of simply believing you received treatment.
Double-blind experiment
An experimental design in which neither the experimenter nor the participants know who is in the experimental group and who is in the control group. In simple terms, both sides are kept in the dark to prevent bias from either direction.
Population
The entire group of individuals that the investigator wants to draw conclusions about. Think of it as the "big picture" group your research question is really about.
Sample
A subset of the population selected for study. The goal is for the sample to represent the population so that findings can generalise.
Internal validity is about control. The tighter the researcher's control over extraneous variables, the more confident you can be that the independent variable caused the observed effect.
Threats to internal validity include confounds, experimenter bias, and demand characteristics.
External validity is about generalisability. A study with high internal validity may still have low external validity if the lab setting is too artificial or the sample too narrow.
There is often a trade-off: increasing control (internal validity) can reduce how natural the setting feels (external validity).
From the experimenter's side:
Experimenter bias: the researcher's expectations unconsciously shape how they interact with participants or interpret data.
Demand characteristics: subtle cues in the study's setup that telegraph the expected behaviour to participants.
From the participant's side:
Research participant bias: participants modify their behaviour because they know they are in a study.
Placebo effect: participants improve (or report improvement) simply because they believe they are receiving treatment.
Socially desirable responding (covered in the companion document): participants answer in ways that make them look favourable.
In a double-blind experiment, neither the participants nor the experimenter knows who is in the experimental group and who is in the control group.
This eliminates experimenter bias and reduces demand characteristics and participant expectancy effects simultaneously.
A placebo is given to the control group so that both groups believe they are receiving the real treatment.
The population is the full group you want your conclusions to apply to.
The sample is the subset you actually study.
If your sample does not represent the population well, your findings may not generalise, regardless of how well-designed the experiment itself is.
Random assignment (within the study) is different from random sampling (selecting participants from the population). Both matter, but they address different problems.
Double-blind trials are the standard in pharmaceutical research. Every drug you take passed through this process: neither the doctor administering the pill nor the patient knew whether it was the real drug or the placebo. External validity is why psychologists worry about running studies only on university students; findings from 19-year-old psychology majors may not apply to the broader population.
Students often confuse internal and external validity. Internal = "Did the IV cause the change in the DV?" External = "Can these results be applied to the real world?"
Students sometimes think the placebo effect means the treatment did not work. The placebo effect is about the control group improving; the treatment may still work above and beyond the placebo.
Students assume "double-blind" means participants do not know they are in a study. It means neither party knows which condition each participant is in. Both know a study is happening.
Students mix up population and sample. The population is the big group you care about; the sample is the smaller group you actually test.
Expect scenario-based questions asking you to identify threats to internal or external validity.
Know every source of bias by name and be able to explain how it distorts results.
Understand the logic of the double-blind design: which biases it controls and why a placebo is necessary.
Be able to distinguish between random assignment and random sampling, and explain what each one protects against.
Questions about population vs. sample often appear as short-answer or multiple-choice items.
True or False: Internal validity refers to whether findings generalise to the real world. (False, that is external validity. Internal validity is about whether the IV caused the change in the DV.)
Fill in the blank: A ______ is a harmless substance with no physiological effect, used as a comparison in experiments. (Placebo)
True or False: In a double-blind experiment, participants do not know they are in a study. (False. They know they are in a study; they do not know which condition they are in.)
Fill in the blank: The entire group a researcher wants to draw conclusions about is called the ______. (Population)
True or False: Demand characteristics are cues that tell participants how the experimenter expects them to behave. (True)
Q: A researcher studying the effects of a new therapy tells the therapists which patients are receiving the real treatment and which are in the control group. The treated patients improve more. What threat to validity is most likely at play?
A: Experimenter bias. The therapists may unconsciously treat the two groups differently because they know who is receiving the real therapy.
Q: Explain the difference between internal validity and external validity, and describe one situation where they might conflict.
A: Internal validity is the confidence that changes in the DV are caused by the IV. External validity is the confidence that findings apply beyond the specific study. They can conflict when a researcher tightly controls the lab environment (boosting internal validity) but creates a setting so artificial that results do not reflect real-world behaviour (lowering external validity).
Q: A drug company tests a new painkiller. The control group receives a sugar pill, but many of them report feeling less pain. What phenomenon explains this?
A: The placebo effect. Participants expected the pill to reduce pain, and that expectation alone produced a perceived improvement.
Q: Why is it important that a sample is representative of the population?
A: If the sample does not represent the population, the results may only apply to the specific group tested and cannot be generalised. This limits the study's external validity.
This material builds directly on the research designs covered in the companion document. Validity and bias are the criteria you use to evaluate those designs. Later in the course, when you encounter statistical significance and effect sizes, you will rely on these concepts to judge whether a statistically significant result is also practically meaningful and trustworthy.
Validity, internal validity, external validity, experimenter bias, observer bias, demand characteristics, confounds, confounding variable, extraneous variable, participant bias, Hawthorne effect, placebo effect, placebo, sugar pill, double-blind, single-blind, blinding, population, sample, sampling, representative sample, random sampling, random assignment, generalisability, research bias, threats to validity