Source: Evaluation Design Assignment
Tags: evaluation design, vaccination policy, RCT, observational study, experimental method, Covid-19, Wyoming, vaccine confidence, herd immunity, behavioural nudges
Difficulty: Intermediate | Prerequisites: Basic understanding of research methods (independent/dependent variables, control groups). Familiarity with policy evaluation concepts is helpful but not required.
This topic sits at the intersection of public health policy and research methodology. The core question: how do you design a study to test whether a policy intervention (here, pro-vaccine advertisements) causes a change in behaviour (vaccination rates)? You need to understand both the substance of the policy problem (why Wyoming's vaccination rate trails the national average) and the mechanics of evaluation design (observational vs. experimental approaches, what makes an RCT strong or weak, and how to control for confounders). If your course covers programme evaluation, causal inference, or evidence-based policy, this is foundational material.
Wyoming's Covid-19 vaccination rate (54.2% as of October 2022) lagged well behind the U.S. average (roughly 80%). The proposed policy response is a mandate for public advertisements promoting vaccine benefits, aimed at building vaccine confidence rather than expanding infrastructure. The assignment walks through two evaluation designs for testing this policy: an observational comparison of two cities and an experimental (RCT-style) pre-test/post-test design with treatment and control groups. A related published RCT on behavioural nudges (text reminders) showed meaningful increases in vaccine uptake.
Vaccinology
The branch of science concerned with the development, production, and effectiveness of vaccines. In simple terms, it is the study of how vaccines work and how to make them better.
Herd immunity (community immunity)
The indirect protection from an infectious disease that occurs when a large enough portion of a population becomes immune, reducing the likelihood of transmission to those who are not immune. Think of it as a firebreak: once enough people are vaccinated, the disease runs out of hosts and even unvaccinated individuals benefit.
Vaccine confidence
The degree to which the general public trusts vaccines to be safe, effective, and worth receiving. In simple terms, it is whether people believe vaccines are a good idea and are willing to get them.
Endogenous relationship
A relationship between two variables where each influences the other, making it difficult to isolate cause and effect. Here, development drives vaccination rates, and vaccination rates support further development. Think of it as a feedback loop.
Observational study
A research design where the researcher observes and measures variables without manipulating them. No treatment is assigned; existing conditions are compared. In simple terms, you watch what happens naturally and draw conclusions from the differences you see.
Experimental method (randomised controlled trial / RCT)
A research design where the researcher randomly assigns participants to a treatment group or a control group, then introduces an intervention to the treatment group only. This is the gold standard for establishing causation because randomisation controls for confounding variables.
Treatment group
The group in an experiment that receives the intervention being tested (e.g. the city exposed to vaccine advertisements).
Control group
The group in an experiment that does not receive the intervention. It serves as the baseline for comparison, allowing the researcher to attribute any difference in outcomes to the treatment rather than to other factors.
Baseline measurement (pre-test)
A measurement of the outcome variable taken before the intervention begins. Establishes the starting point so that any change after the intervention can be quantified.
Behavioural nudge
A small, low-cost change in how choices are presented that predictably influences people's behaviour without restricting their options. In the context of this material, text-message reminders that frame a vaccine dose as "yours" are a nudge designed to increase uptake.
Independent variable
The variable the researcher manipulates or changes (e.g. the presence or absence of vaccine advertisements).
Dependent variable
The variable the researcher measures to see if it was affected by the manipulation (e.g. vaccination rates).
Confounding / extraneous variables
Variables other than the independent variable that could explain changes in the dependent variable. A well-designed experiment controls for these so that any observed effect can be attributed to the treatment.
The source material organises vaccine benefits into three categories: medical, societal, and economic. These form the justification for why increasing vaccination rates matters as a policy goal.
Medical benefits
Vaccines have an established correlation with reduced mortality rates. An estimated 6 million deaths from vaccine-preventable diseases are averted annually (Ehreth, 2003).
Vaccination programmes serve as a gateway to primary care in developing countries, sometimes representing the only recurring contact between families and health services (Shearley).
Childhood vaccination leads to measurable decreases in child mortality, which in turn influences family planning: families with lower child mortality tend to have fewer children.
Societal benefits
Vaccination of children has an outsized impact on women in developing countries, who bear the primary caregiving burden. Protecting children's health frees women's time, energy, and resources.
Access to vaccination gives women a sense of control and empowerment over family health outcomes.
High vaccination coverage increases the probability of achieving herd immunity, which protects entire communities, including those who cannot be vaccinated (e.g. immunocompromised individuals).
Economic benefits
In the U.S., routine childhood vaccination prevents approximately 42,000 deaths and 20 million disease cases per birth cohort, averting an estimated $76 billion in societal costs (Deogaonkar et al., 2012).
Most vaccination programmes cost less than $50 per life gained (Ehreth, 2003).
The U.S. obtains a net economic benefit of $69 billion from its vaccination programmes.
Reduced disease incidence lowers treatment costs and healthcare spending broadly.
The problem
As of October 2022, roughly 80% of the U.S. population had received at least one Covid-19 vaccination shot (CDC data).
Wyoming's rate stood at 54.2%, well below the national average.
The gap is not primarily an infrastructure problem. Wyoming has the healthcare systems needed to deliver vaccines. The barrier is vaccine confidence, not vaccine access.
The proposed policy
A mandate requiring advertisements that promote the Covid-19 vaccine across Wyoming. The advertisements would take the form of billboards, television ads, posters, and newspaper placements. The goal is to increase public knowledge of the medical, societal, and economic benefits of vaccination, thereby building vaccine confidence and encouraging uptake.
Key distinction to understand
In many countries, low vaccination rates result from practical barriers: limited healthcare infrastructure, poor supply chains, lack of trained personnel. In Wyoming, those barriers are largely absent. The assignment frames the problem as attitudinal rather than logistical, which is why the proposed intervention targets confidence (through advertising) rather than access (through clinic expansion or mobile units).
The assignment proposes two ways to evaluate whether vaccine advertisements increase vaccination rates. Understanding the differences between these designs is central to the material.
Design 1: Observational approach
Select two cities in Wyoming. City A already has many Covid-19 vaccine advertisements; City B has none.
Survey every household in both cities to establish each city's vaccination rate.
Compare the two rates to look for a correlation between advertisement presence and higher vaccination.
Strength: relatively simple and inexpensive to execute.
Limitation: cannot establish causation. The two cities may differ in ways that independently affect vaccination rates (income, education, political leaning, demographics). Any observed difference could be due to these confounders rather than the advertisements themselves.
Limitation: selection bias. Cities are not randomly assigned; City A may already have advertisements precisely because it has higher vaccine demand, reversing the causal direction.
Design 2: Experimental approach (pre-test/post-test with control)
Randomly select two cities in Wyoming. Survey every household in both to establish baseline vaccination rates.
City A receives the treatment: billboards, posters, television, and newspaper advertisements promoting the Covid-19 vaccine. City B (the control group) receives nothing.
After the intervention period, survey both cities again to establish final vaccination rates. Compare the change in City A to the change in City B.
Strength: random selection and the use of a control group allow the researcher to make causal claims. If City A's rate increased more than City B's, and no other systematic difference exists, the advertisements are the likely cause.
Strength: baseline measurement means the researcher can measure the magnitude of the change, not just the endpoint.
Limitation: with only two cities, the sample of communities is very small, which limits generalisability.
Limitation: spillover effects are possible (residents of City B may see City A's advertisements through television or travel).
Why this distinction matters
Observational designs can identify correlations. Experimental designs (particularly RCTs) can identify causation. For policy decisions, causation matters: a government needs to know that spending money on advertisements will produce results, not merely that advertisements and higher vaccination tend to appear together.
The assignment reviews a published study that uses a similar methodology. Understanding this study and its strengths and weaknesses is likely exam-relevant.
The study
Dai et al. (2021) conducted two sequential randomised controlled trials testing behavioural nudges (text-message reminders) on Covid-19 vaccine uptake. Participants were drawn from the UCLA Health primary and specialty care patient list.
Group 1: 93,354 individuals received a text reminder one day after receiving a vaccine eligibility notification.
Group 2: 67,092 individuals received a text reminder eight days after eligibility notification.
Results
The first reminder increased the vaccination rate by 6.07 percentage points (an 84% relative increase).
The second reminder increased the vaccination rate by 3.57 percentage points (a 26% relative increase).
The first message was more effective partly because it was framed to create a sense of ownership over the vaccine dose ("your vaccine is waiting for you" rather than a neutral reminder).
Strengths identified
Large sample sizes (93,354 and 67,092) provide high statistical power and make the results more reliable.
Random selection of participants helps ensure the sample approximates the broader population, reducing selection bias.
Weaknesses identified
The study's racial demographic reporting was limited. The Extended Data Table 1 Randomisation check for the first RCT listed only one racial category (white). Without full demographic breakdowns, one cannot confirm whether the sample is representative of the U.S. population.
The absence of a true control group (a group receiving no reminder at all) for both trial arms limits the ability to isolate the effect of the text reminders from other factors.
Proposed improvements
Add a control group that receives no treatment (no text reminders), allowing for a cleaner comparison.
Select cities or participant pools whose racial demographics are representative of the broader U.S. population to improve external validity.
Students often think an observational study can prove causation. It cannot. Observational designs can only identify correlations or associations. To claim that X causes Y, you need an experimental design with random assignment and a control group.
Students often confuse "correlation" with "endogenous relationship." An endogenous relationship specifically means the two variables influence each other (a feedback loop). A correlation simply means they move together, without specifying direction or mechanism.
Students sometimes assume that a large sample size alone makes a study representative. Sample size increases statistical power, but representativeness depends on how the sample was selected and whether it reflects the demographics of the broader population.
Students often overlook spillover effects in community-level experiments. If City A's television advertisements reach residents of City B, the control group is contaminated, and the measured treatment effect will be understated.
⚠️ Be able to articulate the difference between observational and experimental designs, and explain why only experimental designs can establish causation.
⚠️ Know what a control group is and why it matters. A common exam question: "What would happen to the study's conclusions if there were no control group?"
⚠️ Understand the role of random assignment in reducing confounders. Be prepared to explain how selection bias arises in observational studies.
⚠️ The distinction between vaccine access and vaccine confidence is likely testable. Wyoming's problem is framed as attitudinal (confidence), not logistical (access), and the policy intervention matches that diagnosis.
⚠️ The Dai et al. study's results (6.07 and 3.57 percentage-point increases) and the concept of behavioural nudges (especially ownership framing) are the kind of specific findings that appear in short-answer or essay prompts.
⚠️ External validity: be ready to discuss whether findings from a UCLA patient population generalise to Wyoming, and why demographic representativeness matters.
True or False: An observational study can establish a causal relationship between two variables. (False. Observational studies can only establish correlations.)
Fill in the blank: The group in an experiment that does not receive the intervention is called the ________. (Control group.)
True or False: Wyoming's low vaccination rate is primarily due to a lack of healthcare infrastructure. (False. The assignment frames the issue as one of vaccine confidence, not infrastructure.)
Fill in the blank: In the Dai et al. study, the first text reminder increased the vaccination rate by ________ percentage points. (6.07.)
True or False: A large sample size guarantees that a study's participants are demographically representative of the broader population. (False. Representativeness depends on how participants were selected, not just how many were included.)
Q: Explain why an observational comparison of two cities cannot establish that vaccine advertisements cause higher vaccination rates.
A: In an observational design, the researcher does not control which city receives advertisements. The cities may differ in demographic composition, income levels, political attitudes, or other factors that independently affect vaccination rates. Any observed difference in vaccination rates could be driven by these confounders rather than the advertisements. Without random assignment and a control group, correlation cannot be distinguished from causation.
Q: Describe the experimental evaluation design proposed in the assignment and identify its key advantage over the observational approach.
A: The experimental design randomly selects two cities, surveys both to establish baseline vaccination rates, exposes City A to vaccine advertisements (the treatment) while City B receives nothing (the control), then surveys both again to measure the change. The key advantage is that random assignment and the use of a control group allow the researcher to attribute any difference in outcomes to the treatment, controlling for confounding variables.
Q: What were the main findings of the Dai et al. (2021) study on behavioural nudges and Covid-19 vaccine uptake?
A: The first text reminder (sent one day after eligibility notification) increased vaccination rates by 6.07 percentage points (84% relative increase). The second reminder (sent eight days after) increased rates by 3.57 percentage points (26% relative increase). The first message was more effective in part because it was framed to create a sense of ownership over the vaccine dose.
Q: Identify one weakness of the Dai et al. study and explain how it limits the study's conclusions.
A: The study's racial demographic reporting was incomplete. The randomisation check for the first RCT listed only white participants, without breakdowns for other racial groups. This makes it difficult to assess whether the sample is representative of the U.S. population, which limits the external validity of the findings. Results that hold for one demographic group may not generalise to others.
Q: Why does the assignment argue that Wyoming needs vaccine confidence rather than improved vaccine infrastructure?
A: As of October 2022, Wyoming's vaccination rate was 54.2%, well below the national average of approximately 80%. The assignment notes that Wyoming already has the healthcare infrastructure to deliver vaccines. The barrier is not access but willingness. The proposed policy therefore targets attitudes (through advertising) rather than logistics (through new clinics or supply chains).
This material connects directly to research methods and causal inference. The observational vs. experimental distinction is the same framework used across the social sciences, from economics (natural experiments, instrumental variables) to psychology (between-subjects designs). If your course covers programme evaluation or evidence-based policy, the logic here is identical.
The concept of behavioural nudges connects to behavioural economics (Thaler and Sunstein's "Nudge" framework). The Dai et al. study's use of ownership framing is a textbook application of the endowment effect: people value something more when they feel it is already theirs.
The herd immunity concept links to epidemiology and public health coursework. Understanding the threshold at which herd immunity is achieved (and why it varies by disease) is relevant well beyond this assignment.
Evaluation design, programme evaluation, policy evaluation, observational study, experimental design, randomised controlled trial, RCT, control group, treatment group, pre-test post-test design, baseline measurement, independent variable, dependent variable, confounding variables, extraneous variables, selection bias, external validity, internal validity, causal inference, correlation vs. causation, vaccine hesitancy, vaccine confidence, vaccine uptake, herd immunity, community immunity, behavioural nudge, nudge theory, endowment effect, ownership framing, Covid-19 vaccination, Wyoming vaccination rate, public health policy, vaccination programme, cost-effectiveness of vaccines, morbidity, mortality, Dai et al. 2021, Ehreth 2003, Deogaonkar et al. 2012, Shearley