Source: Paper I, Psychology II, Florida International University
Tags: experimental design, one-way ANOVA, chi-square, manipulation check, Tukey HSD, Likert scale, between-subjects design, online rudeness, fake news, impression formation
Difficulty: Intermediate | Prerequisites: Part 1 Literature Review notes; basic understanding of experimental design, independent/dependent variables, and hypothesis testing (p-values, significance).
This section covers how the researchers turned the literature review concepts (fake news, online rudeness, intergroup emotion) into a testable experiment. Study One uses a between-subjects design with three conditions to measure how a social media user's tone (rude vs. polite) affects observers' impressions of that user and their own emotional reactions. If you are revising experimental methods and basic inferential statistics for an exam, this is a clean, concrete example of a one-way ANOVA applied to a real social psychology question.
187 participants were randomly assigned to read a Facebook scenario where a user ("Corey") responded to "fake news" accusations either rudely, politely while disagreeing, or politely while agreeing. Participants in the rude condition rated Corey as significantly ruder and felt significantly angrier than those in either polite condition. The two polite conditions did not differ meaningfully from each other, supporting the hypothesis that tone, not the direction of disagreement, drives negative impressions.
Independent variable (IV)
The variable the researcher manipulates to observe its effect. In this study, the IV is the experimental condition: Rude Disagree (RD), Polite Disagree (PD), or Polite Agree (PA).
In simple terms, the IV is what the researchers changed between groups to see what would happen.
Dependent variable (DV)
The variable being measured as an outcome. Here, the DVs include participants' ratings of "Corey seems like a rude person" and "Reading Corey's post made me feel angry," both on a 1 to 6 Likert scale.
Think of it as the thing the researchers are watching to see if it moves.
Between-subjects design
An experimental design in which each participant is exposed to only one condition. Each participant saw only one version of Corey's response.
In simple terms, nobody saw more than one condition, so their ratings were not contaminated by having read the other versions.
One-way ANOVA
Analysis of Variance with one independent variable (here, condition: RD vs. PD vs. PA) and one dependent variable. It tests whether the means of three or more groups differ significantly.
Think of it as the statistical test that answers: "Do these three groups score differently enough that it is unlikely to be due to chance?"
Tukey HSD (Honestly Significant Difference)
A post hoc test run after a significant ANOVA result to identify which specific pairs of groups differ from each other.
In simple terms, the ANOVA tells you "at least one group is different"; Tukey HSD tells you which ones.
Chi-square test of independence
A statistical test used to determine whether two categorical variables are related. Here it was used for the manipulation check, testing whether condition assignment was associated with what participants remembered Corey's final comment to be.
Think of it as: "Are these two categories linked, or are they independent of each other?"
Cramer's V
An effect-size measure for chi-square tests, ranging from 0 (no association) to 1 (perfect association). A value of .631 in this study indicates a strong association.
In simple terms, Cramer's V tells you how strong the relationship is that the chi-square test detected.
Manipulation check
A question included in the study to verify that participants noticed and correctly understood the experimental manipulation. Here, participants were asked to recall which type of response Corey gave.
Think of it as a sanity check: "Did participants pay attention and see what we wanted them to see?"
Likert scale
An ordinal rating scale (here 1 to 6) where respondents indicate their level of agreement with a statement. 1 = strongly disagree, 6 = strongly agree.
In simple terms, it is the "rate from 1 to 6 how much you agree" format.
N = 187 (100 male, 87 female)
Age range: 16 to 59; mean age 25.35 (SD = 9.109)
Ethnicity: 47.6% Hispanic, 25.1% Caucasian, 16.6% African American, 1.6% Native American, 4.3% Asian American, 4.8% Other
Recruited from Florida International University
Design type: Between-subjects, three conditions
Conditions:
Rude Disagree (RD): Corey calls commenters "liberal idiots" and "morons," mocks their feedback, insists the story is real
Polite Disagree (PD): Corey politely disagrees with the "fake news" label and suggests commenters do their own research
Polite Agree (PA): Corey admits they were wrong and politely agrees the story is fake after further research
Stimulus: A mock Facebook page for "Corey" (gender kept deliberately ambiguous) sharing a fabricated CNN story. Two commenters (Peyton and Riley) point out the story is fake. The only element that varies across conditions is Corey's reply to those comments.
Participants were randomly assigned to one condition.
Part 1: Participants examine Corey's full Facebook page (profile, photos, shared post, comments)
Part 2: Rate impressions of Corey on eight statements using a 1 to 6 Likert scale (reasonable, polite, open-minded, civility, rude, tolerant, willing to mock, likeable)
Part 3: Rate impressions of the comment exchange on seven statements (same scale), including items about anger, hope, willingness to engage, and belief that the story is fake
Part 4: Demographic questions (gender, age, race, first language, FIU student status, political affiliation)
Part 5: Manipulation check, asking participants to select which version of Corey's response they saw
Manipulation check (chi-square):
Chi-square X²(4) = 149.11, p < .05
69.7% of RD participants, 66.7% of PD participants, and 80.0% of PA participants correctly recalled their condition
Cramer's V = .631, indicating a strong association
Conclusion: participants noticed the manipulation as intended
DV 1: "Corey seems like a rude person" (one-way ANOVA):
F(2, 184) = 13.797, p < .05 (this is reported as 7.057 in the ANOVA table, with sum of squares 13.797 between groups; both refer to the same significant result)
Means: RD = 3.38 (SD = 1.187), PD = 2.98 (SD = 0.920), PA = 2.71 (SD = 0.786)
Tukey HSD: RD was rated significantly ruder than PA (mean difference = .670, p = .001). RD vs. PD approached but did not reach significance (mean difference = .394, p = .060). PD vs. PA did not differ significantly (mean difference = .276, p = .280).
DV 2: "Corey's post made me angry" (one-way ANOVA):
F(2, 184) = 26.238, p < .05 (F = 15.606 in the table)
Means: RD = 3.09 (SD = 1.160), PD = 2.23 (SD = 0.760), PA = 2.45 (SD = 0.741)
Tukey HSD: RD produced significantly more anger than both PD (mean difference = .864, p < .001) and PA (mean difference = .636, p = .001). PD and PA did not differ significantly (mean difference = .227, p = .365).
Results supported the main hypothesis: rude disagreement produced significantly worse impressions and more anger than polite responses.
The two polite conditions (PD and PA) did not differ meaningfully, suggesting it is the tone of the response, not whether the person agrees or disagrees, that drives negative reactions.
The PD condition sometimes fell between RD and PA without being significantly different from either, likely because participants varied in their personal thresholds for what counts as rude.
Open question raised by the authors: what exactly constitutes a "rude" vs. "polite" response? This motivated Study Two.
This study design is a textbook example of how social psychologists test causal claims about online behaviour. The same between-subjects approach with a mock social media post could be adapted to study reactions to different types of content moderation messages, brand responses to customer complaints, or the impact of anonymity on comment tone. For anyone designing surveys or experiments in a research methods course, the five-part survey structure and the use of a manipulation check are patterns worth learning to replicate.
Students often confuse the F-statistic with the sum of squares in an ANOVA table. The F-value is the ratio of between-group variance to within-group variance, not the raw sum of squares. In this study, the between-groups sum of squares is 13.797, but F(2, 184) = 7.057 for the rudeness DV.
A significant ANOVA does not tell you which groups differ. You need the post hoc test (Tukey HSD) for that. Students sometimes skip this step in their reasoning.
Students sometimes think that a non-significant difference between PD and PA means those two conditions produced identical results. It means the difference was too small to rule out chance, not that the means were the same.
The manipulation check uses chi-square because both variables (condition and recalled response) are categorical. Students sometimes try to apply ANOVA to categorical data, which is incorrect.
⚠️ Be able to identify the IV (condition: RD, PD, PA) and the DVs (rudeness rating, anger rating) for this study. Exam questions frequently ask you to label these.
⚠️ Know why a manipulation check was necessary and what a strong Cramer's V (.631) tells you about its success.
⚠️ Understand the logic of post hoc tests: the ANOVA says "there is a significant difference somewhere among the three groups," the Tukey HSD pinpoints where. Be ready to interpret specific pairwise comparisons.
⚠️ The key finding: tone (rude vs. polite) drives negative impressions, not the direction of disagreement (agree vs. disagree). This is the central takeaway and a likely exam question.
⚠️ Be aware that the PD condition sitting between RD and PA without significantly differing from both is a nuanced result. The authors attribute this to individual variation in what people consider "rude."
True or False: This study used a within-subjects design. (False. It was between-subjects; each participant saw only one condition.)
Fill in the blank: The three conditions were Rude Disagree, Polite Disagree, and ________. (Polite Agree)
True or False: A significant one-way ANOVA tells you exactly which groups differ from each other. (False. You need a post hoc test such as Tukey HSD for that.)
Fill in the blank: The manipulation check used a ________ test because both variables were categorical. (Chi-square)
True or False: Participants in the Polite Disagree condition rated Corey as significantly less rude than those in the Polite Agree condition. (False. The PD and PA conditions did not differ significantly from each other.)
Q: What was the independent variable in Study One, and what were its levels?
A: The IV was the experimental condition (the tone and content of Corey's response). It had three levels: Rude Disagree (RD), Polite Disagree (PD), and Polite Agree (PA).
Q: Why did the researchers keep Corey's gender ambiguous?
A: To prevent participants from relying on gender expectations or stereotypes when forming their impressions. This controls for a potential confound.
Q: The ANOVA for "Corey seems like a rude person" was significant. What did the Tukey HSD post hoc test reveal about which specific groups differed?
A: RD was rated significantly ruder than PA (p = .001). RD vs. PD approached significance (p = .060) but did not reach the .05 threshold. PD and PA did not differ significantly (p = .280).
Q: What was the purpose of the manipulation check, and what statistical test was used to evaluate it?
A: The manipulation check verified that participants correctly recalled which type of response Corey gave, confirming they attended to the key experimental manipulation. A chi-square test of independence was used because both variables (assigned condition and recalled response) are categorical. Cramer's V of .631 confirmed a strong association.
Q: The authors found that PD did not differ significantly from either RD or PA on the rudeness measure. What does this suggest about how people perceive politeness?
A: It suggests individual variation in where people draw the line between polite and rude. Some participants may have found even polite disagreement somewhat rude, while others did not. The threshold for perceived rudeness differs from person to person.
Q: For the anger DV, which pairwise comparisons were significant?
A: RD produced significantly more anger than both PD (p < .001) and PA (p = .001). PD and PA did not differ significantly (p = .365). This indicates that rude tone, specifically, is what triggers anger in observers.
This study connects to research methods units: it is a clean example of a between-subjects experiment with random assignment, a manipulation check, one-way ANOVA, and post hoc testing. It also links to impression formation in social cognition (how quickly and strongly we judge others based on limited information) and to emotion and cognition research (how anger aroused by incivility might affect subsequent decision-making or engagement online).
between-subjects design, one-way ANOVA, Tukey HSD post hoc, chi-square test of independence, Cramer's V, manipulation check, Likert scale, impression formation, online rudeness experiment, fake news reactions, social media psychology study, Facebook incivility research, PSY 2020 FIU, anger and social media, politeness perception, experimental conditions, dependent variable, independent variable