Source: Handout #13, Tamhane/Dunlop Ch. 8, Ch. 9, Section 14.2
Tags: chi-squared test, homogeneity of proportions, test of independence, Fisher exact test, hypergeometric distribution, odds ratio, relative risk, logit, sensitivity, specificity, Bayes theorem, prevalence, McNemar test, matched pairs, Cochran-Mantel-Haenszel, CMH, Simpson paradox, Breslow-Day, contingency table
Comparing population proportions uses chi-squared tests (for large samples) or Fisher's exact test (for small samples). The chi-squared test of homogeneity asks whether several populations share the same proportion, while the test of independence asks whether two categorical variables are related within a single population. The odds ratio and relative risk quantify the size of association. McNemar's test handles matched-pair proportions, and the Cochran-Mantel-Haenszel test controls for a confounding variable across multiple 2x2 tables, guarding against Simpson's paradox.
Product multinomial sampling model
t independent random samples of fixed sizes n₁, ..., nₜ are drawn from t populations. Each unit is classified into categories. This is the design behind the test of homogeneity.
Pearson chi-squared statistic
χ² = ΣΣ (Oᵢⱼ − Êᵢⱼ)² / Êᵢⱼ. Compares observed cell counts to what would be expected under the null hypothesis. Under H₀, this converges to a chi-squared distribution.
Expected cell count
Êᵢⱼ = (ith row total × jth column total) / N. The count you would expect if the null hypothesis were true.
Fisher's exact test
An exact test for 2×2 tables using the hypergeometric distribution. Used when any expected cell count is below 1, or more than 20% of expected counts are below 5.
Odds ratio (OR)
OR = [p₁/(1 − p₁)] / [p₂/(1 − p₂)] = n₁₁ n₂₂ / (n₁₂ n₂₁). Compares the odds of an event in one group to the odds in another. OR = 1 means no difference; OR > 1 means group 1 has higher odds.
Relative risk (RR)
RR = p₁ / p₂. The ratio of the probability of the event in group 1 to the probability in group 2. More interpretable than OR but only valid in prospective or cross-sectional studies.
Sensitivity
P(Test + | Event Present). The proportion of true positives the test correctly identifies.
Specificity
P(Test − | Event Absent). The proportion of true negatives the test correctly identifies.
McNemar's test
A test for matched-pair proportions. Tests whether the off-diagonal counts n₁₂ and n₂₁ differ, which is equivalent to testing marginal homogeneity p₁. = p.₁.
Cochran-Mantel-Haenszel (CMH) test
Tests for conditional independence of two variables given a third (stratifying) variable. Combines evidence across K separate 2×2 tables.
Simpson's paradox
A reversal of an observed association when data is aggregated vs stratified. An overall marginal association can be opposite in direction to the conditional associations within every stratum.
This test asks: do t populations have the same proportion of a given type?
Setup: t populations, independent samples of size n₁, ..., nₜ, each unit classified as Type A or Type B. The data forms a 2 × t table.
Hypotheses:
H₀: p₁ = p₂ = ... = pₜ = p₀ (unknown common proportion) H₁: pᵢ ≠ pⱼ for at least one pair
Expected counts under H₀:
Ê₁ⱼ = nⱼ R₁/N and Ê₂ⱼ = nⱼ R₂/N, where R₁ is the row total and N is the grand total.
Test statistic:
χ² = ΣΣ (Oᵢⱼ − Êᵢⱼ)² / Êᵢⱼ
Under H₀, this converges to a chi-squared distribution with df = t − 1.
Degrees of freedom derivation: Under H₀ ∪ H₁ there are t unknown parameters (p₁, ..., pₜ). Under H₀ alone there is 1 unknown parameter (p₀). So df = t − 1.
Validity condition: If any Êᵢⱼ < 1 or more than 20% of the Êᵢⱼ < 5, the chi-squared approximation is unreliable. Use Fisher's exact test instead.
R: chisq.test() or fisher.test() for exact test.
When comparing exactly two proportions with large samples, the Z-test is equivalent to the chi-squared test and allows direct power and sample size calculations.
Test statistic:
Z = (p̂₁ − p̂₂) / √[p̂(1 − p̂)(1/n₁ + 1/n₂)]
where p̂ = (n₁p̂₁ + n₂p̂₂) / (n₁ + n₂) is the pooled proportion. Under H₀ (p₁ = p₂), Z is approximately N(0,1).
Power function (upper one-sided):
γ(p₁, p₂) = 1 − Φ[(z_α √[p(1−p)(1/n₁ + 1/n₂)] − (p₁ − p₂)) / σ]
where σ = √[p₁(1−p₁)/n₁ + p₂(1−p₂)/n₂] and p = (n₁p₁ + n₂p₂)/(n₁ + n₂).
Multiple comparisons: When t > 2, follow up a significant overall chi-squared test with pairwise Z-tests using a Bonferroni correction: α_pc = α_Exp / M, where M = C(t,2) is the number of pairs.
One-sided test:
n = [z_α √((p₁+p₂)(2−p₁−p₂)/2) + z_β √(p₁(1−p₁) + p₂(1−p₂))]² / D²
where D = p₁ − p₂.
Two-sided: Replace z_α with z_(α/2).
R: power.prop.test(n=, p1=, p2=, sig.level=, power=, alternative=)
When the proportions are very small (rare events), the required sample sizes can be enormous. For example, detecting a difference between p₁ = .0003 and p₂ = .00015 at α = .05 and 90% power requires n₁ = n₂ ≈ 171,237 per group.
When the chi-squared approximation is invalid, the Fisher exact test uses the hypergeometric distribution directly.
Setup: A 2×2 table with fixed row and column totals. Under H₀ (p₁ = p₂), the number of successes in group 1 (X) has a hypergeometric distribution:
P(X = x | X + Y = m) = C(n₁, x) C(n₂, m−x) / C(n, m)
One-sided p-value (H₁: p₁ > p₂):
p-value = P(X ≥ x) = Σ from k=x to min(n₁,m) of hypergeometric probabilities
R: 1 - phyper(x - 1, n1, n2, m) or fisher.test(M, alternative="g")
Two-sided p-value: Sum the probabilities of all tables no more likely than the observed table. This is not simply 2 × the one-sided p-value.
R: fisher.test(M, alternative="two.sided")
This differs from the homogeneity test in design and hypothesis, though the arithmetic is similar.
Design: A single random sample of N units from one population. Each unit is classified on two categorical variables (X with r levels, Y with c levels), producing an r × c contingency table.
Hypotheses:
H₀: pᵢⱼ = pᵢ. p.ⱼ for all (i,j) – X and Y are independent H₁: pᵢⱼ ≠ pᵢ. p.ⱼ for at least one (i,j)
Expected counts:
Êᵢⱼ = nᵢ. n.ⱼ / N = (row total × column total) / grand total
Test statistic:
χ² = ΣΣ (Oᵢⱼ − Êᵢⱼ)² / Êᵢⱼ, with df = (r − 1)(c − 1)
Degrees of freedom derivation:
Under H₀ ∪ H₁: rc − 1 unknown parameters (the pᵢⱼ, minus the constraint that they sum to 1)
Under H₀: (r − 1) + (c − 1) unknown parameters (marginal probabilities, each summing to 1)
df = (rc − 1) − (r − 1) − (c − 1) = (r − 1)(c − 1)
The same df formula applies to the homogeneity test, but the justification differs: r(c − 1) parameters under H₀ ∪ H₁ minus (c − 1) under H₀ gives the same (r − 1)(c − 1).
These two tests produce identical calculations but differ in design and interpretation:
Homogeneity: separate random samples from r populations, each classified into c categories. Tests whether all r populations share the same distribution across categories.
Independence: one random sample from a single population, each unit classified on two variables. Tests whether the two variables are statistically independent.
Odds: p / (1 − p). Compares the probability of an event to the probability of non-event.
Odds ratio: OR = [p₁/(1−p₁)] / [p₂/(1−p₂)] = n₁₁ n₂₂ / (n₁₂ n₂₁)
OR = 1 means no association
OR < 1 means group 1 is less likely to have the trait
OR > 1 means group 1 is more likely to have the trait
Range: 0 to ∞
If any cell count is zero, add 0.5 to all cells before computing OR (but use original counts for Fisher's exact test)
Logit representation: logit(p) = log(p / (1−p)). The log odds ratio equals logit(p₁) − logit(p₂). This is the foundation of logistic regression.
Confidence interval for OR: Work on the log scale. L̂ = log(OR) has asymptotic variance σ²_L = 1/n₁₁ + 1/n₁₂ + 1/n₂₁ + 1/n₂₂. The CI for OR is then [exp(L̂ − z_(α/2) σ_L), exp(L̂ + z_(α/2) σ_L)].
Relative risk: RR = p₁ / p₂. More directly interpretable than OR. RR and OR are approximately equal when the outcome is rare (n₁₁ and n₂₁ small relative to n₁₂ and n₂₂).
RR can be computed from OR: RR = OR × (1 + n₂₁/n₂₂) / (1 + n₁₁/n₁₂)
⚠️ Overlapping individual confidence intervals for p₁ and p₂ do not imply that the CI for p₁ − p₂ contains zero. Always use a CI on the difference or ratio directly.
Definitions:
Prevalence = P(Event)
Sensitivity = P(Test + | Event Present) – the true positive rate
Specificity = P(Test − | Event Absent) – the true negative rate
Posterior probability via Bayes' theorem:
P(Present | Test +) = (Sensitivity × Prevalence) / [(Sensitivity × Prevalence) + (1 − Specificity)(1 − Prevalence)]
Critical insight from the simulation table: When prevalence is low (.001), even a test with 99% sensitivity and 95% specificity yields P(Present | Test +) = .0194. Specificity has far more impact on this probability than sensitivity does. Increasing specificity from .90 to .999 changes the posterior from .0098 to .4977. Increasing sensitivity from .90 to .99 only changes it from .0089 to .0098.
Improving the posterior with repeated testing: If k independent tests all come back positive:
P(Present | k positives) = (sensitivity)^k × prevalence / [(sensitivity)^k × prevalence + (1 − specificity)^k × (1 − prevalence)]
With sensitivity = .99, specificity = .95, and prevalence = .001: 1 test gives .0194, 3 tests give .886, 5 tests give .9997.
Used when the 2×2 table contains paired (not independent) observations, such as before/after measurements on the same subjects, or matched case-control pairs.
Setup:
Response 1 Yes | Response 1 No | |
|---|---|---|
Response 2 Yes | n₁₁ | n₁₂ |
Response 2 No | n₂₁ | n₂₂ |
n₁₂ and n₂₁ are the "discordant pairs" (the pairs that switched).
Hypotheses: H₀: p₁. = p.₁ (marginal proportions are equal), which is equivalent to H₀: p₁₂ = p₂₁.
Exact test: Under H₀, n₁₂ is Binomial(m, 0.5), where m = n₁₂ + n₂₁.
H₁: p₁. > p.₁ → p-value = P(B ≥ n₁₂) with B ~ Bin(m, .5)
H₁: p₁. < p.₁ → p-value = P(B ≤ n₁₂)
H₁: p₁. ≠ p.₁ → p-value = 2 × min(P(B ≥ n₁₂), P(B ≤ n₁₂))
Large-sample version (McNemar's Q):
Q_MN = (n₁₂ − n₂₁)² / (n₁₂ + n₂₁)
Under H₀, Q_MN is approximately chi-squared with df = 1. This is only valid for two-sided alternatives.
Approximate CI on p₁. − p.₁:
(p̂₁. − p̂.₁) ± z_(α/2) × (1/n) √[(n₁₂ + n₂₁) + (1/n)(n₁₂ − n₂₁)²]
Simpson's paradox: An association observed in aggregated data can reverse direction when the data is stratified by a confounding variable.
The classic example from the handout: overall, smokers had a lower death rate (24%) than non-smokers (31%). But within every age group, smokers had higher death rates. The explanation: younger women were more likely to smoke and far less likely to die in the 20-year follow-up, confounding the marginal association.
CMH test: Tests conditional independence of X and Y given a stratifying variable Z across K 2×2 tables.
H₀: pᵢⱼₖ = pᵢ.ₖ p.ⱼₖ for all k = 1, ..., K (X and Y are independent within each stratum)
CMH statistic:
CMH = [Σ_k (n₁₁ₖ − µ₁₁ₖ)]² / Σ_k Var(n₁₁ₖ)
where µ₁₁ₖ = n₁.ₖ n.₁ₖ / n..ₖ and Var(n₁₁ₖ) = n₁.ₖ n₂.ₖ n.₁ₖ n.₂ₖ / [n²..ₖ (n..ₖ − 1)]
Under H₀, CMH is approximately chi-squared with df = 1.
CMH pooled odds ratio estimate:
θ̂_CMH = Σ_k (n₁₁ₖ n₂₂ₖ / n..ₖ) / Σ_k (n₁₂ₖ n₂₁ₖ / n..ₖ)
Breslow-Day test: Tests whether the K odds ratios are homogeneous:
H₀: OR₁ = OR₂ = ... = OR_K
If the Breslow-Day test fails to reject, summarise with the common CMH odds ratio. If it rejects, analyse each stratum separately with Bonferroni correction (α_pc = α/K).
Recommended workflow:
Step 1: Breslow-Day test for homogeneity of odds ratios.
Step 2: If not rejected, use CMH test for conditional independence.
Step 3: If rejected, test each K table separately with α_pc = α/K.
R:
library(lawstat)
library(DescTools)
BreslowDayTest(y, OR=NA, correct=FALSE)
cmh.test(y)
Chi-squared test statistic:
χ² = ΣΣ (Oᵢⱼ − Êᵢⱼ)² / Êᵢⱼ
Homogeneity: df = t − 1 (for a 2 × t table)
Independence: df = (r − 1)(c − 1) (for an r × c table)
Fisher exact p-value (one-sided, H₁: p₁ > p₂):
p-value = Σ from k=x to min(n₁,m) of C(n₁,k)C(n₂,m−k) / C(n,m)
McNemar's Q: Q = (n₁₂ − n₂₁)² / (n₁₂ + n₂₁), df = 1
Bayes' theorem for screening:
P(Present | Test+) = (Sens × Prev) / (Sens × Prev + (1 − Spec)(1 − Prev))
⚠️ If any expected count < 1 or more than 20% of expected counts < 5, the chi-squared approximation is unreliable. Use Fisher's exact test.
⚠️ The chi-squared test of homogeneity and the test of independence produce identical calculations but answer different questions from different study designs. Know the distinction.
⚠️ Overlapping individual CIs for two proportions do not mean the difference is non-significant. Always use a CI on the difference directly.
⚠️ OR and RR are approximately equal only when the outcome is rare. For common outcomes, they can diverge substantially.
⚠️ Simpson's paradox occurs when a lurking variable confounds the marginal association. Always consider whether stratification by a potential confounder changes the conclusion.
⚠️ McNemar's test applies to matched pairs. The Pearson chi-squared test is not valid for paired data because the cell counts are correlated.
⚠️ When prevalence is very low, even highly sensitive and specific tests produce mostly false positives. Specificity matters more than sensitivity for the positive predictive value.
Q: In a 2×5 contingency table from independent samples, how many degrees of freedom does the chi-squared test of homogeneity have?
A: df = t − 1 = 5 − 1 = 4.
Q: You compute a chi-squared statistic of 16.96 with df = 4. What is the p-value in R?
A: 1 - pchisq(16.96, 4) ≈ 0.002.
Q: A 2×2 table has expected counts of 4.8, 32.2, 6.3, and 117.8. Can you use the chi-squared test?
A: No. One expected count (4.8) is less than 5, and since there are only 4 cells, 25% of them have expected counts below 5, exceeding the 20% threshold. Use Fisher's exact test.
Q: Drug PV has a success rate of 90.5% and Drug P has 66.7%. The odds ratio is 5.85. What does this mean?
A: The odds of success for Drug PV are about 5.85 times the odds of success for Drug P. This is a large effect size indicating substantially better outcomes with PV.
Q: A screening test has 99% sensitivity and 95% specificity. The disease prevalence is 0.1%. What is P(Disease | Positive)?
A: P = (.99 × .001) / (.99 × .001 + .05 × .999) = .00099 / (.00099 + .04995) ≈ .0194, or about 2%. Despite the impressive-sounding test characteristics, 98% of positives are false positives.
Q: In a McNemar test, n₁₂ = 21 and n₂₁ = 39. What is Q_MN and what does R report for the p-value?
A: Q_MN = (21 − 39)² / (21 + 39) = 324/60 = 5.4. p-value = P(χ²₁ ≥ 5.4) ≈ 0.020 (two-sided).
Q: In the smoking and survival example, the overall odds ratio of smokers to non-smokers for death is 0.658, but the CMH common odds ratio is 1.56. Explain the reversal.
A: This is Simpson's paradox. Younger women were more likely to smoke and far less likely to die. Age confounds the marginal association. Within each age group, smokers had higher odds of death. The CMH test, which conditions on age, correctly reveals OR = 1.56 > 1.
chi-squared test, Pearson chi-squared, test of homogeneity, test of independence, contingency table, expected counts, observed counts, Fisher exact test, hypergeometric distribution, odds ratio, log odds, logit, relative risk, prevalence ratio, risk ratio, sensitivity, specificity, positive predictive value, Bayes theorem, false positive rate, McNemar test, matched pairs, marginal homogeneity, discordant pairs, Cochran-Mantel-Haenszel test, CMH, conditional independence, stratified analysis, Breslow-Day test, common odds ratio, Simpson paradox, confounding variable, marginal analysis, conditional analysis, multiple comparisons, Bonferroni correction, product multinomial, multinomial sampling