Difficulty: Intermediate | Prerequisites: Basic hypothesis testing (t-tests, p-values), introductory probability
This material sits in the second half of an introductory analytics or data science course, after you have covered basic parametric hypothesis testing. The core problem: standard tests like the t-test assume your data follows a known distribution (usually normal). When that assumption breaks down, or when your data is binary, ranked, or sparse, you need a different toolkit. This week introduces that toolkit (non-parametric tests and Bayesian models) and, just as importantly, teaches you how to choose the right test for a given situation. If you skipped or forgot the basics of hypothesis testing, go back and review p-values, null hypotheses, and the Student's t-test before diving in.
Non-parametric tests (McNemar's, Wilcoxon, Mann-Whitney) let you compare treatments or groups without assuming a specific distribution. Bayesian modelling uses prior knowledge and observed data together to update your beliefs about a parameter, and Empirical Bayes is a practical shortcut when you have population-level estimates but limited sample data. Knowing which test to pick depends on two questions: how many data sets, and what kind of metric (exact values, ranks, or binary outcomes).
Non-parametric test
A hypothesis test that does not assume the data follows a specific probability distribution. In simple terms, it is the test you reach for when you cannot assume normality or when your data is ordinal or binary.
McNemar's test
A binomial-based test for paired binary (yes/no) data that focuses only on the cases where two treatments disagree. Think of it as: "ignoring all the ties, did one treatment win more of the contested cases?"
Wilcoxon signed-rank test
A non-parametric test for numeric data that checks whether the median of a distribution differs from a specific value m, assuming only that the distribution is continuous and symmetric. In simple terms, it ranks the distances from m and asks whether the positive distances are systematically larger than the negative ones.
Mann-Whitney U test
A non-parametric test that compares two independent (unpaired) samples by ranking all observations together and checking whether one group's ranks are systematically higher. Think of it as the non-parametric equivalent of the two-sample t-test.
Parametric test
A test that uses the exact values of the data (means, standard deviations) and assumes a known underlying distribution. The Student's t-test is the classic example.
Bayes' theorem (Bayes' rule)
A formula for updating the probability of an event given new evidence: P(A|B) = P(B|A) * P(A) / P(B). In simple terms, it tells you how to revise your belief about something after seeing data.
Prior probability
Your belief about a parameter before seeing the data. In the medical-test example, the prior is the 1% disease prevalence in the population.
Posterior probability
Your updated belief after combining the prior with the observed data via Bayes' theorem.
Empirical Bayes
A practical approach where you estimate the prior distribution from historical or population-level data, then apply Bayes' theorem with limited new observations. Think of it as borrowing strength from the crowd to make better estimates for individuals.
Censored data
Data where the true value is only partially known, typically because observation stopped before the event occurred. Common in survival analysis.
Tags: McNemar test, paired binary, binomial test, treatment comparison, contingency table
McNemar's test applies when you have paired observations with binary outcomes (success/failure, yes/no) and you want to know whether one treatment outperforms the other. The key insight is that it only looks at the discordant pairs, the cases where the two treatments gave different results. Cases where both succeeded or both failed are uninformative for deciding which is better.
Setup: two treatments tested on the same 100 subjects. Treatment A succeeds in 61 cases, Treatment B in 68. Is B truly better, or is the gap random noise?
Scenario 1 – nearly identical results
32 cases: neither worked
61 cases: both A and B worked
7 cases: B worked, A did not
0 cases: A worked, B did not
All 7 discordant cases favour B. p = 0.02, so we accept that B is better. B succeeds everywhere A does, plus seven additional cases.
Scenario 2 – very different results
12 cases: neither worked
41 cases: both worked
27 cases: B worked, A did not
20 cases: A worked, B did not
47 discordant cases, split 27 to 20. p = 0.38, so we cannot conclude B is better. The treatments disagree on many subjects, but the imbalance is not large enough to be statistically significant.
The lesson: the same headline numbers (61% vs 68% success) can lead to opposite conclusions depending on how the disagreements are distributed.
Tags: Wilcoxon signed rank, median test, non-parametric, paired samples, symmetric distribution
The Wilcoxon signed-rank test is used for numeric data when you want to know whether the median of a distribution differs from a hypothesised value m. It assumes only that the distribution is continuous and symmetric (no normality required).
One-sample procedure:
Compute the absolute differences |y_i - m| for each observation.
Rank these absolute differences from smallest to largest.
Sum the ranks of all observations where y_i > m. Call this sum W.
Compute the p-value for W. If p is small, the true median is likely different from m.
The p-value calculation follows a logic similar to the normal distribution. The intuition: if the data is truly centred on m, then positive and negative deviations should carry roughly equal total rank. A lopsided W suggests the centre is not at m.
Paired-sample version:
When you have two matched samples (y_1, z_1), ..., (y_n, z_n) and want to test whether they share the same median, compute the absolute differences |y_i - z_i| and run the same rank procedure on those differences. This is the non-parametric counterpart to the paired t-test.
Tags: Mann-Whitney, U test, rank sum, independent samples, non-parametric two-sample
The Mann-Whitney test compares two independent (not paired) groups of numeric data. It is the non-parametric alternative to the independent two-sample t-test.
Procedure:
Pool all observations from both groups (y_1, ..., y_n and z_1, ..., z_m) and rank them together.
Compute the adjusted rank sums for each group:
U_y = (sum of ranks for y group) - n(n+1)/2
U_z = (sum of ranks for z group) - m(m+1)/2
Take U = min(U_y, U_z).
Find the significance of U using software or a statistical table.
The adjustment subtracts the minimum possible rank sum for each group (the sum it would get if all its members ranked lowest). A very small U means one group's observations tend to cluster at the low end of the combined ranking, suggesting the two populations differ.
Tags: test selection, parametric vs non-parametric, one-sample, two-sample, paired, unpaired, decision tree
Choosing the right test comes down to two questions asked in order.
Question 1: How many data sets?
One data set: you are comparing a sample's mean, median, or success rate against a fixed value m. Use a one-sample test.
Two data sets: you are comparing two groups against each other.
Are the observations paired (same subject measured under both conditions)? Use a two-sample paired test.
Are the observations independent? Use a two-sample unpaired test.
Question 2: What kind of metric are you computing?
Exact values (mean, standard deviation): use a parametric test (e.g. Student's t-test).
Ranks (median, ordinal position): use a non-parametric test (e.g. Wilcoxon, Mann-Whitney).
Binary outcomes (success/failure counts): use a binomial-based test (e.g. McNemar's test).
Quick-reference grid:
Parametric (exact values) | Non-Parametric (ranks) | Binomial (binary) | |
|---|---|---|---|
One-sample | One-sample t-test | Wilcoxon signed-rank | Binomial test |
Two-sample, paired | Paired t-test | Wilcoxon signed-rank (paired) | McNemar's test |
Two-sample, unpaired | Independent t-test | Mann-Whitney U | Chi-square / Fisher's exact |
Tags: Bayes theorem, Bayes rule, posterior, prior, likelihood, base rate, false positive, Empirical Bayes, basketball prediction
Bayes' theorem refresher
P(A|B) = P(B|A) * P(A) / P(B). Read it as: the probability of A given that B happened equals the likelihood of B under A, multiplied by the prior probability of A, divided by the overall probability of B.
Classic medical-test example
A disease test has a 98% true positive rate, an 8% false positive rate, and the disease prevalence in the population is 1%. If someone tests positive, what is the probability they are truly sick?
P(Disease | Positive) = (0.98 * 0.01) / 0.089 = 0.11
Only 11%. The base rate matters enormously. Because 99% of people are healthy, the 8% false-positive rate generates far more false alarms than the 98% detection rate generates true alarms. This is the single most counterintuitive result in introductory Bayesian reasoning.
Empirical Bayes – basketball example
When you know (or can estimate) the overall distribution of a parameter but have limited data for a specific case, Empirical Bayes lets you combine the two.
Setup: the point difference X between home and away teams is modelled as X ~ normal(m + h, sigma squared), where h is home-court advantage, m is the true strength difference, and sigma squared is random variation. The prior on m comes from historical data: m ~ normal(0, tau squared).
Estimated parameters:
h is roughly 4 (home-court advantage in points)
tau is roughly 6 (standard deviation of true team-strength differences)
sigma is roughly 11 (standard error from game-to-game randomness)
For a 20-point home victory, the Bayesian decomposition attributes roughly 4 points to home-court advantage, 12.5 points to random variation, and only 3.5 points to actual team-strength difference. The practical takeaway: a big home win tells you less about true team quality than you might expect, because most of the margin comes from noise and venue advantage. That team may well lose when visiting the opponent's court.
Bayes' theorem:
P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}Wilcoxon test statistic (one-sample):
W = sum of rank(y_i - m) for all y_i > m
Mann-Whitney U statistic:
U_y = (sum of ranks for y group) - n(n+1)/2
U_z = (sum of ranks for z group) - m(m+1)/2
U = min(U_y, U_z)
Bayesian posterior for the basketball model:
P(M = m | Y = y) = \frac{P(X = x | M = m) \cdot P(M = m)}{P(X = x)}where X ~ normal(m + h, sigma squared) and m ~ normal(0, tau squared)
McNemar's test is used in clinical trials to compare two treatments applied to the same patients, and in machine learning to compare two classifiers on the same test set.
The Wilcoxon and Mann-Whitney tests appear whenever data is skewed, ordinal, or has outliers that would distort a mean-based test, for example comparing customer satisfaction scores or reaction times.
Bayesian models are used in medical diagnostics (interpreting test results given disease prevalence), sports analytics (adjusting for home-court advantage), spam filtering, and any setting where you want to combine prior knowledge with limited new data.
Students often think McNemar's test uses all observations. It does not. Only the discordant pairs (where the two treatments disagree) enter the calculation.
Students often assume that a higher success rate automatically means a treatment is significantly better. The p-value depends on the pattern of disagreements, not just the totals.
Students often confuse the Wilcoxon signed-rank test with the Mann-Whitney test. Wilcoxon is for paired or one-sample data; Mann-Whitney is for two independent groups.
Students often think a positive medical test means you probably have the disease. In low-prevalence conditions, most positive results are false positives. The base rate (prior probability) matters more than the test's accuracy.
The test-selection framework (how many data sets, what metric) is a near-certainty on the exam. Be able to pick the right test for a described scenario.
The medical-test Bayes' theorem calculation (the 11% result) is a classic exam problem. Know the formula, know how to plug in prevalence, sensitivity, and false-positive rate.
Be able to explain why the same overall success rates can lead to different McNemar conclusions (Scenario 1 vs Scenario 2).
Understand the Empirical Bayes decomposition: given estimated parameters, break a result into home advantage, randomness, and true difference.
True or false: McNemar's test uses all paired observations, including those where both treatments agree. (False – it uses only discordant pairs.)
Fill in the blank: The Wilcoxon signed-rank test assumes the distribution is continuous and ________. (Symmetric.)
True or false: The Mann-Whitney test requires paired data. (False – it is for independent samples.)
True or false: If a medical test has 98% sensitivity and 8% false-positive rate, testing positive in a low-prevalence population means you almost certainly have the disease. (False – the posterior probability can be very low.)
Fill in the blank: In the basketball Empirical Bayes model, the largest contributor to a 20-point home win was ________, not team-strength difference. (Random variation.)
Q: You have tested the same 200 patients with Drug A and Drug B. Both drugs have binary outcomes (cured / not cured). Which statistical test should you use to determine whether one drug is significantly better?
A: McNemar's test, because the data is paired (same patients) and binary.
Q: A researcher collects reaction-time data from two independent groups and suspects the data is heavily skewed. She wants to know if the groups differ. Which test is appropriate?
A: The Mann-Whitney U test, because the samples are independent, the data is numeric, and the skewness makes a parametric test unreliable.
Q: A disease affects 2% of the population. A screening test has 95% sensitivity and 10% false-positive rate. If a patient tests positive, what is the approximate probability they have the disease?
A: P(D|T) = (0.95 * 0.02) / ((0.95 * 0.02) + (0.10 * 0.98)) = 0.019 / 0.117 = roughly 16%. Most positives are still false alarms.
Q: In the basketball Empirical Bayes model, why might a team that won by 20 at home still be expected to lose on the road?
A: Because most of the 20-point margin is attributed to home-court advantage (roughly 4 points) and random variation (roughly 12.5 points), leaving only about 3.5 points of true team-strength difference. That small edge can easily be overwhelmed when the opponent has home-court advantage instead.
Q: A medical study measures pain scores (ordinal, 1 to 10) for the same patients before and after treatment. Which test should you use?
A: The Wilcoxon signed-rank test (paired version), because the data is paired, ordinal/ranked, and you cannot assume normality.
Non-parametric tests connect back to the hypothesis-testing framework from earlier in the course (null hypothesis, p-value, significance level). The difference is the assumption set: parametric tests assume a distribution, non-parametric tests relax that assumption.
Bayesian modeling connects forward to the Empirical Bayes applications in later modules and to survival models (which also involve updating beliefs with observed data). It also ties into decision-making under uncertainty, a theme that reappears in game theory and competitive models.
The test-selection framework is a synthesising topic. It draws together every test you have learned so far and asks you to pick the right one for a described scenario, which is how exam questions are typically structured.
Non-parametric tests, distribution-free tests, McNemar's test, binomial test, discordant pairs, Wilcoxon signed-rank test, median test, Mann-Whitney U test, rank-sum test, Wilcoxon rank-sum, parametric vs non-parametric, one-sample test, two-sample test, paired test, unpaired test, independent samples, Bayes' theorem, Bayes' rule, conditional probability, prior probability, posterior probability, base rate fallacy, false positive paradox, Empirical Bayes, shrinkage, borrowing strength, home-court advantage, sports analytics, test selection, hypothesis testing, p-value, statistical significance