Source: Tanhane/Dunlop, Section 6.3, Chapter 7, Sections 9.1, 14.1
Tags: sign test, Wilcoxon signed-rank test, population median, population proportion, population standard deviation, chi-square test, bootstrap hypothesis test, nonparametric, distribution-free, binomial test, robust methods
When the population is non-normal (especially skewed or heavy-tailed), the t-test can give misleading results. The sign test and Wilcoxon signed-rank test offer robust alternatives for testing hypotheses about the median. For population proportions, exact binomial tests handle small samples, while a Z-based test works for large n. Variance tests use the chi-square distribution but are extremely sensitive to non-normality. Bootstrap methods provide a flexible, distribution-free way to test any parameter when classical assumptions fail.
Sign test
A nonparametric test for the population median. Counts how many observations exceed the hypothesised median and uses the binomial distribution. Works for any continuous distribution.
Wilcoxon signed-rank test
A nonparametric test for the population median (or mean, when the distribution is symmetric). Uses both the signs and magnitudes of deviations from μ̃₀ by ranking |Yᵢ − μ̃₀| and summing the ranks of positive deviations.
Distribution-free (at the null)
A property meaning the rejection region and p-value do not depend on the form of the population distribution when H₀ is true. Both the sign test and Wilcoxon signed-rank test have this property under H₀.
S₊ (sign test statistic)
The number of observations greater than μ̃₀. Under H₀: μ̃ = μ̃₀, S₊ ~ Bin(n, 0.5).
W₊ (Wilcoxon signed-rank statistic)
The sum of the ranks associated with positive deviations Xᵢ = Yᵢ − μ̃₀ > 0.
Chi-square test for σ
Test statistic TS = (n − 1)S²/σ₀², which follows χ²(n − 1) when data are normal and σ = σ₀. Extremely sensitive to non-normality.
Bootstrap test
A resampling-based approach that estimates the sampling distribution of the test statistic under H₀ by drawing bootstrap samples from transformed data where H₀ holds.
Let Y₁, ..., Yₙ be iid with continuous cdf F and median μ̃. If the distribution is symmetric, then μ̃ = μ, so testing the median is the same as testing the mean.
Procedure:
Let Xᵢ = Yᵢ − μ̃₀
S₊ = number of positive Xᵢ (equivalently, number of Yᵢ > μ̃₀)
Under H₀: μ̃ = μ̃₀, S₊ ~ Bin(n, 0.5)
If any Yᵢ = μ̃₀ exactly, discard those observations and reduce n to n* = n − k
The connection to the median: when μ̃ > μ̃₀, the probability p = P[Yᵢ > μ̃₀] > 1/2; when μ̃ = μ̃₀, p = 1/2; when μ̃ < μ̃₀, p < 1/2. So testing the median reduces to testing whether p = 1/2.
Right-sided: H₀: μ̃ ≤ μ̃₀ vs H₁: μ̃ > μ̃₀
Reject H₀ if S₊ ≥ B_{n,α} = qbinom(1 − α, n, 0.5)
p-value = 1 − pbinom(S₊ − 1, n, 0.5)
Left-sided: H₀: μ̃ ≥ μ̃₀ vs H₁: μ̃ < μ̃₀
Reject H₀ if S₊ ≤ B_{n,1−α} = qbinom(α, n, 0.5)
p-value = pbinom(S₊, n, 0.5)
Two-sided: H₀: μ̃ = μ̃₀ vs H₁: μ̃ ≠ μ̃₀
Let S_max = max(S₊, n − S₊)
Reject H₀ if S_max ≥ B_{n,α/2} = qbinom(1 − α/2, n, 0.5)
p-value = 2(1 − pbinom(S_max − 1, n, 0.5))
Properties:
Distribution-free when μ̃ = μ̃₀ (rejection region and p-value do not depend on the population pdf)
Not distribution-free under the alternative, because the power function depends on the population pdf
The exact level cannot always match the nominal α due to the discreteness of the binomial distribution
R implementation: binom.test(x, n, p=0.5, alternative="greater") where x = S₊ and n = sample size.
Requires the population distribution to be continuous and symmetric. Because of symmetry, the median equals the mean.
Procedure:
Compute Xᵢ = Yᵢ − μ̃₀
Discard any Xᵢ = 0; let n* be the number of nonzero values
Rank the absolute values |Xᵢ| from smallest to largest (average ranks for ties)
W₊ = sum of ranks where Xᵢ > 0
W₋ = sum of ranks where Xᵢ < 0 (when no ties, W₊ + W₋ = n(n + 1)/2)
Right-sided: H₀: μ̃ ≤ μ̃₀ vs H₁: μ̃ > μ̃₀
Reject H₀ if W₊ ≥ W_{n,α} = qsignrank(α, n, FALSE)
p-value = psignrank(W₊ − 1, n, FALSE)
Left-sided: H₀: μ̃ ≥ μ̃₀ vs H₁: μ̃ < μ̃₀
Reject H₀ if W₊ ≤ W_{n,1−α} = qsignrank(α, n, TRUE)
p-value = psignrank(W₊, n, TRUE)
Two-sided: H₀: μ̃ = μ̃₀ vs H₁: μ̃ ≠ μ̃₀
Let W_max = max(W₊, W₋)
Reject H₀ if W_max ≥ W_{n,α/2} = qsignrank(α/2, n, FALSE)
p-value = 2 · psignrank(W_max − 1, n, FALSE)
Large-sample approximation (asymptotic):
Under H₀, the standardised statistic converges to N(0, 1):
μ_W = n(n + 1)/4
σ_W = √(n(n + 1)(2n + 1)/24)
Z = (W₊ − μ_W) / σ_W
Approximate critical value: W_{n,α} ≈ μ_W + z_α · σ_W
R implementation: wilcox.test(x, y=rep(mu0, n), alternative="greater", paired=TRUE, conf.int=TRUE)
Note: paired=TRUE gives the signed-rank test; paired=FALSE gives the rank-sum test.
Feature | t-Test | Wilcoxon Signed-Rank | Sign Test |
|---|---|---|---|
Designed for | N(μ, σ²) data | Symmetric continuous | Any continuous |
Tests | Mean μ (= median) | Median μ̃ (= mean) | Median μ̃ |
Uses magnitude info | Yes | Yes (via ranks) | No (only signs) |
Simulation findings (from Randles and Wolfe):
Normal population: t-test is only slightly better than Wilcoxon; sign test is not competitive
Double exponential (heavy tails, symmetric): Wilcoxon is best for large shifts; sign test is best for small shifts; t-test is third
Cauchy (very heavy tails): t-test becomes very conservative (true size drops well below .05) and loses substantial power; sign test has greatest power, followed by Wilcoxon
General rule: Wilcoxon is the best all-round choice for symmetric populations; sign test is safest when symmetry is uncertain
Data is highly right-skewed (Shapiro-Wilk p-value = .000003). Testing H₀: μ̃ ≤ 5 vs H₁: μ̃ > 5 at α = .05.
Sign test: S₊ = 13 (out of 25 values above 5). p-value = 0.5. Fail to reject H₀.
Wilcoxon signed-rank test: W₊ = 193. p-value ≈ 0.21. Fail to reject H₀.
t-test (incorrectly applied): T = 1.995, p-value = 0.029. Would reject H₀.
The t-test gives a misleading rejection because the data are far from normal. This example demonstrates why checking assumptions matters.
Let Y = number of "successes" in n independent Bernoulli trials. Y ~ Bin(n, p) and p̂ = Y/n.
Small sample (min[np₀, n(1 − p₀)] < 5): use exact binomial
Right-sided: H₀: p ≤ p₀ vs H₁: p > p₀
Reject H₀ if Y ≥ qbinom(1 − α, n, p₀)
p-value = 1 − pbinom(Y − 1, n, p₀)
Left-sided: H₀: p ≥ p₀ vs H₁: p < p₀
Reject H₀ if Y ≤ qbinom(α, n, p₀)
p-value = pbinom(Y, n, p₀)
Two-sided: H₀: p = p₀ vs H₁: p ≠ p₀
Reject H₀ if Y ≤ qbinom(α/2, n, p₀) or Y ≥ qbinom(1 − α/2, n, p₀)
p-value = 2 · min[pbinom(Y, n, p₀), 1 − pbinom(Y − 1, n, p₀)]
Large sample (min[np₀, n(1 − p₀)] ≥ 5): use normal approximation
Test statistic: TS = (p̂ − p₀) / √(p₀(1 − p₀)/n)
Under H₀, TS is approximately N(0, 1).
Power function (right-sided, large sample):
γ(p₁) = 1 − Φ(z_α · √(p₀(1 − p₀)/(p₁(1 − p₁))) + √n(p₀ − p₁)/√(p₁(1 − p₁)))
Sample size for proportion test (one-sided):
n = ⌈(z_α · √(p₀(1 − p₀)) + z_β · √(p₁(1 − p₁))) / δ⌉²
where δ = |p₁ − p₀|. For two-sided, replace z_α with z_{α/2}.
Test H₀: p ≤ .10 vs H₁: p > .10, α = .05, want power ≥ .99 to detect p = .25.
n = [(1.645√(.1 × .9) + 2.326√(.25 × .75)) / .15]² = 100.09, so n ≥ 101.
Requires Y₁, ..., Yₙ iid from N(μ, σ²). The normality assumption is crucial here.
Test statistic: TS = (n − 1)S² / σ₀²
Under H₀ (σ = σ₀), TS ~ χ²(n − 1).
Right-sided: H₀: σ ≤ σ₀ vs H₁: σ > σ₀
Reject H₀ if TS ≥ χ²_{n−1, α} = qchisq(1 − α, n − 1)
p-value = 1 − pchisq(TS, n − 1)
Left-sided: H₀: σ ≥ σ₀ vs H₁: σ < σ₀
Reject H₀ if TS ≤ χ²_{n−1, 1−α} = qchisq(α, n − 1)
p-value = pchisq(TS, n − 1)
Two-sided: H₀: σ = σ₀ vs H₁: σ ≠ σ₀
Reject H₀ if TS ≤ χ²_{n−1, 1−α/2} or TS ≥ χ²_{n−1, α/2}
p-value = 2 · min[pchisq(TS, n − 1), 1 − pchisq(TS, n − 1)]
Power function (right-sided):
γ(σ₁) = 1 − pchisq((σ₀²/σ₁²) · qchisq(1 − α, n − 1), n − 1)
Note: The power curve for the two-sided test of σ is not symmetric about σ₀ (unlike the power curves for mean tests), because the chi-square distribution is not symmetric.
This test is far less robust to non-normality than the t-test for means.
The CLT does not rescue the chi-square test for σ even with large n
Skewed populations produce grossly inflated Type I error rates
Heavy-tailed symmetric populations also inflate the significance level
Simulation results (nominal α = .05, H₁: σ > σ₀):
Distribution | n = 10 | n = 20 | n = 50 |
|---|---|---|---|
Normal | .046 | .050 | .051 |
Uniform | .018 | .011 | .018 |
t(df=5) | .119 | .140 | .157 |
Gamma(1) | .202 | .213 | .220 |
Gamma(.1) | .213 | .578 | .528 |
For the highly skewed Gamma(.1) distribution, the true level reaches .578 with n = 20. This is catastrophic.
Bottom line: Always check normality before using the chi-square test for variance. If data are non-normal, use bootstrap methods instead.
Two approaches: CI-based and test-statistic-based.
Construct a bootstrap confidence interval and check whether θ₀ falls inside.
Two-sided (H₁: θ ≠ θ₀): Build a 100(1 − α)% bootstrap CI (θ̂_L, θ̂_U). Reject H₀ if θ₀ is outside the CI.
Right-sided (H₁: θ > θ₀): Build a 100(1 − α)% lower bound (θ̂_L, ∞). Reject H₀ if θ₀ ≤ θ̂_L.
Left-sided (H₁: θ < θ₀): Build a 100(1 − α)% upper bound (−∞, θ̂_U). Reject H₀ if θ₀ ≥ θ̂_U.
This method gives a binary decision (p-value ≤ α or p-value > α) rather than an exact p-value.
This method estimates the p-value directly.
Steps:
From the data X₁, ..., Xₙ, compute the observed test statistic T = t₀
Transform the data to Yᵢ = f(Xᵢ) so that the Yᵢ satisfy H₀
For testing μ: Yᵢ = Xᵢ − X̄ + μ₀ (shifts the sample mean to μ₀)
For testing σ: Yᵢ = σ₀ · Xᵢ / S_x (rescales to have sd = σ₀)
Draw B bootstrap samples of size n from Y₁, ..., Yₙ (sampling with replacement)
Compute t₁, t₂, ..., t*_B from each bootstrap sample
Estimate the p-value:
Two-sided: p ≈ (number of |t*ᵢ| ≥ |t₀|) / B
Right-sided: p ≈ (number of t*ᵢ ≥ t₀) / B
Left-sided: p ≈ (number of t*ᵢ ≤ t₀) / B
Data: 25 observations, heavily right-skewed (Shapiro-Wilk p = .0000000038). S = 129.36, testing H₀: σ ≤ 100 vs H₁: σ > 100.
Chi-square test (incorrectly applied): TS = (24)(129.36²)/(100²) = 40.16, p-value = .021, lower 95% bound = 105.02. Would reject H₀.
Bootstrap test: Transform data by Y = 100 · X/sd(X), then resample 9999 times.
Bootstrap p-value = .2022
Bootstrap lower 95% confidence bound = 84.15
The bootstrap correctly indicates insufficient evidence that σ > 100, whereas the chi-square test was misled by the non-normality.
Any level α test of H₀: θ = η can be inverted to produce a confidence interval:
C(y) = {η : H₀: θ = η is not rejected at level α}
Then C(Y) is a 100(1 − α)% confidence interval for θ.
Example for σ²: The set of σ₀ values not rejected by the two-sided chi-square test gives:
CI for σ² = ((n − 1)S² / χ²_{n−1, α/2}, (n − 1)S² / χ²_{n−1, 1−α/2})
This is exactly the standard chi-square confidence interval for σ².
Sign test statistic: S₊ = Σ I(Yᵢ > μ̃₀), distributed Bin(n, 0.5) under H₀
Wilcoxon large-sample approximation:
μ_W = n(n + 1)/4
σ_W = √(n(n + 1)(2n + 1)/24)
Proportion test statistic (large sample): TS = (p̂ − p₀) / √(p₀(1 − p₀)/n)
Proportion sample size: n = ⌈(z_α√(p₀(1 − p₀)) + z_β√(p₁(1 − p₁))) / δ⌉²
Chi-square test statistic: TS = (n − 1)S² / σ₀²
CI for σ² (from inverting chi-square test): ((n − 1)S²/χ²_{n−1,α/2}, (n − 1)S²/χ²_{n−1,1−α/2})
Bootstrap data transformation (testing μ): Yᵢ = Xᵢ − X̄ + μ₀
Bootstrap data transformation (testing σ): Yᵢ = σ₀ · Xᵢ / S_x
⚠️ The sign test requires only a continuous distribution. The Wilcoxon signed-rank test additionally requires symmetry. Know which assumption goes with which test.
⚠️ The sign test is distribution-free under H₀ but not under H₁ (power depends on the population pdf). Same for Wilcoxon.
⚠️ For normal data, the t-test is slightly better than Wilcoxon, which is much better than the sign test. For heavy-tailed or skewed data, the ranking reverses.
⚠️ The chi-square test for variance is the most non-robust classical procedure. Even moderate non-normality can destroy its validity, and increasing n does not fix it (unlike the t-test for means).
⚠️ Bootstrap tests require transforming the data so that H₀ is true before resampling. Forgetting this step produces meaningless p-values.
⚠️ The exact level of the sign test may not equal the nominal α due to the discreteness of the binomial distribution. Always check the true level.
⚠️ When data are skewed, the median is a more appropriate measure of centre than the mean. Use the sign test or Wilcoxon rather than the t-test.
⚠️ The recyclable waste example is a classic exam scenario: applying the t-test blindly to skewed data gives a significant result (p = .029), while appropriate nonparametric tests both fail to reject (p = .50 and .21). This illustrates the danger of ignoring assumptions.
Q: What is the test statistic for the sign test, and what is its distribution under H₀?
A: S₊ = the number of observations exceeding μ̃₀. Under H₀: μ̃ = μ̃₀, S₊ ~ Bin(n, 0.5).
Q: The Wilcoxon signed-rank test requires what additional assumption beyond what the sign test requires?
A: Symmetry of the population distribution.
Q: In the recyclable waste example, why did the t-test reject H₀ while the sign test and Wilcoxon did not?
A: The data were heavily right-skewed. A few very large values inflated the sample mean (9.528) well above the median (5.3), making the t-statistic appear significant. The nonparametric tests, which are based on ranks or signs rather than magnitudes, were not distorted by the outliers.
Q: Why is the chi-square test for σ not robust to non-normality, even with large samples?
A: Because the Central Limit Theorem ensures the sampling distribution of the mean becomes approximately normal with large n, but no analogous result holds for the sample variance. The distribution of (n − 1)S²/σ² can remain far from chi-square even with large samples if the population is non-normal.
Q: In a bootstrap test of H₀: μ ≤ μ₀, how do you transform the data before resampling?
A: Set Yᵢ = Xᵢ − X̄ + μ₀, which shifts the sample mean to μ₀ so that H₀ holds in the resampling population.
Q: For a large-sample test of H₀: p ≤ 0.3 vs H₁: p > 0.3 with n = 100 and Y = 38 successes, compute the test statistic and p-value.
A: TS = (0.38 − 0.3)/√(0.3 × 0.7/100) = 0.08/0.04583 = 1.746. p-value = 1 − Φ(1.746) ≈ 0.040.
sign test, Wilcoxon signed-rank test, Wilcoxon rank-sum test, nonparametric test, distribution-free, population median, robust methods, binomial test, population proportion, sample proportion, chi-square test, variance test, standard deviation test, bootstrap test, bootstrap p-value, bootstrap confidence interval, resampling, test inversion, power comparison, recyclable waste example, skewed data, heavy tails, R functions, binom.test, wilcox.test, pchisq, qchisq, psignrank, qsignrank