Difficulty: Intermediate | Prerequisites: Probability and Random Variables notes
Hypothesis testing is how statisticians move from sample data to conclusions about a population. You state a claim (the null hypothesis), collect data, compute a test statistic, and decide whether the data provide enough evidence to reject that claim. This section ties together the test-statistic formulas, error types, p-values, power and confidence intervals that make up the largest exam topic.
A Type I error is rejecting a true null; a Type II error is failing to reject a false null. The p-value is the smallest significance level at which you would reject. Power = 1 - beta. Use Z when the variance is known (or n is large), t when it is unknown and n is small, chi-square for one variance, F for two variances. Increasing sample size (holding alpha constant) reduces beta and raises power.
Null hypothesis (H0)
The default claim about a population parameter, assumed true until the data provide sufficient evidence against it. Think of it as the "nothing special is happening" baseline.
Alternative hypothesis (H1 or Ha)
The claim you are trying to find evidence for. It can be one-sided (mu < value or mu > value) or two-sided (mu is not equal to value).
Type I error (false positive)
Rejecting the null hypothesis when it is in fact true. The probability of a Type I error is alpha, the significance level.
Type II error (false negative)
Failing to reject the null hypothesis when it is in fact false. The probability of a Type II error is beta.
Significance level (alpha)
The maximum probability of a Type I error that you are willing to tolerate. Common values: 0.05, 0.01, 0.10. In simple terms, it is the threshold for "unlikely enough to reject."
p-value
The smallest level of significance at which the null hypothesis can be rejected given the observed data. If p-value <= alpha, reject H0. The p-value is not the probability that H0 is true.
Power of a test
The probability of correctly rejecting a false null hypothesis: Power = P(reject H0 | H1 is true) = 1 - beta. Higher power means the test is better at detecting a real effect.
Z-test statistic
Used when the population variance is known or the sample size is large. Z0 = (x-bar - mu0) / (sigma / sqrt(n)).
t-test statistic
Used when the population variance is unknown and n is small. T0 = (x-bar - mu0) / (S / sqrt(n)), where S is the sample standard deviation. Follows a t-distribution with n-1 degrees of freedom.
Chi-square test statistic
Used to test hypotheses about the variance of a single normal population. chi-square0 = (n-1)S^2 / sigma0^2.
F-test statistic
Used to compare two variances from independent normal populations. F0 = S1^2 / S2^2.
Confidence interval
A range of plausible values for a population parameter, constructed from sample data and a chosen confidence level (e.g. 95%). In simple terms, it brackets where you believe the true value lies.
Every hypothesis test follows the same five steps:
State H0 and H1.
Choose the significance level alpha.
Compute the test statistic from the sample data.
Determine the critical value (or compute the p-value).
Compare and conclude: reject H0 if the test statistic falls in the rejection region (or if p-value <= alpha).
Testing a population mean, variance known (or large n): Z-test. Z0 = (x-bar - mu0) / (sigma / sqrt(n)).
Testing a population mean, variance unknown, small n: t-test. T0 = (x-bar - mu0) / (S / sqrt(n)).
Testing a single population variance: chi-square test. chi-square0 = (n-1)S^2 / sigma0^2.
Comparing two population variances: F-test. F0 = S1^2 / S2^2.
Two samples do not need to be the same size for a two-sample t-test; the formula accommodates different n1 and n2.
Claim: the average weight of a component is 500g. A sample of 25 gives x-bar = 495g, with known sigma = 10g. Test at alpha = 0.05 whether the mean is less than 500g.
H0: mu = 500. H1: mu < 500 (left-tailed).
Z0 = (495 - 500) / (10 / sqrt(25)) = -5 / 2 = -2.5.
Critical value for a left-tailed test at alpha = 0.05: z = -1.645.
Since -2.5 < -1.645, the test statistic falls in the rejection region. Reject H0. There is sufficient evidence at the 5% level that the mean weight is less than 500g.
Alpha (significance level) is the probability of a Type I error.
Beta is the probability of a Type II error.
Power = 1 - beta.
There is an inverse relationship: if you decrease alpha (make the test stricter), beta typically increases (and power decreases) because the rejection region shrinks.
Increasing the sample size n (while holding alpha constant) decreases the standard error, which narrows both the null and alternative sampling distributions. This makes it easier to distinguish the two, so beta decreases and power increases.
In short: larger samples give you more power to detect real effects without changing your tolerance for false positives.
A confidence interval estimates the population parameter within a margin of error. For a population mean with known variance: x-bar +/- z(alpha/2) times sigma/sqrt(n). The interval does not say there is a 95% probability the parameter lies inside it (that is a common misstatement). It means that 95% of intervals constructed this way across repeated samples would contain the true parameter.
Students often think alpha is the probability of a Type II error. It is the probability of a Type I error (rejecting a true H0). Beta is the Type II error probability.
Students sometimes believe the p-value is the probability that H0 is true. It is not. It is the probability of observing data at least as extreme as the sample, assuming H0 is true.
Students assume that decreasing alpha increases power. The opposite is true: a stricter alpha makes it harder to reject, which increases beta and decreases power (all else equal).
Students think two samples must be the same size for a two-sample t-test. They do not; the formula handles different n1 and n2.
Students confuse the chi-square and F tests. The chi-square tests one population's variance; the F-test compares two populations' variances.
Type I vs Type II error definitions are tested in both multiple-choice and true/false questions. Know them cold.
The p-value definition ("smallest significance level at which you would reject") is a favourite multiple-choice distractor.
Power = P(reject H0 | H1 is true) is tested directly. Students who write P(reject H0 | H0 is true) are describing alpha, not power.
A full hypothesis test (state hypotheses, compute the test statistic, find the critical value, conclude) appears as a short-answer question worth 5 marks. Practise the Z-test version.
The long-answer section asks you to discuss the alpha-beta-power relationship and the effect of sample size. Prepare a structured answer.
True or false: alpha is the probability of making a Type II error. (False, it is Type I)
True or false: if p-value <= alpha, we reject H0. (True)
True or false: decreasing alpha increases power. (False, it decreases power)
True or false: two samples must have the same size for a two-sample t-test. (False)
Fill in the blank: the chi-square distribution is used to test the ____ of a single normal population. (Variance)
Q: A Type I error occurs when we ____.
A: Reject the null hypothesis when it is true.
Q: The p-value of a statistical test represents ____.
A: The smallest level of significance at which the null hypothesis can be rejected given the observed data.
Q: The power of a test is defined as ____.
A: P(reject H0 | H1 is true) = 1 - beta.
Q: Which test statistic is appropriate for a one-sample test on the mean when the variance is unknown and n is small?
A: The t-statistic, T0 = (x-bar - mu0) / (S / sqrt(n)).
Q: Discuss the relationship between alpha, beta and the power of a test. How does increasing sample size affect these, holding alpha constant?
A: Alpha is the probability of a Type I error; beta is the probability of a Type II error; power = 1 - beta. There is an inverse relationship between alpha and beta: lowering alpha raises beta and reduces power. When sample size increases (with alpha fixed), the standard error decreases, the sampling distributions narrow, and it becomes easier to detect a true difference. This reduces beta and increases power.
Q: An engineer tests whether mean weight is below 500g. Sample: n = 25, x-bar = 495g, sigma = 10g, alpha = 0.05. What is your conclusion?
A: H0: mu = 500, H1: mu < 500. Z0 = (495 - 500) / (10/5) = -2.5. Critical value for left-tailed at 0.05: -1.645. Since -2.5 < -1.645, reject H0. Sufficient evidence that the mean is less than 500g.
Hypothesis testing builds on the probability and distribution theory from the previous notes. Confidence intervals are the two-sided counterpart of two-tailed tests. The choice of test statistic depends on what you know about the population (variance known or not, one sample or two), which connects back to the parameter-vs-statistic distinction. Power analysis feeds into study design: before collecting data, researchers use power calculations to determine the sample size needed to detect a meaningful effect.
Related Terms / Search Tags
hypothesis testing, null hypothesis, alternative hypothesis, Type I error, Type II error, significance level, alpha, beta, power of a test, p-value, Z-test, t-test, chi-square test, F-test, one-tailed test, two-tailed test, critical value, rejection region, confidence interval, margin of error, sample size, standard error, degrees of freedom, NYU analytical statistics midterm