Analytical Statistics (T), NYU | Chapter 1, Sections 1.3–1.5 | Source: Hogg & Tanis, Probability and Statistical Inference, 9th ed.
Difficulty: Intermediate
Prerequisites: Probability Basics and Set Theory notes (Sections 1.1–1.1.3) and Methods of Enumeration notes (Section 1.2). You need the probability axioms, set operations and counting methods before this material makes sense.
Conditional probability answers the question "what is the probability of A, given that B has occurred?" Independence tells you when knowing B changes nothing about A. Bayes' theorem lets you reverse the direction of conditioning: if you know P(evidence | hypothesis), you can find P(hypothesis | evidence). These three ideas are the bridge between the abstract axioms of probability and the practical statistical reasoning (estimation, hypothesis testing) that occupies the rest of the course. Almost every real-world probability problem involves at least one of them.
Conditional probability is P(A|B) = P(A ∩ B) / P(B). Two events are independent when P(A ∩ B) = P(A)P(B), which means conditioning on one does not change the probability of the other. Bayes' theorem flips the conditioning: P(B|A) = P(A|B)P(B) / P(A), where P(A) is expanded via the law of total probability. The Monty Hall problem is a famous application showing that switching doors doubles your chance of winning.
Conditional probability, P(A|B)
The probability of event A given that event B has occurred: P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0. Think of it as restricting your attention to the outcomes inside B and asking how much of B is also inside A.
Multiplication rule for P(A ∩ B)
P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A). In simple terms, the joint probability of A and B equals the conditional probability of one given the other, multiplied by the probability of the other.
Independent events
Two events A and B are independent (statistically independent) when P(A ∩ B) = P(A)P(B). Equivalently, P(A|B) = P(A) and P(B|A) = P(B). Knowing one occurred tells you nothing about whether the other occurred.
Dependent events
Events that are not independent. Knowing one occurred changes the probability of the other.
Pairwise independence
Events A, B, C are pairwise independent when every pair is independent: P(A ∩ B) = P(A)P(B), P(A ∩ C) = P(A)P(C), P(B ∩ C) = P(B)P(C).
Mutual independence
Events A, B, C are mutually independent when they are pairwise independent and P(A ∩ B ∩ C) = P(A)P(B)P(C). Pairwise independence alone is not sufficient.
Partition
A collection of events B₁, ..., Bₘ that are mutually exclusive and exhaustive: they cover the entire sample space with no overlap.
Law of total probability
For a partition B₁, ..., Bₘ: P(A) = Σ P(A|Bᵢ)P(Bᵢ). In simple terms, break A into pieces by conditioning on each member of the partition, then add.
Bayes' theorem
P(Bₖ|A) = P(A|Bₖ)P(Bₖ) / Σ P(A|Bᵢ)P(Bᵢ). It reverses the direction of conditioning.
Prior probability
P(B) before observing the evidence A. "What you believed before the data came in."
Posterior probability
P(B|A) after observing the evidence A. "What you believe after updating with the data."
P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0.
Visually, you are shrinking the sample space from S down to B and asking what fraction of B is also in A.
The conditional probability function P(·|B) satisfies the same axioms as any probability function: P(A|B) ≥ 0, P(B|B) = 1, and additivity for mutually exclusive events.
Example (HIV test): out of 1,000,000 people, 5,000 have the virus (B), 4,885 of whom test positive (A). P(A|B) = 4,885/5,000 = 0.977. Among the 78,515 who test positive, only 4,885 have the virus: P(B|A) = 4,885/78,515 = 0.062. These two numbers are very different.
Rearranging the definition: P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A).
This is useful whenever the conditional probability is easier to find than the joint probability.
Example (cards): probability that the 3rd spade appears on the 6th draw. Let A = exactly 2 spades in the first 5 draws, B = 6th draw is a spade. P(A) = [₁₃C₂ x ₃₉C₃] / ₅₂C₅ = 0.274. Given A, there are 11 spades left among 47 cards, so P(B|A) = 11/47 = 0.234. Then P(A ∩ B) = 0.274 x 0.234 = 0.064.
A and B are independent when P(A ∩ B) = P(A)P(B).
Equivalently, P(A|B) = P(A) and P(B|A) = P(B). Conditioning on one event does not change the other's probability.
Drawing with replacement produces independent draws. Drawing without replacement produces dependent draws.
Example (dependent): a red die and a white die. A = {5 on red}, B = {sum is 11}. P(A) = 1/6, P(B) = 2/36, P(A ∩ B) = 1/36. P(A)P(B) = 2/216 ≠ 1/36, so A and B are dependent.
Example (independent): same dice. C = {4 on red}, D = {sum is odd}. P(C) = 1/6, P(D) = 1/2, P(C ∩ D) = 3/36 = 1/12 = P(C)P(D). So C and D are independent, even though knowing C changes which white-die outcomes make the sum odd.
If A and B are mutually exclusive with P(A) > 0 and P(B) > 0, then P(A|B) = 0 ≠ P(A), so they are dependent.
If A and B are independent with P(A) > 0, then P(A|B) = P(A) > 0, so A can occur when B occurs, meaning they are not mutually exclusive.
In short: mutually exclusive events (with positive probability) are always dependent, and independent events (with positive probability) are never mutually exclusive.
If A and B are independent, so are: A and B', A' and B, A' and B'.
Pairwise independence: every pair from A, B, C is independent.
Mutual independence: pairwise independence plus P(A ∩ B ∩ C) = P(A)P(B)P(C). Pairwise independence does not guarantee mutual independence.
P(B|A) = P(A|B)P(B) / [P(A|B)P(B) + P(A|B')P(B')]
The denominator is the law of total probability for the partition {B, B'}.
Example (short hair): P(W) = 0.75 (prior: conversation partner is a woman). P(S|W) = 0.1, P(S|W') = 0.9. Then P(W|S) = (0.1 x 0.75) / (0.1 x 0.75 + 0.9 x 0.25) = 0.075 / 0.300 = 0.25. The prior of 0.75 drops to a posterior of 0.25 after observing short hair.
For a partition B₁, ..., Bₘ:
P(Bₖ|A) = P(A|Bₖ)P(Bₖ) / Σᵢ P(A|Bᵢ)P(Bᵢ)
This is the same logic scaled up. The denominator is the law of total probability over the whole partition.
You pick a door (say No. 1). The host, who knows where the car is, opens another door (say No. 3) to reveal a goat, then offers you the chance to switch to No. 2.
Switching wins with probability 2/3; staying wins with probability 1/3.
Intuition: when you first picked, you had a 1/3 chance of being right. The host's reveal does not change your original pick's probability, but it concentrates the remaining 2/3 on the one unopened door. Bayes' theorem confirms this formally.
Conditional probability:
P(A|B) = P(A ∩ B) / P(B)
Multiplication rule:
P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Independence test:
A and B are independent iff P(A ∩ B) = P(A)P(B)
Law of total probability (partition B₁, ..., Bₘ):
P(A) = Σᵢ P(A|Bᵢ)P(Bᵢ)
Bayes' theorem (two-event):
P(B|A) = P(A|B)P(B) / [P(A|B)P(B) + P(A|B')P(B')]
Bayes' theorem (general partition):
P(Bₖ|A) = P(A|Bₖ)P(Bₖ) / Σᵢ P(A|Bᵢ)P(Bᵢ)
Mutual independence (three events):
P(A ∩ B) = P(A)P(B), P(A ∩ C) = P(A)P(C), P(B ∩ C) = P(B)P(C), and P(A ∩ B ∩ C) = P(A)P(B)P(C)
Bayes' theorem is the engine behind medical diagnostic reasoning: a test's sensitivity P(positive|disease) and specificity P(negative|no disease) are known from clinical trials, but the patient wants P(disease|positive), which depends on the prevalence (the prior). This is why a positive screening test for a rare disease often has a low posterior probability of the disease being present. The same logic drives spam filters, fraud detection and machine-learning classifiers.
Students often confuse P(A|B) with P(B|A). The HIV-test example makes this vivid: P(positive|virus) = 0.977 but P(virus|positive) = 0.062. Swapping these is sometimes called the "prosecutor's fallacy."
Independence and mutual exclusivity are frequently confused. Remember: mutually exclusive events with positive probability are dependent, and independent events with positive probability are never mutually exclusive.
In Bayes' theorem problems, students sometimes forget the denominator. The denominator is the law of total probability and serves as a normalisation constant, so the posterior adds up correctly.
Pairwise independence does not imply mutual independence. When three events are involved, you must check the three-way product condition as well.
⚠️ Know the definition of conditional probability and be able to derive the multiplication rule from it.
⚠️ Be able to test whether two events are independent by checking P(A ∩ B) = P(A)P(B). Do not rely on intuition alone.
⚠️ Bayes' theorem problems are high-frequency exam questions. Practise setting up the prior, the likelihoods and the law of total probability in the denominator.
⚠️ The Monty Hall problem is a classic conceptual question. Be prepared to explain why switching is better, using conditional probability.
⚠️ Understand the distinction between pairwise and mutual independence for three or more events.
True or false: P(A|B) = P(B|A) whenever P(A) = P(B).
Fill in the blank: if A and B are independent, P(A ∩ B) = ___.
True or false: mutually exclusive events with positive probability are independent.
Fill in the blank: in the Monty Hall problem, the probability of winning by switching is ___.
True or false: the law of total probability requires the conditioning events to form a partition.
Q: From the HIV-test table (1,000,000 people, 5,000 with the virus, 4,885 true positives, 73,630 false positives), what is P(virus | positive)?
A: P(virus | positive) = 4,885 / 78,515 ≈ 0.062. Despite the test being highly sensitive, the low prevalence means most positives are false positives.
Q: A boy transfers a random marble from his left pocket (5 blue, 4 white) to his right pocket (4 blue, 5 white), then draws from the right pocket. What is the probability of drawing blue?
A: If a blue marble was transferred (probability 5/9), the right pocket has 5 blue and 5 white, so P(blue | blue transferred) = 5/10. If a white marble was transferred (probability 4/9), the right pocket has 4 blue and 6 white, so P(blue | white transferred) = 4/10. By the law of total probability: P(blue) = (5/9)(5/10) + (4/9)(4/10) = 25/90 + 16/90 = 41/90 ≈ 0.456.
Q: Two cards are drawn from a standard deck without replacement. Let A = first card is an ace, B = second card is an ace. Are A and B independent?
A: No. P(B|A) = 3/51 while P(B|A') = 4/51, so the first draw affects the second. They are dependent.
Q: A red die and a white die are rolled. C = {4 on red die}, D = {sum is odd}. Verify independence.
A: P(C) = 1/6, P(D) = 18/36 = 1/2, P(C ∩ D) = 3/36 = 1/12. P(C)P(D) = (1/6)(1/2) = 1/12 = P(C ∩ D). Independent.
Q: In the Monty Hall problem, what is the probability of winning if you switch? Explain briefly.
A: 2/3. Your initial pick is correct with probability 1/3, so the car is behind the other door with probability 2/3. The host's reveal gives you no new information about your own door, so switching wins 2/3 of the time.
Conditional probability and Bayes' theorem are the foundation of Bayesian statistics, which you may encounter later in the course or in more advanced classes. The multiplication rule is essential for deriving the probability mass functions of discrete distributions (binomial, geometric, negative binomial). Independence is the key assumption behind random sampling, which underpins virtually all of statistical inference. The law of total probability appears again whenever you condition on a latent variable or a hidden state.
Conditional probability, P(A given B), conditioning, multiplication rule, joint probability, independent events, statistical independence, dependent events, pairwise independence, mutual independence, Bayes theorem, Bayes rule, prior probability, posterior probability, law of total probability, partition, mutually exclusive and exhaustive, Monty Hall problem, prosecutor's fallacy, base rate neglect, sensitivity, specificity, false positive, Hogg and Tanis Chapter 1, Analytical Statistics NYU