Difficulty: Introductory to Intermediate | Prerequisites: Part 1 study notes (sample spaces, probability measures, inclusion-exclusion, complement rule, uniform distribution).
This second set of notes builds on the fundamentals. Conditional probability lets you update a probability when you learn something new. Independence tells you when one event has no bearing on another. Bayes theorem flips a conditional probability around, which turns out to be extraordinarily useful in practice. Expected value and variance give you a way to summarise an entire probability distribution with just two numbers. Together, these tools let you reason about uncertainty in a structured, quantitative way.
Conditional probability is the probability of one event given that another has occurred: divide the joint probability by the probability of the given event. Two events are independent when knowing one tells you nothing about the other. Bayes theorem lets you reverse the direction of a conditional probability using priors and likelihoods. Expected value is the probability-weighted average outcome; variance measures how spread out the outcomes are.
Conditional probability, p(E|F)
The probability that event E occurs given that event F has already occurred. Calculated as p(E ∩ F) / p(F). Think of it as zooming in on the part of the sample space where F is true, then asking how much of that part also satisfies E.
Independence
Two events E and F are independent when knowing that one occurred does not change the probability of the other. Formally, p(E ∩ F) = p(E) × p(F). In simple terms, they have nothing to do with each other.
Bayes theorem
A formula that reverses the direction of a conditional probability: p(A|B) = p(B|A) × p(A) / p(B). Think of it as updating your belief about A after observing evidence B.
Prior probability, p(A)
Your initial belief about how likely A is before seeing any new evidence.
Posterior probability, p(A|B)
Your updated belief about A after taking evidence B into account.
Likelihood, p(B|A)
How likely the evidence B is if A were true. This is the bridge between the prior and the posterior in Bayes theorem.
Expected value, E(X)
The probability-weighted average of all possible values of a random variable X. In simple terms, it is the long-run average you would see if you repeated the experiment many times.
Variance, V(X)
A measure of how spread out the values of X are around the expected value. Larger variance means more unpredictability.
Linearity of expectation
The principle that E(X + Y) = E(X) + E(Y), regardless of whether X and Y are independent. This is one of the most powerful and frequently used properties in probability.
p(E|F) = p(E ∩ F) / p(F). This is only defined when p(F) > 0.
Intuition: you restrict the sample space to only those outcomes where F occurred, then find the fraction of those that also satisfy E.
Worked example, 4-bit strings:
S = all 4-bit strings, |S| = 16.
F = the first bit is 0. There are 8 such strings, so p(F) = 8/16 = 1/2.
E = the string contains at least two consecutive 0s.
E ∩ F = strings starting with 0 that also have at least two consecutive 0s. These are: 0000, 0001, 0010, 0011, 0100. That gives p(E ∩ F) = 5/16.
p(E|F) = (5/16) / (8/16) = 5/8.
Events E and F are independent if and only if: p(E ∩ F) = p(E) × p(F).
Equivalent ways to state the same thing:
p(E|F) = p(E)
p(F|E) = p(F)
If any one of these three conditions holds, the other two follow automatically.
Example, card deck (without replacement):
Draw two cards from a standard 52-card deck without replacement.
Let A = "first card is a jack" and B = "second card is an 8."
p(A ∩ B) = (4/52) × (4/51) = 16/2652.
p(A) × p(B) = (4/52) × (4/52) = 16/2704.
These are not equal, so A and B are not independent. Removing the first card changes the deck, which changes the probability of the second draw.
Contrast with replacement: if you put the first card back before drawing the second, the draws become independent because the deck resets.
Bayes theorem lets you "flip" a conditional probability. If you know p(B|A), you can find p(A|B).
Formula: p(A|B) = p(B|A) × p(A) / p(B).
The three ingredients:
Prior, p(A): how likely A is before you see any evidence.
Likelihood, p(B|A): how likely the evidence B is if A is true.
Marginal, p(B): the overall probability of observing B, regardless of A.
Worked example, meningitis diagnosis:
p(M) = 0.0001 (meningitis is rare).
p(SN|M) = 0.8 (a patient with meningitis has a stiff neck 80% of the time).
p(SN) = 0.1 (10% of all patients present with a stiff neck, for any reason).
p(M|SN) = (0.8 × 0.0001) / 0.1 = 0.0008.
Even with a strong symptom, the posterior probability of meningitis is still very low because the prior is so small. This is the base-rate effect.
Practical applications:
Chip manufacturing: a defect test flags a chip. Bayes theorem tells you the probability that a flagged chip is truly defective, accounting for the test's false-positive rate.
Spam detection: given that an email contains certain keywords, Bayes theorem calculates the probability it is spam versus legitimate.
Expected value E(X) = the sum of each possible value times its probability:
E(X) = x₁ × p(X = x₁) + x₂ × p(X = x₂) + ... + xₙ × p(X = xₙ).
It is the long-run average, not necessarily a value X can take. (The expected value of a fair die roll is 3.5, which is not on any face.)
Example, fair die: E(X) = (1 + 2 + 3 + 4 + 5 + 6) / 6 = 3.5.
Linearity of expectation: E(X + Y) = E(X) + E(Y), always, whether or not X and Y are independent.
Example: the expected number of heads in 3 fair coin flips. Each flip has E = 0.5, so E(total heads) = 0.5 + 0.5 + 0.5 = 1.5.
Variance V(X) = E((X - E(X))²). It measures how far values typically sit from the mean.
A variance of 0 means X always takes the same value.
Higher variance means the outcomes are more spread out.
p(E|F) = \frac{p(E \cap F)}{p(F)}p(E \cap F) = p(E) \times p(F) \quad \text{(if E and F are independent)}p(A|B) = \frac{p(B|A) \, p(A)}{p(B)}E(X) = \sum_{i=1}^{n} x_i \, p(X = x_i)V(X) = E\bigl((X - E(X))^2\bigr)E(X + Y) = E(X) + E(Y)Bayes theorem is the engine behind spam filters, medical diagnostic tools and recommendation systems. Every time your email client sorts a message into spam, it is running a version of Bayes theorem on the words in the message. Expected value is how insurance companies price policies: they calculate the average payout per policyholder and charge accordingly. Variance tells a portfolio manager how volatile an investment is, which drives decisions about risk.
Students often confuse p(A|B) with p(B|A). These are almost never equal. Bayes theorem exists precisely because they differ.
Students frequently assume that "not independent" means "mutually exclusive" (or vice versa). Mutually exclusive events (A ∩ B = ∅) are never independent unless one has probability 0. These are separate concepts.
Treating the expected value as a guaranteed outcome. E(X) = 3.5 for a fair die does not mean you will ever roll a 3.5. It is a long-run average.
Forgetting the base rate when applying Bayes theorem. A highly accurate test applied to a rare condition will still produce mostly false positives. The prior matters enormously.
⚠️ Bayes theorem problems are among the most commonly tested items. You will be given a prior, a likelihood and a marginal (or enough to compute one) and asked for the posterior.
⚠️ Be ready to determine whether two events are independent by checking p(E ∩ F) against p(E) × p(F). If they are not equal, state clearly that the events are dependent.
⚠️ Expected value questions often use linearity of expectation. If the problem involves a sum of simpler random variables (number of heads in n flips, total of n dice), break it into parts.
⚠️ Conditional probability questions frequently involve enumerating a restricted sample space. Practice listing outcomes that satisfy both E and F before dividing.
True or false: p(A|B) = p(B|A). (False. They are almost never equal.)
Fill in the blank: two events are independent if p(E ∩ F) = ____. (p(E) × p(F))
True or false: the expected value of a fair six-sided die is 3. (False. It is 3.5.)
Fill in the blank: in Bayes theorem, p(A|B) = p(B|A) × p(A) / ____. (p(B))
True or false: linearity of expectation requires the random variables to be independent. (False. It holds regardless of independence.)
Q: A disease affects 1 in 10,000 people. A test for it is positive 99% of the time when the person has the disease and 5% of the time when they do not. If a person tests positive, what is the probability they have the disease?
A: Let D = disease, + = positive test. p(D) = 0.0001, p(+|D) = 0.99, p(+|no D) = 0.05. p(+) ≈ 0.99 × 0.0001 + 0.05 × 0.9999 ≈ 0.050094. p(D|+) = (0.99 × 0.0001) / 0.050094 ≈ 0.00198, roughly 0.2%. The base rate is so low that most positives are false positives.
Q: You flip a fair coin 4 times. What is the expected number of heads?
A: By linearity of expectation, each flip contributes E = 0.5. So E(total heads) = 4 × 0.5 = 2.
Q: Are drawing a heart and drawing a king from a full deck independent events?
A: p(heart) = 13/52 = 1/4. p(king) = 4/52 = 1/13. p(heart ∩ king) = 1/52. Check: (1/4) × (1/13) = 1/52. They are equal, so yes, these events are independent.
Q: A 4-bit string is generated uniformly at random with the first bit 0. What is the probability it contains at least two consecutive 0s?
A: Given the first bit is 0, there are 8 equally likely strings. Of these, 5 contain at least two consecutive 0s (0000, 0001, 0010, 0011, 0100). So p = 5/8.
Q: What is the variance of a random variable that always takes the value 7?
A: V(X) = 0. There is no spread around the mean when X is constant.
Conditional probability and independence are prerequisites for studying Markov chains, hidden Markov models and Bayesian networks. Bayes theorem leads directly into Bayesian inference, which underpins much of modern machine learning. Expected value and variance are the starting point for studying probability distributions (binomial, geometric, Poisson), the law of large numbers and the central limit theorem. Linearity of expectation is used throughout algorithm analysis, for instance in the average-case analysis of randomised quicksort.
Conditional probability, Bayes theorem, Bayes rule, prior probability, posterior probability, likelihood, independence, dependent events, mutually exclusive, expected value, expectation, mean, variance, spread, linearity of expectation, random variable, base rate, false positive, spam filter, Purdue CS, foundations of computer science, discrete probability, intro to probability