Difficulty: Introductory–Intermediate | Prerequisites: Basic set theory notation (unions, intersections, complements)
Probability theory gives you the rules for assigning numbers to uncertain events and then computing with those numbers. This part of the course covers the axioms of probability, counting techniques (permutations, combinations, partitions), conditional probability, independence, and Bayes' theorem. Master these and every later topic in the course, from distributions to inference, has a foundation to sit on.
Sample space (S)
The set of all possible outcomes of an experiment. Every probability question starts here.
In simple terms, it is the complete list of everything that could happen.
Event
A subset of the sample space. Any collection of outcomes you might care about.
Think of it as the specific thing you are asking "what is the probability of?"
Simple probability
When S is finite and every outcome is equally likely, P(A) = n(A) / n(S), the count of outcomes in A divided by the total count. This is the classical "equally likely outcomes" setup.
Complement (A^C)
Everything in S that is not in A. P(A^C) = 1 - P(A).
In simple terms, the probability that A does not happen.
Independence
Two events A and B are independent when knowing one occurred tells you nothing about the other. Formally: P(A ∩ B) = P(A) · P(B).
Conditional probability
The probability of A given that B has occurred: P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0.
Think of it as: shrink your universe to only the outcomes where B happened, then ask how much of that smaller universe A covers.
Law of total probability
If B₁, B₂, …, Bₙ partition S (they are mutually exclusive and cover everything), then P(A) = Σ P(A|Bᵢ) P(Bᵢ). This lets you break a hard probability into easier conditional pieces.
Bayes' theorem
A formula for flipping the direction of conditioning: P(A|B) = P(B|A) P(A) / P(B). Classical use: you know how likely the evidence is given the hypothesis, and you want the probability of the hypothesis given the evidence.
Permutation
An ordered arrangement of objects. Drawing k items from N without replacement where order matters: N! / (N-k)!.
Combination
An unordered selection. Drawing k items from N without replacement where order does not matter: C(N, k) = N! / ((N-k)! k!).
Partition (stars and bars)
Allocating N identical objects across k groups: C(N+k-1, k-1) possibilities. Also called "stars and bars" in combinatorics.
Every probability function P must satisfy these three rules:
(P1) Non-negativity. P(A) ≥ 0 for every event A ⊂ S.
(P2) Normalisation. P(S) = 1, the entire sample space has probability one.
(P3) Countable additivity. If A₁, A₂, … are mutually exclusive (disjoint), then P(A₁ ∪ A₂ ∪ …) = P(A₁) + P(A₂) + …
Everything else in probability follows from these three statements.
P(A^C) = 1 - P(A)
P(∅) = 0
If A ⊂ B, then P(A) ≤ P(B)
0 ≤ P(A) ≤ 1 for any event A
Addition rule: P(A ∪ B) = P(A) + P(B) - P(A ∩ B). You subtract the overlap so you do not count it twice.
When you sit down with a problem, work through these steps in order:
Define the sample space S and express the event of interest in terms of outcomes.
For simple (equally likely) probabilities, make sure the sample space is set up so that every outcome has the same chance.
If you are stuck, write out the outcomes in S explicitly. Seeing them listed often reveals the structure you need.
Counting rules let you figure out n(A) and n(S) so you can apply simple probability. The starting point is always a set of N distinct objects {X₁, …, X_N} from which you draw k items.
If an experiment can be decomposed into k stages and the number of outcomes at each stage does not depend on what happened at the other stages, then the total number of outcomes is m₁ × m₂ × … × mₖ.
This is the foundation. All the counting formulas below are special cases of it.
With replacement, order matters. Each draw has N choices, and draws are independent. Total: N^k.
Without replacement, order matters (ordered samples, permutations). First draw has N choices, second has N-1, and so on. Total: N! / (N-k)!. When k = N this is a full permutation: N!.
Without replacement, order does not matter (combinations). Same as ordered samples, but divide out the k! rearrangements of the same group. Total: C(N, k) = N! / ((N-k)! k!). This is the binomial coefficient, and it shows up constantly (e.g. counting the number of ways to get exactly x successes in N trials).
Distributing N identical objects across k distinct bins (e.g. five identical blue balls into four urns). Because the objects are identical, only the counts per bin matter, not which object goes where.
Total: C(N + k - 1, k - 1).
All of these counting rules trace back to the multiplication rule. Where the outcomes you care about can be reached by more than one ordered sequence (e.g. {A, B} is the same combination as {B, A}), you divide by the number of orderings that map to the same event.
A and B are independent if and only if P(A ∩ B) = P(A) · P(B).
An equivalent way to say it: P(A|B) = P(A). Learning that B occurred does not change the probability of A.
P(A|B) = P(A ∩ B) / P(B), defined whenever P(B) > 0.
Intuitively, you are restricting attention to the sub-universe where B has happened and asking what fraction of that sub-universe A occupies.
If B₁, …, Bₙ form a partition of S (mutually exclusive, exhaustive), then:
P(A) = P(A|B₁)P(B₁) + P(A|B₂)P(B₂) + … + P(A|Bₙ)P(Bₙ)
This is the tool you reach for when calculating P(A) directly is hard, but you can split the problem into cases. The classic example from lectures: computing an overall death rate by aggregating across different types of bypass surgery.
P(A|B) = P(B|A) · P(A) / P(B)
The denominator can be expanded via total probability:
P(A|B) = P(B|A) P(A) / [P(B|A) P(A) + P(B|A^C) P(A^C)]
Bayes' theorem lets you reverse the direction of conditioning. You know how likely the data (B) is under the hypothesis (A), and you want the probability of the hypothesis given the data.
Classic application: medical testing. You know the sensitivity P(positive test | disease) and specificity P(negative test | no disease), and you want P(disease | positive test). The answer depends heavily on the base rate P(disease) in the population.
Even a highly accurate test can produce a low posterior probability if the condition is rare. This is the single most common source of confusion in applied Bayes problems. If only 1 in 10,000 people have a disease, a test with 99% sensitivity and 99% specificity still gives a positive predictive value of roughly 1%.
P(A^C) = 1 - P(A)P(A \cup B) = P(A) + P(B) - P(A \cap B)P(A|B) = \frac{P(A \cap B)}{P(B)}P(A) = \sum_{i=1}^{n} P(A|B_i)\,P(B_i)P(A|B) = \frac{P(B|A)\,P(A)}{P(B|A)\,P(A) + P(B|A^C)\,P(A^C)}Counting formulas
\text{With replacement, order matters: } N^k\text{Without replacement, order matters: } \frac{N!}{(N-k)!}\text{Combinations: } \binom{N}{k} = \frac{N!}{(N-k)!\,k!}\text{Partitions (stars and bars): } \binom{N+k-1}{k-1}Confusing independent with mutually exclusive. If A and B are mutually exclusive (they cannot both happen), then P(A ∩ B) = 0. If they are also non-trivial, that means P(A ∩ B) ≠ P(A)P(B), so they are dependent, not independent. Mutually exclusive events with positive probability are never independent.
Forgetting to check that outcomes are equally likely. The simple probability formula P(A) = n(A)/n(S) only works when every outcome in S has the same probability. If you define S carelessly (e.g. for two dice, listing sums 2 through 12 as "outcomes"), the formula gives wrong answers because the outcomes are not equally likely.
Double-counting in the addition rule. P(A ∪ B) = P(A) + P(B) only if A and B are disjoint. Otherwise you must subtract P(A ∩ B). Forgetting the subtraction is a classic exam error.
Ignoring base rates in Bayes' theorem. Students often compute P(A|B) using only the likelihood P(B|A) and forget how much the prior P(A) matters. A rare disease with a good test still yields many false positives relative to true positives.
⚠️ You should know the three axioms of probability and all the properties listed above by heart.
⚠️ Know when to use each counting rule (with/without replacement, order matters/doesn't). The exam will not always tell you which one; you need to identify the scenario.
⚠️ Bayes' theorem and the law of total probability are heavily tested. Be ready to set up a problem from a word description, identify the prior and the likelihood, and compute.
⚠️ The professor notes that class examples serve as "models" for exam problems. Recognising the structure of a new problem by analogy to a familiar one is a key exam skill.
⚠️ If stuck on a counting problem, try writing out the sample space explicitly. It is slow but reliable.
True or false: If P(A) = 0.3, then P(A^C) = 0.7.
True or false: Mutually exclusive events with positive probability are independent.
Fill in the blank: P(A ∪ B) = P(A) + P(B) - ______.
True or false: In the formula P(A) = n(A)/n(S), the outcomes in S must be equally likely.
Fill in the blank: The number of ways to choose 3 items from 10 without replacement, order not mattering, is C(10, 3) = ______.
Q: A bag contains 5 red and 3 blue marbles. You draw 2 without replacement. What is the probability both are red?
A: The sample space for ordered draws has 8 × 7 = 56 outcomes. The number of ordered ways to draw 2 red from 5 is 5 × 4 = 20. So P(both red) = 20/56 = 5/14. Equivalently, (5/8)(4/7) = 5/14.
Q: A disease affects 1% of the population. A test has 95% sensitivity (true positive rate) and 90% specificity (true negative rate). If a person tests positive, what is the probability they have the disease?
A: Using Bayes' theorem, P(D|+) = P(+|D)P(D) / [P(+|D)P(D) + P(+|D^C)P(D^C)] = (0.95 × 0.01) / (0.95 × 0.01 + 0.10 × 0.99) = 0.0095 / 0.1085 ≈ 0.088, or about 8.8%. The low base rate means most positive tests are false positives.
Q: How many ways can you allocate 6 identical prizes among 4 students?
A: This is a partitions (stars and bars) problem. N = 6 objects, k = 4 bins. Total = C(6 + 4 - 1, 4 - 1) = C(9, 3) = 84.
Q: A and B are events with P(A) = 0.4, P(B) = 0.5, and P(A ∩ B) = 0.2. Are A and B independent?
A: Check whether P(A ∩ B) = P(A) × P(B). We have 0.4 × 0.5 = 0.2, and P(A ∩ B) = 0.2. They are equal, so yes, A and B are independent.
Q: You roll two fair dice. What is P(sum = 7)?
A: The sample space has 36 equally likely outcomes (each die shows 1 through 6). The pairs that sum to 7 are (1,6), (2,5), (3,4), (4,3), (5,2), (6,1), giving 6 outcomes. P(sum = 7) = 6/36 = 1/6.
Probability axioms and counting rules feed directly into the next major topic: random variables and their distributions. The binomial distribution, for instance, is derived by combining the multiplication rule for independent trials with the combinations formula.
Conditional probability and Bayes' theorem reappear throughout statistical inference, particularly in hypothesis testing (where you condition on the null being true) and in Bayesian statistics more broadly.
Related Terms / Search Tags
probability axioms, Kolmogorov axioms, sample space, event, simple probability, equally likely outcomes, complement rule, addition rule, inclusion-exclusion, counting rules, multiplication rule, permutation, combination, binomial coefficient, stars and bars, partition, conditional probability, independence, Bayes' theorem, Bayes' rule, law of total probability, base rate, prior probability, posterior probability, ECON-UA 20, analytical statistics, midterm 1 review