Analytical Statistics (T), NYU | Chapter 1, Sections 1.1–1.1.3 | Source: Hogg & Tanis, Probability and Statistical Inference, 9th ed.
Difficulty: Introductory
Prerequisites: Basic algebra and comfort with set notation. No prior statistics required.
This is the foundation of everything else in Analytical Statistics. Probability gives you the language and the rules for reasoning about uncertain outcomes, and set theory gives you the notation to express events precisely. If you can work confidently with sample spaces, events, complements, unions and intersections, and you know the three axioms of probability, every later topic (distributions, inference, hypothesis testing) will click far more quickly. You should already be comfortable with basic algebra; no prior statistics knowledge is assumed.
Probability is a function that assigns a number between 0 and 1 to every event in a sample space, following three axioms (non-negativity, total probability equals 1, additivity for mutually exclusive events). Events are subsets of the sample space, and you manipulate them with unions, intersections and complements, governed by rules like DeMorgan's laws. From these basics you can derive every result in this section, including the complement rule P(A) = 1 - P(A') and the inclusion-exclusion formula P(A ∪ B) = P(A) + P(B) - P(A ∩ B).
Random experiment
A process whose outcome cannot be predicted with certainty beforehand, even if the set of possible outcomes is known. Think of it as any situation where you know what could happen but not what will happen.
Outcome space (sample space), S
The set of all possible outcomes of a random experiment. In simple terms, it is the complete list of everything that could result from running the experiment.
Event
Any subset of the sample space. An event A "occurs" when the outcome of the experiment falls inside A. Think of it as a question you can answer yes or no after observing the result.
Complement, A' (also written A^C)
The set of all outcomes in S that are not in A. In simple terms, the complement is "everything except A."
Union, A ∪ B
The set of outcomes that belong to A, or to B, or to both. Think of it as "at least one of A or B happens."
Intersection, A ∩ B
The set of outcomes that belong to both A and B simultaneously. In simple terms, "both A and B happen."
Empty set, ∅
A set with no elements. It represents an impossible event.
Mutually exclusive events
Events A₁, A₂, ..., Aₖ are mutually exclusive when no two of them can occur at the same time: Aᵢ ∩ Aⱼ = ∅ for all i ≠ j. Think of it as "if one happens, the others cannot."
Exhaustive events
Events A₁, A₂, ..., Aₖ are exhaustive when their union covers the entire sample space: A₁ ∪ A₂ ∪ ... ∪ Aₖ = S. In simple terms, "at least one of them must happen."
Probability function, P
A function that assigns a number P(A) to each event A in S, satisfying three axioms: non-negativity, normalisation, and countable additivity.
Relative frequency
The ratio of the number of times an outcome occurs to the total number of trials. This is the most common intuitive interpretation of probability: what the proportion approaches as you repeat the experiment many times.
Random variable
A variable that takes on different numerical values determined by the outcome of a random experiment. Think of it as a rule that turns each outcome into a number.
Distribution (of a random variable)
The complete description of which values a random variable can take and how probable each value is.
A random experiment is any process with more than one possible outcome where the result cannot be predicted with certainty.
Example: a consumer choosing among Carlsberg, Heineken and Budweiser. The sample space is S = {Carlsberg, Heineken, Budweiser}.
Example: rolling a die. The sample space is S = {1, 2, 3, 4, 5, 6}.
Randomness can be inherent (a die roll) or can come from the observer's ignorance (the economist who cannot predict which beer the consumer will pick, even if the consumer already knows).
Once you know all the possible outcomes and the probability of each one, you have the distribution of the random variable.
The relative frequency of an outcome x is: (number of trials with outcome x) / (total number of trials).
Probability is understood as what the relative frequency converges to as the number of trials grows very large.
For a small number of trials, relative frequencies can differ substantially from the true probabilities. Roll a fair die six times and you will not typically get each face exactly once.
The frequency interpretation is natural for repeatable experiments (dice, coins) but awkward for one-off events (the GDP of Korea in 2015). Probability can still be assigned to such events, though the "repeat many times" intuition does not apply directly.
An event is any subset A of the sample space S. When the experiment's outcome lands in A, we say A has occurred.
Membership: x ∈ A means x is in A; x ∉ A means x is not in A.
Subset: A ⊂ B means every element of A is also in B, but B may contain elements not in A.
Equality: A = B when A ⊂ B and B ⊂ A.
Complement: A' = {x ∈ S | x ∉ A}.
Key properties: (A')' = A, A ∪ A' = S, A ∩ A' = ∅, S' = ∅, ∅' = S.
Union: A ∪ B = {x ∈ S | x ∈ A or x ∈ B or both}.
Intersection: A ∩ B = {x ∈ S | x ∈ A and x ∈ B}.
Distributive laws:
(A ∩ B) ∪ C = (A ∪ C) ∩ (B ∪ C)
(A ∪ B) ∩ C = (A ∩ C) ∪ (B ∩ C)
Partition of B using A: B = (A ∩ B) ∪ (A' ∩ B), where the two parts are mutually exclusive.
(A ∪ B)' = A' ∩ B' . The complement of a union is the intersection of the complements.
(A ∩ B)' = A' ∪ B' . The complement of an intersection is the union of the complements.
These laws generalise to any number of sets and are used constantly when simplifying probability expressions.
Events A₁, ..., Aₖ are mutually exclusive when Aᵢ ∩ Aⱼ = ∅ for all i ≠ j. No two can occur at the same time.
They are exhaustive when A₁ ∪ ... ∪ Aₖ = S. At least one must occur.
A set of events that is both mutually exclusive and exhaustive is called a partition of S.
Given a sample space S, a probability function P assigns a number P(A) to every event A ⊂ S, satisfying:
P(A) ≥ 0 for every event A.
P(S) = 1.
For any finite or countably infinite collection of mutually exclusive events A₁, A₂, ..., P(A₁ ∪ A₂ ∪ ...) = P(A₁) + P(A₂) + ...
Everything else in the probability section is derived from these three axioms.
Complement rule: P(A) = 1 - P(A'). Proof: S = A ∪ A', and A and A' are mutually exclusive, so 1 = P(A) + P(A').
Probability of the empty set: P(∅) = 0. Proof: ∅ = S', so P(∅) = 1 - P(S) = 0.
Monotonicity: If A ⊂ B, then P(A) ≤ P(B). Proof: B = A ∪ (B ∩ A'), where the two parts are mutually exclusive, so P(B) = P(A) + P(B ∩ A') ≥ P(A).
Upper bound: P(A) ≤ 1 for every event A. Proof: A ⊂ S, so P(A) ≤ P(S) = 1.
Decomposition of B: P(B) = P(A ∩ B) + P(A' ∩ B).
Inclusion-exclusion (two events): P(A ∪ B) = P(A) + P(B) - P(A ∩ B). Proof: write A ∪ B = A ∪ (A' ∩ B), apply additivity, then substitute P(A' ∩ B) = P(B) - P(A ∩ B).
Example: Train A is on time with probability 0.93, train B with probability 0.89, and both are on time with probability 0.87. The probability that at least one is on time = 0.93 + 0.89 - 0.87 = 0.95.
Relative frequency:
relative frequency of outcome x = (number of trials with outcome x) / (total number of trials)
Three axioms of probability:
P(A) ≥ 0
P(S) = 1
P(A₁ ∪ A₂ ∪ ...) = P(A₁) + P(A₂) + ... when the Aᵢ are mutually exclusive
Complement rule:
P(A) = 1 - P(A')
Inclusion-exclusion for two events:
P(A ∪ B) = P(A) + P(B) - P(A ∩ B)
Decomposition of B by A:
P(B) = P(A ∩ B) + P(A' ∩ B)
DeMorgan's laws:
(A ∪ B)' = A' ∩ B'
(A ∩ B)' = A' ∪ B'
The inclusion-exclusion formula is the basis of any situation where you need the probability that at least one of two things happens: at least one server stays online, at least one train arrives on time, at least one test detects a fault. Insurance and reliability engineering use these identities constantly to calculate the chance that a system with redundant components keeps working.
Students often confuse "mutually exclusive" with "independent." They are closer to opposites: if two events with positive probability are mutually exclusive, they are dependent, because knowing one occurred tells you the other did not.
Students sometimes assume that relative frequencies equal the true probabilities after a small number of trials. They do not. Convergence to the probability requires a large number of repetitions.
The complement rule P(A) = 1 - P(A') is sometimes forgotten when it would be the simplest route. Whenever computing P(A) directly looks hard, try computing P(A') instead.
DeMorgan's laws are frequently mis-applied by swapping unions and intersections without also swapping the complement. Both sides of each law must change: the operation and the complement.
⚠️ Be able to state and apply the three axioms of probability from memory.
⚠️ The complement rule and inclusion-exclusion for two events appear in nearly every problem set and exam. Practise both until they are automatic.
⚠️ DeMorgan's laws are a common exam question, either as a "prove this" or as a step within a longer derivation.
⚠️ Know the difference between mutually exclusive and exhaustive, and be ready to identify whether a given collection of events forms a partition of S.
True or false: P(∅) = 0.
True or false: If A and B are mutually exclusive, then P(A ∪ B) = P(A) + P(B).
Fill in the blank: (A ∪ B)' = A' ___ B'.
True or false: P(A) can be negative.
Fill in the blank: P(A) + P(A') = ___.
Q: A fair die is rolled. What is P({2, 3})?
A: There are two favourable outcomes out of six equally likely ones, so P({2, 3}) = 2/6 = 1/3.
Q: If P(A) = 0.6 and P(B) = 0.5, and P(A ∩ B) = 0.3, what is P(A ∪ B)?
A: By inclusion-exclusion, P(A ∪ B) = 0.6 + 0.5 - 0.3 = 0.8.
Q: Prove that P(∅) = 0 using only the axioms.
A: S' = ∅, so P(∅) = P(S') = 1 - P(S) = 1 - 1 = 0, using the complement rule derived from axioms 2 and 3.
Q: Train A is on time with probability 0.93, train B with probability 0.89, and both are on time with probability 0.87. What is the probability that at least one train is on time?
A: P(A ∪ B) = 0.93 + 0.89 - 0.87 = 0.95.
Q: State DeMorgan's first law and give a one-sentence intuition for why it holds.
A: (A ∪ B)' = A' ∩ B'. An outcome is outside "A or B" exactly when it is outside A and outside B simultaneously.
The set operations and probability axioms in this section are the machinery behind conditional probability (Section 1.3), independence (Section 1.4) and Bayes' theorem (Section 1.5). Every derivation there starts by decomposing events with unions, intersections and complements. If you are comfortable here, the rest of Chapter 1 is largely about applying these same tools in new ways. The concept of a partition (mutually exclusive and exhaustive events) returns directly in the law of total probability and in the general form of Bayes' theorem.
Probability axioms, Kolmogorov axioms, sample space, outcome space, event, subset, complement, union, intersection, DeMorgan's laws, De Morgan, mutually exclusive, disjoint events, exhaustive events, partition, inclusion-exclusion principle, addition rule, complement rule, random experiment, random variable, distribution, relative frequency, frequency interpretation of probability, set theory for probability, probability basics, Hogg and Tanis Chapter 1, Analytical Statistics NYU