A First Course in Probability
An elementary introduction to the theory of probability for students in mathematics, statistics, engineering, and the sciences. It covers the basic principles of combinatorial analysis, probability axioms, conditional probability, random variables, and limit theorems.
Lessons
Lesson
Combinatorial analysis provides the mathematical framework for counting system configurations and outcomes without the need for exhaustive listing. This lesson introduces foundational techniques, including the Generalized Principle of Counting, recursive modeling, and the use of slack variables to solve constrained distribution problems.
This lesson introduces the fundamental concepts of probability theory, focusing on defining sample spaces as the set of all possible outcomes and events as specific subsets within that space. Students learn to categorize sample spaces as discrete or continuous and apply set theory and counting principles to calculate probabilities in various experimental contexts.
Conditional probability is a dynamic process of updating beliefs by restricting the sample space based on new information, defined by the formula $P(E|F) = P(EF)/P(F)$. This lesson explores how this mathematical framework allows us to refine likelihoods and avoid common logical errors, such as the Prosecutor's Fallacy, in real-world scenarios.
This lesson introduces discrete random variables as functions that map experimental outcomes to numerical values, enabling the use of summation to analyze probability. You will learn to define and verify Probability Mass Functions (PMFs) and use them to calculate the likelihood of specific events or ranges of outcomes.
This lesson introduces continuous random variables, explaining how they shift from discrete sums to integrals for calculating probabilities, expected values, and variance. Students learn to use probability density functions (PDFs) and cumulative distribution functions (CDFs) to model real-world phenomena and solve optimization problems.
This lesson introduces joint probability distributions, explaining how to model multiple random variables simultaneously using joint cumulative distribution functions and probability density functions. Students learn to analyze variable dependency, geometric constraints, and marginalization to understand how individual outcomes interact within a shared probability space.
This lesson introduces the principle of linearity of expectation, which allows for the calculation of the expected value of a sum of random variables by summing their individual expectations, regardless of their dependence. Students will learn to apply this powerful tool using indicator variables to simplify complex problems, analyze unbiased estimators like the sample mean, and understand the necessary convergence conditions for infinite series.
This lesson explores the Law of Averages, demonstrating how increasing sample sizes reduces individual volatility to reveal stable, predictable patterns. Students learn to quantify this stability using the signal-to-noise ratio and understand how probabilistic averages converge toward deterministic limits.
This lesson explores the dynamics of stochastic processes, focusing on Markovian state transitions and the Poisson process for modeling discrete arrivals over time. It also introduces Shannon entropy as a mathematical framework to quantify uncertainty and the information gain derived from random events.
This lesson introduces simulation as a powerful empirical method for estimating probabilities in complex systems where analytical solutions are mathematically intractable. By using indicator variables to track outcomes and applying the Strong Law of Large Numbers, we can use computational repetition to converge on accurate probability estimates.
Course Overview
📚 Content Summary
An elementary introduction to the theory of probability for students in mathematics, statistics, engineering, and the sciences. It covers the basic principles of combinatorial analysis, probability axioms, conditional probability, random variables, and limit theorems.
A classic, comprehensive foundation to the mathematical theory and applications of probability.
Author: Sheldon Ross
Acknowledgments: Hossein Hamedani, Joe Blitzstein, Peter Nuesch, Ivan Ardestani, and several university reviewers/contributors are credited for accuracy and feedback.
🎯 Learning Objectives
- Apply the Basic and Generalized Principles of Counting to multi-stage experiments.
- Differentiate between and calculate permutations and combinations for both distinct and indistinguishable objects.
- Prove combinatorial identities using algebraic induction and logical combinatorial arguments.
- Define sample spaces and events for diverse experiments and apply DeMorgan's laws to set operations.
- Calculate probabilities using the three fundamental Axioms of Probability and simple propositions (complements, unions, and subsets).
- Solve complex combinatorial problems involving equally likely outcomes, such as poker hands, the Matching problem, and the Birthday problem.
- Define and calculate conditional probabilities using the formula P(E|F) = \frac{P(EF)}{P(F)}.
- Apply Bayes's Formula to solve complex problems involving multiple hypotheses and diagnostic testing.
- Distinguish between independent and conditionally independent events in practical scenarios like genetics and engineering.
- Define discrete random variables and compute their PMFs and CDFs.
Lessons
Overview: This lesson covers the fundamental mathematical theory of counting, known as combinatorial analysis. It progresses from the basic principles of multiplication for independent experiments to the formal study of permutations and combinations. Students will master the binomial and multinomial theorems, explore various proof techniques, and solve complex distribution problems using integer-valued equations.
Learning Outcomes:
- Apply the Basic and Generalized Principles of Counting to multi-stage experiments.
- Differentiate between and calculate permutations and combinations for both distinct and indistinguishable objects.
- Prove combinatorial identities using algebraic induction and logical combinatorial arguments.
Overview: This lesson establishes the formal mathematical foundation of probability theory, beginning with the definition of sample spaces and events. It introduces Kolmogorov’s three Axioms of Probability and derived propositions, such as the Inclusion-Exclusion Principle. The content extends into combinatorial applications including the Birthday Problem and Matching Problem.
Learning Outcomes:
- Define sample spaces and events for diverse experiments and apply DeMorgan's laws to set operations.
- Calculate probabilities using the three fundamental Axioms of Probability and simple propositions.
- Solve complex combinatorial problems involving equally likely outcomes, such as poker hands, the Matching problem, and the Birthday problem.
Overview: This lesson explores how the probability of an event is revised in light of new information. It transitions from unconditional probabilities to conditional frameworks, formalizing the relationship between dependent and independent events through Bayes's Formula and Laplace's Rule of Succession. Students will learn to update prior probabilities with empirical evidence to reach posterior conclusions.
Learning Outcomes:
- Define and calculate conditional probabilities using the formula P(E|F) = \frac{P(EF)}{P(F)}.
- Apply Bayes's Formula to solve complex problems involving multiple hypotheses and diagnostic testing.
- Distinguish between independent and conditionally independent events in practical scenarios like genetics and engineering.
Overview: This lesson explores the fundamental theory and application of discrete random variables, which are variables whose set of possible values is finite or countably infinite. We define their probability mass functions (PMF) and cumulative distribution functions (CDF), while establishing core measures of central tendency and dispersion. Finally, the lesson examines specific distribution families used to model real-world phenomena.
Learning Outcomes:
- Define discrete random variables and compute their PMFs and CDFs.
- Calculate the Expected Value and Variance of a random variable and its functions.
- Identify and apply the appropriate discrete probability distribution to solve complex word problems.
Overview: This lesson explores the properties and applications of continuous random variables, focusing on their expectations, variances, and specific probability distributions. Students will learn to model real-world phenomena using Uniform, Normal, Exponential, Gamma, Weibull, Cauchy, and Beta distributions. The lesson also covers techniques for approximating discrete distributions and transforming random variables.
Learning Outcomes:
- Calculate the expectation and variance for continuous random variables and functions of those variables.
- Apply the Normal distribution and its approximation to the Binomial distribution using the continuity correction.
- Analyze reliability and lifetimes using Exponential distributions, the memoryless property, and hazard rate functions.
Overview: This lesson explores the mathematical framework for handling multiple random variables simultaneously. It covers the transition from individual distributions to joint probability density/mass functions, the rigorous definition of independence, and the behavior of sums of independent variables. Furthermore, the curriculum extends into advanced topics including order statistics and transformations of random vectors using Jacobians.
Learning Outcomes:
- Compute marginal distributions and conditional densities for continuous and discrete joint random variables.
- Apply factorization criteria to determine if random variables are independent and model complex processes.
- Use the Jacobian determinant method to find the joint distribution of functions of random variables and calculate the distributions of order statistics.
Overview: This lesson explores the advanced properties of mathematical expectation, moving beyond simple averages to the linearity of sums, covariance, and the power of conditioning. Students will learn to apply these tools to algorithmic analysis, probabilistic bounds, and predictive modeling.
Learning Outcomes:
- Apply the linearity of expectation to complex sums, including indicator variables and infinite series.
- Calculate and interpret covariance, correlation, and the variance of sums for dependent and independent variables.
- Utilize conditional expectation and variance to simplify the analysis of compound random variables and solve optimization problems.
Overview: This lesson covers the fundamental asymptotic results of probability theory, specifically how the sum and average of random variables behave as the number of observations grows to infinity. We explore the Weak and Strong Laws of Large Numbers and the Central Limit Theorem. Specific probability bounds like the one-sided Chebyshev inequality are also addressed.
Learning Outcomes:
- Distinguish between the Weak and Strong Laws of Large Numbers in terms of convergence criteria.
- Apply the Central Limit Theorem (CLT) to approximate probabilities for sums of random variables using the normal distribution.
- Utilize the one-sided Chebyshev inequality to provide upper bounds for tail probabilities.
Overview: This lesson explores the mathematical frameworks for modeling random events over time and quantifying information. It covers the Poisson Process and its interarrival times, the structure and long-term behavior of Markov Chains, and the fundamental principles of Information Theory. Specifically, it addresses entropy and its application to optimal coding.
Learning Outcomes:
- Define and calculate probabilities for the Poisson Process and determine the distribution of interarrival times.
- Construct transition matrices for Markov Chains and utilize the Chapman-Kolmogorov equations to find n-step probabilities.
- Calculate limiting probabilities for Ergodic Markov Chains and solve random walk problems.
Overview: This lesson explores the principles and applications of simulation for empirically determining probabilities and expected values. It covers the generation of random permutations, techniques for simulating continuous and discrete random variables, and advanced methods for variance reduction. These techniques improve the efficiency and accuracy of simulation estimates.
Learning Outcomes:
- Understand the role of pseudorandom number generators and seeds in simulation.
- Implement algorithms to generate random permutations and simulate variables from both discrete and continuous distributions.
- Apply the Polar Method for generating unit normals and simulate chi-squared variables.