Math401 Topic 1: Probability under language of measure theory

Section 1: Uniform Random Numbers

Basic Definitions

Definition of Random Variables

A random variable is a function $f:[0,1]\to S$, where $[0,1]\subset \mathbb{R}$ and $S$ is a set of potential outcomes of a random phenomenon.

Definition of Uniform Distribution

The uniform distribution is defined by the length of function on subsets of $[0,1]$ as a measure of probability (Lebesgue measure  by default).

Let $X$ be a random number taken from $[0,1]$ and having the uniform distribution. The probability that $X$ should be the probability of the event that $X$ lies in $A$.

Definition of Expectation

Let $f:[0,1]\to \mathbb{R}$ be a random variable (with nice properties such that it is integrable). Then the expectation of $f$ is defined as

Definition of Indicator Function

The indicator function of an event $A$ is defined as

Definition of Law of variable X

The law of a random variable $X$ is the probability distribution of $X$.

Let $Y$ be the outcome of $f(X)$. Then the law of $Y$ is the probability distribution of $Y$.

1.1 Mathematical Coin Flip model

A coin flip if a random experiment with two possible outcomes: $S=\{0,1\}$. with probability $p$ for $0$ and $1-p$ for $1$, where $p\in (0,1)\subset \mathbb{R}$.

Definition of Independent Events

Two events $A$ and $B$ are independent if

or equivalently,

Generalization to $n$ events:

Definition of Outcome selecting function

Let the set of all possible outcomes represented by a Cartesian product $S=\{0,1\}^{\mathbb{N}}$. $(a_1,a_2,a_3,\cdots)\subset S$ is an infinite (or finite) sequence of coin flips.

$\pi_i:S\to \{0,1\}$ is the $i$-th projection function defined as $\pi_i((a_1,a_2,a_3,\cdots))=a_i$.

Note, this representation is isomorphic to the dyadic rationals (i.e., numbers that can be written as a fraction whose denominator is a power of 2) in the interval $[0,1]$.

Section 2: Formal definitions

Recall, the $\sigma$-algebra (denoted as $\mathcal{A}$ in Math4121) is the collection of all subsets of a set $S$ satisfying the following properties:

1. $\emptyset\in \mathcal{A}$ (empty set is in the $\sigma$-algebra)
2. If $A\in \mathcal{A}$, then $A^c\in \mathcal{A}$ (if a set is in the $\sigma$-algebra, then its complement is in the $\sigma$-algebra)
3. If $A_1,A_2,A_3,\cdots\in \mathcal{A}$, then $\bigcup_{i=1}^{\infty}A_i\in \mathcal{A}$ (if a countable sequence of sets is in the $\sigma$-algebra, then their union is in the $\sigma$-algebra)

Event, probability, and random variable

Let $\Omega$ be a non-empty set.

Let $\mathscr{F}$ be a $\sigma$-algebra on $\Omega$ (Note, $\mathscr{F}$ is a collection of subsets of $\Omega$ that satisfies the properties of a $\sigma$-algebra).

Definition of Event

An event is a element of $\mathscr{F}$.

Definition of Probability Measure

A probability measure $P$ is a function $P:\mathscr{F}\to [0,1]$ satisfying the following properties:

1. $P(\Omega)=1$
2. If $A_1,A_2,A_3,\cdots\in \mathscr{F}$ are pairwise disjoint ($\forall i\neq j, A_i\cap A_j=\emptyset$), then $P(\bigcup_{i=1}^{\infty}A_i)=\sum_{i=1}^{\infty}P(A_i)$

Definition of Probability Space

A probability space is a triple $(\Omega, \mathscr{F}, P)$ defined above.

An event $A$ is said to occur almost surely (a.s.) if $P(A)=1$.

Definition of Random Variable

A random variable is a function $X:\Omega\to \mathbb{R}$ that is measurable with respect to the $\sigma$-algebra $\mathscr{F}$.

That is, for any Borel set $B\subset \mathbb{R}$, the preimage $f^{-1}(B)\in \mathscr{F}$.

Definition of sigma-algebra generated by a random variable

Let $\{f_\alpha:\Omega\to \mathbb{R},\alpha\in I\}$ be a family of functions where $I$ is an index set which is not necessarily finite or countable. The $\sigma$-algebra generated by the family of functions $\{f_\alpha:\alpha\in I\}$, denoted as $\sigma\{f_\alpha:\alpha\in I\}$, is the smallest $\sigma$-algebra containing all the subsets of $\Omega$ of the form

for all $\alpha\in I$ and $B\in \mathscr{B}(\mathbb{R})$.

Equivalently,

the sigma-algebra generated by a random variable $X$ is the intersection of all $\sigma$-algebras on $\Omega$ containing the sets $f_\alpha^{-1}(B)$ for all $\alpha\in I$ and $B\in \mathscr{B}(\mathbb{R})$.

Definition of distribution of random variable

Let $f:\Omega\to \mathbb{R}$ be a random variable. The distribution of $f$ is the probability measure $P_f$ on $\mathbb{R}$ defined by

also noted as $f_*P$.

Definition of joint distribution of random variables

Let $f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R}$ be random variables. The joint distribution of $f_1,f_2,\cdots,f_n$ is the probability measure $P_{f_1,f_2,\cdots,f_n}$ on $\mathbb{R}^n$ defined by

Expectation of a random variable

Let $f:\Omega\to \mathbb{R}$ be a random variable. The expectation of $f$ is defined as

Note, $P$ is the probability measure on $\Omega$.

Definition of variance

The variance of a random variable $f$ is defined as

Definition of covariance

The covariance of two random variables $f,g:\Omega\to \mathbb{R}$ is defined as

Point measures

Definition of Dirac measure

The Dirac measure is a probability measure on $\Omega$ defined as

Note that $\int_\Omega f(x)d\delta_\omega(x)=f(\omega)$.

Infinite sequence of independent coin flips

Side notes from basic topology:

Definition of product topology:

It is a set constructed by the Cartesian product of the sets. Suppose $X_i$ is a set for all $i\in I$. The element of the product set is a tuple $(x_i)_{i\in I}$ where $x_i\in X_i$ for all $i\in I$.

For example, if $X_i=[0,1]$ for all $i\in \mathbb{N}$, then the product set is $[0,1]^{\mathbb{N}}$. An element of such product set is $(1,0.5,0.25,\cdots)$.

The set of outcomes of such infinite sequence of coin flips is the product set of the set of outcomes of each coin flip.

Conditional probability

Definition of conditional probability

The conditional probability of an event $A$ given an event $B$ is defined as

The law of total probability:

Bayes’ theorem:

Definition of independence of random variables

Two random variables $f,g:\Omega\to \mathbb{R}$ are independent if for any Borel sets $A,B\subset \mathscr{B}(\mathbb{R})$ the events

are independent.

In general, a finite or infinite family of random variables $f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R}$ are independent if every finite collection of random variables from this family are independent.

Definition of independence of sigma-algebras

Let $\mathscr{G}$ and $\mathscr{H}$ be two $\sigma$-algebras on $\Omega$. They are independent if for any Borel sets $A\subset \mathscr{B}(\mathbb{R})$ and $B\subset \mathscr{B}(\mathbb{R})$, the finite collection of events are independent.

Section 3: Further definitions in measure theory and integration

$L^2$ space

Definition of $L^2$ space

Let $(\Omega, \mathscr{F}, P)$ be a measure space. The $L^2$ space is the space of all square integrable, complex-valued measurable functions on $\Omega$.

Denoted by $L^2(\Omega, \mathscr{F}, P)$.

The square integrable functions are the functions $f:\Omega\to \mathbb{C}$ such that

With inner product defined by

The $L^2(\Omega, \mathscr{F}, P)$ space is a Hilbert space.