Professor: Erik Hintz | Term: Fall 2023

Introduction to Probability

Definitions of Probability

Definition

Classical Definition of Probability

$\frac{number of ways an event can occur}{total # of possible outcomes}$

$^{*}$ Provided things are equally likely.

Definition

Relative Frequency

The probability of an event is proportionate to number of times the event occurs in long repetitions. The probability of rolling a 2 is $\frac{1}{6}$ because after 6 million rolls, 2 will appear ~1 million times. (this is not very practical).

Definition

Subjective Definition

Persons belief on how likely something will happen (unclear since varies by person).

Mathematical Probability Models

Samples Spaces and Probability

Sample Spaces & Sets

Sample space $S$ is a set of distinct outcomes of an experiment with the property that in a single trial, only one outcome will occur.

Die: $S = {1, 2, 3, 4, 5, 6}$ or $S = {even, odd}$
Number of coin flips until heads occurs: $S = N^{+}$
Waiting time in minutes until a sunny day. $S = [0, \infty) = {x \in R : x \geq 0}$

Sample space is discrete if it is finite or countably infinite (one-to-one correspondence with $N$ ). Otherwise non-discrete.

$N$ is discrete (countably infinite).

Event is a subset of a sample space $S$ .

$A$ is an event if $A \subseteq S$ ( $A$ is a subset of $S$ , $A$ is contained in $S$ ).

Die shows $6$ : $A = {6}$
$20$ or fewer tosses till heads. $A = {1, \dots, 20}$
$A = {60, \dots \infty}$

Notation

Element of: $x \in A (x$ in $A)$
Union: $A \cup B : {x ∣ x \in A$ or $x \in B}$
Intersection: $A \cap B : {x ∣ x \in A$ and $x \in B}$
Complement: $A^{∁} : {x ∣ x \in S, x \in A} = A^{'} = \overline{A}$
Empty: $\emptyset$
Disjoint $⟹ A \cap B = \emptyset$

$2$ die rolled.

$S = {(x, y) : x, y \in {(1, \dots, 6)}$
$A = {(x, y) : (x, y) \in S \land x + y = 7}$
- $A = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}$
$B^{∁} = {(x, y) : (x, y) \in S \land x + y < 4}$
- $B^{∁} = {(1, 1), (1, 2), (2, 1)}$
$A \cap B^{∁} = \emptyset$
$A \cup B^{∁} = {A, B^{∁}}$

Definition

Probability

Let $S$ denote the set of all events on a given sample $S$ . Probability defined on $S$ is defined as
$P : S \to [0, 1]$

Scale: $0 \leq P (A) \leq 1$
$P (S) = 1$
Additivity (infinite): $P (⋃_{i = 1}^{\infty} A_{i}) = \sum_{i = 1}^{\infty} P (A_{i})$

e.g. $A_{1} = {1}, A_{2} = {2}, A_{3} = A_{4} = \emptyset$

$P (1$ or $2) = P (A_{1}) + P (A_{2})$

$S$ (discrete) and $A \subset S$ is an event.

$A$ is indivisible $⟺ A$ is a simple point, otherwise compound.

e.g.

$S = {1, 2, 3, 4, 5, 6}$ $A = {1} \leftarrow$ simple $B = {2, 4, 6} \leftarrow$ compound.

Assign so that

$0 \leq P (a_{i}) \leq 1$
$\sum_{i} P (a_{i}) = 1$

{P (a_{i}) : i = 1, 2, 3, \dots} = probability distribution

$S = {a_{1}, a_{2}, \dots}$ discrete, $A \subset S$ is an event.

P (A) = a_{i} \in A \sum P (a_{i})

$A$ = Number is odd

$P (i) = \frac{1}{6}$ for $1, 2, 3, 4, 5, 6$

P (A) = P (1) + P (3) + P (5) = \frac{1}{6} + \frac{1}{6} + \frac{1}{6} = \frac{3}{6} = \frac{1}{2}

Sample space $S$ is equally likely if probability of every outcome is the same.

$∣ A ∣ = #$ of elements in set.

1 = P (S) = i = 1 \sum ∣ S ∣ P (a_{i}) P (a_{i}) = P (a_{i}) ∣ S ∣ = \frac{1}{∣ S ∣}

P (A) = i : a_{i} \in A \sum P (a_{i}) = \frac{∣ A ∣}{∣ S ∣}

Probability and Counting Techniques

Fact

$A$ and $B$ are disjoint $A \cap B = \emptyset$ , then
$∣ A \cup B ∣ = ∣ A ∣ + ∣ B ∣$

Addition and Multiplication Rules

Addition Rule

Sum larger than 8.

$B =$ Sum of Die larger than 8

$A_{9} =$ Sum 9, $A_{10} =$ Sum 10, $A_{11} =$ Sum 11, $A_{12} =$ Sum 12.

$B = A_{9} \cup A_{10} \cup A_{11} \cup A_{12}$ all $A_{j}$ are disjoint.

∣ B ∣ = ∣ A_{9} ∣ + ∣ A_{10} ∣ + ∣ A_{11} ∣ + ∣ A_{12} ∣ = 4 + 3 + 2 + 1 = 10

Definition

Thus,
$∣ A ∣ = ∣ i = 1 ⋃ n A_{i} ∣ = i = 1 \sum n ∣ A_{i} ∣$

Multiplication Rules

Ordered $k -$ tuple, $(a_{1}, a_{2}, \dots, a_{k})$ . $n_{1}$ choices for $a_{1}$ , $n_{k}$ choices for $a_{k}$ etc.

Definition

$∣ A ∣ = n_{1} n_{1} \dots n_{k} = \prod_{i = 1}^{k} n_{i}$

If there are $p$ ways to do task 1, and $q$ ways to do task 2, there are $p \times q$ ways to do tasks 1 and 2.

With replacement: When the object is selected, put it back after.

Without replacement: Once an object is picked, it stays out of the pool.

Counting Arrangements or Permutations

Definition

Factorials

$n$ distinct objects, $n$ factorial.
$n! = n \cdot (n - 1) \cdot (n - 2) \dots 2 \cdot 1$

$0! = 1$ and $n! = n \cdot (n - 1)!$

10 people standing next to each other, $10! = 3628800$ arrangements.

Out of 5 students, must choose 1 president and 1 secretary. $5 \times 4 = 20$ .

Note, Stirling’s Approximation.

n! \approx 2 πn (\frac{n}{e})^{n}

Definition

Permutations

$n$ distinct objects, permutation of size $k$ is an ordered subset of $k$ individuals. (without replacement)
$n^{(k)} = n (n - 1) \dots (n - k + 1) = \frac{n !}{( n - k )!}$

With replacement,

n^{k} = n (n) \dots (n) k times

$n$ to the $k$ factors.

Our example of president and secretary above can thus be seen as $5^{(2)} = 5 \times 4$ .

Counting Subsets or Combinations

Question

If we have $5$ members on a club, how can we select $2$ to serve on a committee?

Here, order does not matter. Unlike a permutation, this is a combination.

In general, if we have $k$ objects, there are $k!$ ways to reorder such objects. We can therefore get the combination count by dividing the permutation count $n^{(k)}$ by the number of ways of ordering the objects $k!$ .

Definition

Definition

Given $n$ distinct objects, a combination of size $k$ is an unordered subset of $k$ of the objects (without replacement). The number of combinations of size $k$ taken from $n$ objects is denoted $(k n)$ , which reads ” $n$ choose $k$ , and
$(k n) = \frac{n ^{(k)}}{k !} = \frac{n !}{( n - k )! k !}$

For combinatorial proofs of these identities see MATH239.

For our previous example we have
$(k n) = \frac{n !}{( n - k )! k !} = \frac{5 !}{3 ! 2 !} = 10$

Properties of $(k n)$

$n^{(k)} = \frac{n !}{( n - k )!} = n (n - 1)^{(k - 1)}$ for $k \geq 1$
$(k n) = \frac{n !}{k ! ( n - k )!} = \frac{n ^{(k)}}{k !}$
$(k n) = \frac{n !}{( n - k )! k !} = \frac{n !}{( n - ( n - k ))! ( n - k )!} = (n - k n)$
If we define $0! = 1$ , then $(0 n) = (n n) = 1$
$(k n) = (k - 1 n - 1) + (k n - 1)$
Binomial Theorem $(1 + x)^{n} = (0 n) + (1 n) x + (2 n) x^{2} + \dots + (n n) x^{n}$

Example

Group of $5$ women and $7$ men, a committee of $2$ women and $3$ men is formed at random. $2$ of them men dislike each other. What is the probability they don’t serve together?

First, get the size of the sample space $∣ S ∣$ .

Pick $2$ women from $5$ women, $(2 5)$

Pick $3$ men from $7$ men, $(3 7)$

Thus $∣ S ∣ = (2 5) (3 7)$

Consider the event $A = {1$ and $2$ do not serve in the committee together $}$

Consider $A^{∁} = {$ 1 and 2 in the committee together $}$

The size of the sample space $∣ A^{∁} ∣$ is given by:

Pick $2$ women from $5$ women, $(2 5)$ .

$1$ and $2$ are in the committee already, we need to pick $1$ man from the $5$ left, $(1 5)$ .

Thus,
$P (A) = 1 - P (A^{∁}) = 1 - \frac{∣ A ^{∁} ∣}{∣ S ∣} = 1 - \frac{( 2 5 ) ( 1 5 )}{( 2 5 ) ( 3 7 )} = \frac{( 2 5 ) [ ( 3 7 ) - ( 1 5 ) ]}{( 2 5 ) ( 3 7 )}$

Arrangements when Symbols are Repeated

Definition

Definition

Consider $n$ objects of $k$ types. Suppose $n_{1}$ objects of type $1$ , $n_{2}$ objects of type $2$ , and $n_{k}$ objects of type $k$ . Thus there are,
$\frac{n !}{n _{1} ! n _{2} ! \dots n _{k} !}$
distinguishable arrangements of $n$ objects. This is the multinomial coefficient.

Letters of “SLEEVELESS” are arranged at random. What is the probability the word begins and ends with “S”?

Sample space, $∣ S ∣ = \frac{10 !}{4 ! 3 ! 2 ! 1 !} = 12600$

Event, $∣ A ∣ = \frac{8 !}{1 ! 4 ! 2 ! 1 !} = \frac{40320}{48} = 840$

Thus $P (A) = \frac{840}{12600} = \frac{1}{15}$

Useful Series and Sums

Theorem

Finite Geometric Series:
$i = 0 \sum n - 1 t^{i} = 1 + t + t^{2} + \dots + t^{n - 1} = \frac{1 - t ^{n}}{1 - t} if t \neq = 1$

Theorem

Infinite Geometric series if $∣ t ∣ < 1$ :
$x = 0 \sum \infty t^{x} = 1 + t + t^{2} + \dots = \frac{1}{1 - t}$

Theorem

Binomial Theorem $(i)$ , if $n$ is a positive integer and $t$ is any real number:
$(1 + t)^{n} = 1 + (1 n) t^{2} + (2 n) t^{2} + \dots + (n n) t^{n} = x = 0 \sum n (x n) t^{x}$

Theorem

Binomial Theorem $(ii)$ , if $n$ is not a positive integer but $∣ t ∣ < 1$ :
$(1 + t)^{n} = x = 0 \sum \infty (x n) t^{x}$

Theorem

Multinomial Theorem:
$(t_{1} + t_{2} + \dots t_{k})^{n} = \sum \frac{n !}{x _{1} ! x _{2} ! \dots x _{k} !} t_{1}^{x_{1}} t_{2}^{x_{2}} \dots t_{k}^{x_{k}}$
where the summation is over all non-negative integers $x_{1}, x_{2}, \dots, x_{k}$ such that $\sum_{i = 1}^{k} x_{i} = n$ where $n$ is a positive integer.

Theorem

Hypergeometric Identity:
$(n a + b) = x = 0 \sum \infty (x a) (n - x b)$

Theorem

Exponential Series:
$e^{t} = \frac{t ^{0}}{0 !} + \frac{t ^{1}}{1 !} + \frac{t ^{2}}{2 !} + \dots = x = 0 \sum \infty \frac{t ^{n}}{n !}$
for all $t$ in the real numbers.

A related identity:

e^{t} = n \to \infty lim (1 + \frac{t}{n})^{n}

Series involving integers:

1 + 2 + 3 + \dots + n 1^{2} + 2^{2} + 3^{3} + \dots + n^{2} 1^{3} + 2^{3} + 3^{3} + \dots + n^{3} = \frac{n ( n + 1 )}{2} = \frac{n ( n + 1 ) ( 2 n + 1 )}{6} = [\frac{n ( n + 1 )}{2}]^{2}

Probability Rules and Conditional Probability

General Methods

Rules: $P (S) = \sum_{all i} P (a_{i}) = 1$

For any event $A, 0 \leq P (A) \leq 1$ .

If $A$ and $B$ are two events with $A \subseteq B$ , then $P (A) \leq P (B)$ .

Fact

Fundamental Laws of Set Algebra

Commutativity
$A \cup B = B \cup A A \cap B = B \cap A$
Associativity
$(A \cup B) \cup C (A \cap B) \cap C = A \cup (B \cup C) = A \cap (B \cap C)$
Distributivity
$A \cup (B \cap C) = (A \cup B) \cap (A \cup C) A \cap (B \cup C) = (A \cap B) \cup (A \cap C)$
De Morgan’s Law
$(\overline{A \cup B}) = \overline{A} \cap \overline{B}$
The complement of a union is the intersection of the complements.
$(\overline{A \cap B}) = \overline{A} \cup \overline{B}$
The complement of an intersection is the union of the complements.

Applied for $n$ events
$\overline{(A_{1} \cup A_{2} \cup \dots \cup A_{n})} = \overline{A_{1}} \cap \overline{A_{2}} \cap \dots \cap \overline{A_{n}}$

Rules for Unions of Events

For arbitrary events $A, B, C$ ,

Fact

If $A$ and $B$ are disjoint $(A \cap B = \emptyset)$ then,
$P (A \cup B) = P (A) + P (B)$

Fact

What if $A \cap B \neq = \emptyset$ ?

Then
$P (A \cup B) = P (A) + P (B) - P (A \cap B)$

Fact

The probability of the complement event is,
$P (A) = 1 - P (\overline{A})$
If $A, B,$ and $C$ are disjoint (mutually exclusive), then
$P (A \cup B \cup C) = P (A) + P (B) + P (C)$
Otherwise,
$P (A \cup B \cup C) = P (A) + P (B) + P (C) - P (A \cap B) - P (A \cap C) - P (B \cap C) + P (A \cap B \cap C)$
In general,
$P (i = 1 ⋃ n A_{1}) = i \sum P (A) - i < j \sum P (A_{i} A_{j}) + i < j < k \sum P (A_{i} A_{j} A_{k}) - i < j < k < 1 \sum P (A_{i} A_{j} A_{k} A_{1}) + \dots$

Note, $P (A \cap B)$ is often written as $P (A B)$

Intersection of Events and Independence

Independent Events

Rolling die twice is an example of independent events. The outcome of the first doesn’t affect the second.

Definition

Events are independent if and only if
$P (A \cap B) = P (A) P (B)$

$A =$ ”roll $6$ on first” $B =$ “roll $6$ on second”

P (A \cap B) = P ("both 6") = \frac{1}{36} = P (A) P (B)

Not independent. $C =$ First roll is $6$ , $D =$ first roll is even.

$P (C \cap D) = \frac{1}{6}$

P (C) P (D) = \frac{1}{12} \neq = P (C \cap D)

A common misconception is that if $A$ and $B$ are mutually exclusive, then $A$ and $B$ are independent.

Fact

If $A$ and $B$ are independent, $A$ and $\overline{B}$ are independent. Law of total probability.
$P (A \cap \overline{B}) = P (A) - P (A \cap B) = P (A) - P (A) P (B) = P (A) [1 - P (B)] = P (A) P (\overline{B})$

Fact

We can also see,
$P (A) = P (A \cap B) + P (A \cap \overline{B})$

Conditional Probability

Definition

$P (A ∣ B)$ represents the probability that event $A$ occurs, when we know that $B$ occurs. This is the conditional probability of $A$ given $B$ .
$P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )} provided P (B) > 0$

Fact

If $A$ and $B$ are independent, then
$P (A \cap B) = P (A) P (B) P (A ∣ B) = \frac{P ( A ) P ( B )}{P ( B )} = P (A) provided P (B) > 0$
$A$ and $B$ are independent events if and only if either of the following statements is true
$P (A ∣ B) = P (A) or P (B ∣ A) = P (B)$

Example

$A$ = The sum of two die is 10 $B$ = The first die is 6
$P (B ∣ A) = \frac{P ( B \cap A )}{P ( A )} = \frac{\frac{1}{36}}{\frac{3}{36}} = \frac{1}{3}$

If $P (A) = 0$ or $P (B) = 0$ , then $A$ and $B$ are independent.

Properties

$P (B ∣ B) = 1$
$0 \leq P (A ∣ B) \leq 1$
If $A \subseteq C, P (A ∣ B) \leq P (C ∣ B)$
$P (A_{1} \cup A_{2} ∣ B) = P (A_{1} ∣ B) + P (A_{2} ∣ B) - P (A_{1} \cap A_{2} ∣ B)$
If $A_{1}$ and $A_{2}$ are disjoint: $P (A_{1} \cup A_{2} ∣ B) = P (A_{1} ∣ B) + P (A_{2} ∣ B)$
$P (\overline{A} ∣ B) = 1 - P (A ∣ B)$

Product Rules, Law of Total Probability and Bayes’ Theorem

Fact

Product Rules

For events $A$ and $B$ ,
$P (A \cap B) = P (B) P (A ∣ B) = P (A) P (B ∣ A)$

That means if we know $P (A ∣ B)$ and $P (B)$ ; or $P (B ∣ A)$ and $P (A)$ , we can find $P (A \cap B)$ .

More events:

$P (A \cap B) = P (A) P (B ∣ A)$
$P (A \cap B \cap C) = P (A) P (B ∣ A) P (C ∣ A \cap B)$
$P (A \cap B \cap C \cap D) = P (A) P (B ∣ A) P (C ∣ A \cap B) P (D ∣ A \cap B \cap C)$

Definition

Law of Total Probability

Let $A_{1}, A_{2}, \dots, A_{k}$ be a partition of the sample space $S$ into disjoint (mutually exclusive) events, that is
$A_{1} \cup A_{2} \cup \dots \cup A_{k} = S and A_{i} \cap A_{j} = \emptyset if i \neq = j$
Let $B$ be an arbitrary event in $S$ . Then
$P (B) = P (B A_{1}) + P (B A_{2}) + \dots + P (B A_{k}) = i = 1 \sum k P (B ∣ A_{i}) P (A_{i})$
A common way in which this is used is that
$P (B) = P (B ∣ A) P (A) + P (B ∣ \overline{A}) P (\overline{A})$
since $A$ and $\overline{A}$ partition $S$ .

Theorem

Bayes Theorem Suppose $A$ and $B$ are events defined on a sample space $S$ . Suppose also that $P (B) > 0$ . Then
$P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )} = \frac{P ( B ∣ A ) P ( A )}{P ( B ∣ A ) P ( A ) + P ( B ∣ A ) P ( A )}$

Proof:

\frac{P ( B ∣ A ) P ( A )}{P ( B ∣ A ) P ( A ) + P ( B ∣ A ) P ( A )} = \frac{P ( A B )}{P ( A B ) + P ( A B )} = \frac{P ( A B )}{P ( B )} = P (A ∣ B) ■

Discrete Random Variables

Random Variables and Probability Functions

Definition

A random variable is a function that assigns a real number to each point in a sample space $S$ . Often a random variable is abbreviated with RV or rv.

Definition

The values that a random variable can take on are called the range of the random variable. We often denote the range of a random variable $X$ by $X (S)$ .

Example

If we roll a 6-sided dice, our sample space is $S = {(1, 1), (1, 2), \dots, (6, 5), (6, 6)}$ and if we define $X =$ sum of die rolls, the range is ${2, 3, \dots, 11, 12}$

Definitions

We say that a random variable is discrete if it takes values in a countable set (finite or countably infinite).

We say that a random variable is continuous if it takes values in some interval of real numbers, e.g. $[0, 1], (0, \infty), R$ .

Important: Don’t forget that a random variable can be infinite and discrete.

Definition

Let $X$ be a discrete random variable with range $A$ . The probability function (p.f.) of $X$ is the function
$f (x) = P (X = x), defined for all x \in A$

The set of pairs ${(x, f (x)) : x \in A}$ is called the probability distribution of $X$ .

A probability function has two important properties:

$0 \leq f (x) \leq 1$ for all $x \in A$
$\sum_{all x \in A} f (x) = 1$

Example

A random variable $X$ has a range $A = {0, 1, 2}$ with $f (0) = 0.19, f (1) = 0.2 k^{2}, f (2) = 0.8 k^{2}$ , what values of $k$ makes $f (x)$ a probability function.

From our rules,
$⟹ 0.19 + 0.2 k^{2} + 0.8 k^{2} = 0.19 + k^{2} = 1 k^{2} = 0.81 k = \pm 0.9$

Definition

The Cumulative Distribution Function (CDF) of a random variable $X$ is
$F (x) = P (X \leq x), define for all x \in R$
We use the shorthand that $X \sim F$ if $X$ has CDF $F$ . Here,
$P (X \leq x) = P ({a \in S : X (a) \leq x})$
where ${X \leq a}$ is the event that contains all outcomes with $X (a) \leq x$ .

In general, for any $x \in R$
$F (x) = P (X \leq x) = u \leq x \sum P (X = u) = u \leq x \sum f (u)$

The CDF satisfies that

$0 \leq F (x) \leq 1$
$F (x) \leq F (y)$ for $x < y$ (and we say $F (x)$ is a non-decreasing function of $x$ )
$lim_{x \to - \infty} F (x) = 0$ , and $lim_{x \to \infty} F (x) = 1$
$lim_{x \to a^{+}} F (x) = F (a)$ (right continuous)

If $X$ takes value on $a_{1} < a_{2} < \dots < a_{n} < \dots$ , we can get probability function from CDF:

f (a_{i}) = F (a_{i}) - F (a_{i - 1})

In general, we have

P (a < X \leq b) = F (b) - F (a)

Note, we often use

P (X \geq 1) = 1 - P (X < 1) = 1 - P (x \leq 0)

Definition

Definition

Two random variables $X$ and $Y$ are said to have the same distribution if $F_{X} (x) = F_{Y} (x)$ for all $x \in R$ . We denote this by
$X \sim Y$

Note, $X$ and $Y$ having the same distribution does not mean $X = Y$ .

Discrete Uniform Distribution

Setup

Suppose the range of $X$ is ${a, a + 1, \dots, b}$ where $a$ and $b$ are integers and suppose all values are equally probable. Then $X$ has a Discrete Uniform Distribution on the set ${a, a + 1, \dots, b}$ . The variables $a$ and $b$ are called the parameters of the distribution.

Illustrations

If $X$ is the number obtained when a die is rolled, then $X$ has a discrete Uniform distribution with $a = 1$ and $b = 6$ .

Definition

Probability Function

There are $b - a + 1$ values in the set ${a, a + 1, \dots, b}$ so the probability of each value must be $\frac{1}{b - a + 1}$ in order for $\sum_{x = a}^{b} f (x) = 1$ . Therefore
$f (x) = P (X = x) = {\frac{1}{b - a + 1} for x = a, a + 1, \dots, b 0 otherwise$

Example

Suppose a fair die is thrown once and let $X$ be the number on the face. Find the cumulative distribution function of $X$ .

Solution

This is an example of a Discrete Uniform distribution on the set ${1, 2, 3, 4, 5, 6}$ having $a = 1, b = 6$ and probability function
$f (x) = P (X = x) = {\frac{1}{6} for x = 1, 2, \dots, 6 0 otherwise$
The cumulative distribution function is $F (x) = P (X \leq x)$ ,
$F (x) = P (X \leq x) = ⎩ ⎨ ⎧ 0 for x < 1 \frac{[ x ]}{6} for 1 \leq x < 6 1 for x \geq 6$
where $[x]$ is the largest integer less than or equal to $x$ .

Example

Let $X$ be the largest number when a die is rolled 3 times. First find the cumulative distribution function, and then find the probability function of $X$ .

Solution

This is another example of a distribution constructed from the Discrete Uniform.
$S = {(1, 1, 1), (1, 1, 2), \dots, (6, 6, 6)}$
The probability that the largest number is less than or equal to $x$ is,
$F (x) = \frac{x ^{3}}{6 ^{3}}$
for $x = 1, 2, 3, 4, 5, 6$ . Here is the CDF for all real values of $x$ :
$F (x) = P (X \leq x) = ⎩ ⎨ ⎧ \frac{[ x ] ^{3}}{216} for 1 \leq x < 6 0 for x < 1 1 for x \geq 6$

To find the p.f. we may use the fact that $x \in {1, 2, 3, 4, 5, 6}$ we have $P (X = x) = P (X \leq x) - P (X < x)$ so that

f (x) = F (x) - F (x - 1) = \frac{x ^{3} - ( x - 1 ) ^{3}}{216} = \frac{[ x - ( x - 1 )] [ x ^{2} + x ( x - 1 ) + ( x - 1 ) ^{2} ]}{216} = \frac{3 x ^{2} - 3 x + 1}{216} for x = 1, 2, 3, 4, 5, 6 ■

Hypergeometric Distribution

Setup

Consider a population of $N$ objects, of which $r$ are considered “successes” (S) and the remaining $N - r$ are considered “failures” (F).

Suppose that a subset of size $n$ is chosen at random from the population without replacement

We say that the random variable $X$ has a hypergeometric distribution if $X$ denotes the number of successes in the subset (shorthand: $X \sim h y p (N, r, n)$ ).

$N$ : Number of objective
$r$ : Number of successes
$n$ : Number of draws

Illustrations

Drawing $2$ balls without replacement from a bag with $3$ blue balls and $4$ red balls. Let $X$ denote the number of blue balls drawn. Then

X \sim h y p (7, 3, 2)

Drawing $5$ cards from a deck of cards. Let $X$ denote the number of aces. Then

X \sim h y p (52, 4, 5)

Definition

Probability Function

Suppose $X \sim h y p (N, r, n)$
$f_{X} (x) = \frac{( x r ) ( n - x N - r )}{( n N )} ma x {0, n - (N - r)} \leq x \leq min {r, n}$

$x \leq min {r, n}$

$x \leq n$ : the number of successes drawn cannot exceed the number drawn
$x \leq r$ : we have at most $r$ success

$x \geq ma x {0, n - (N - r)}$

$x \geq 0$ obviously
$x \geq n - (N - r)$ : if $n$ exceeds the number of failures $N - r$ , we have at least $n - (N - r)$ of successes.

We can verify the probability function of the hypergeometric distribution sums to $1$ .

all x \sum f_{X} (x) = all x \sum \frac{( x r ) ( n - x N - r )}{( n N )} = \frac{1}{( n N )} all x \sum (x r) (n - x N - r) = \frac{( n r + N - r )}{( n N )} = 1

Example

Consider drawing a $5$ -card hand at random from a standard $52$ -card deck. What is the probability that the hand contains at least $3$ K’s?

$X :$ the number of K in hand. Success type: $13$ K’s Failure type: Not K cards $N = 52 r = 13 n = 5$ $X \sim h y p (52, 13, 5)$
$P (X \geq 3) = P (X = 3) + P (X = 4) + P (X = 5) = \frac{( 3 13 ) ( 2 39 )}{( 5 52 )} + \frac{( 4 13 ) ( 1 39 )}{( 5 52 )} + \frac{( 5 13 ) ( 0 39 )}{( 5 52 )} = 0.00175$

Binomial Distribution

Definition

Definition

A Bernoulli trial with probability of success $p$ is an experiment that results in either a success or failure, and the probability of success is $p$ .

Setup

Consider an experiment in which $n$ Bernoulli trials are independently formed, each with probability of success $p$ . $X$ denotes the number of successes from $n$ trials.

X \sim B in o mia l (n, p)

Illustrations

Flip a coin independently $20$ times, let $X$ denote the number of heads observed. Then

X \sim B in o mia l (20, 0.5)

Drawing $2$ balls with replacement from a bag with $3$ blue balls and $4$ red balls. Let $X$ denote the number of blue balls drawn.

X \sim B in o mia l (2, \frac{3}{7})

Assumptions:

Trials are independent
The probability of success, $p$ , is the same in each Bernoulli trial

Definition

Probability Function
$f (x) = (x n) p^{x} (1 - p)^{n - x} x = 0, 1, \dots, n$

Proof that $\sum_{all x} f (x) = 1$ for $0 < p < 1$ :

x = 0 \sum n f (x) = x = 0 \sum n (x n) p^{x} (1 - p)^{n - x} = (1 - p)^{n} x = 0 \sum n (x n) (\frac{p}{1 - p})^{x} = (1 - p)^{n} (1 + \frac{p}{1 - p})^{n} = (1 - p)^{n} (\frac{1 - p + p}{1 - p})^{n} = 1^{n} = 1

If $N$ is very large, and we keep the number of success $r = pN$ where $p \in (0, 1)$ . We choose a relatively small $n$ without replacement from $N$ .

Let $X \sim h y p (N, r, n)$ and $Y \sim B in o mia l (n, p)$ . Then

P (X = k) \approx P (Y = k)

The approximation is good if $N$ and $r$ are large compared to $n$ .

Negative Binomial Distribution

Setup

Consider an experiment in which Bernoulli trials are independently performed, each with probability of success $p$ , until exactly $k$ successes are observed. Then if $X$ denotes the number of failures before observing $k$ successes, we say that $X$ is Negative Binomial with parameters $k$ and $p$ .

X \sim NB (k, p)

Let $Y$ be the number of trials until exactly $k$ successes are observed. We have $Y = X + k$ .

Illustrations

Flip a coin until $5$ heads are observed, and let $X$ denote the number of tails observed. Then

X \sim NB (5, 0.5)

Definition

Probability Function
$f (x) = (k - 1 x + k - 1) p^{k} (1 - p)^{k}, x = 0, 1, 2, \dots$

Proof it is valid

x = 0 \sum \infty f (x) = x = 0 \sum \infty (x - k) (- 1)^{x} p^{k} (1 - p)^{k} = p^{k} x = 0 \sum \infty (x - k) [(- 1) (1 - p)]^{x} = p^{k} [1 + (- 1) (1 - p)]^{- k} if 0 < p < 1 = p^{k} p^{- k} = 1 ■

Geometric Distribution

Setup

Geometric distribution is a special case of the Negative Binomial where we stop after the first success $k = 1$ .

X \sim G eo (p)

Definition

Probability Function
$f (x) = (1 - p)^{x} p, x = 0, 1, 2, 3, \dots$

Geometric Distribution: CDF For an integer $x$ , the CDF of $X$ is

F (x) = P (X \leq x) = t = 0 \sum x (1 - p)^{x} p = p t = 0 \sum x (1 - p)^{x} = p \frac{1 - ( 1 - p ) ^{x + 1}}{1 - ( 1 - p )} = 1 - (1 - p)^{x + 1}

So $F (x) = 1 - (1 - p)^{[x] + 1}$ for $x \geq 0$ , and 0 otherwise.

Example

The number of times I have to roll $2$ dice before I get snake eyes.
$X \sim G eo (\frac{1}{36})$

Poisson Distribution (from Binomial)

As $n \to \infty$ and $p \to 0$ , the binomial function approaches

f (x) \approx e^{- μ} \frac{μ ^{x}}{x !}

for $μ = n p$

Setup

Let $μ = n p$ . Then if $n$ is large, and $p$ is close to zero,

(x n) p^{x} (1 - p)^{n - x} \approx e^{- μ} \frac{μ ^{x}}{x !}

Definition

Probability Function

A random variable $X$ has a Poisson distribution with parameter $μ$ ( $X \sim P o i ss i o n (μ)$ ) if
$f (x) = e^{- μ} \frac{μ ^{x}}{x !}, x = 0, 1, 2, 3, \dots$

Poisson Distribution from Poisson Process

A process satisfying the following $3$ conditions is called a Poisson process.

Independence: The number of occurrences in non-overlapping intervals are independent. For $t > s, (X_{t} - X_{s})$ and $X_{s}$ are independent.
Individuality or Singularity: Events occur singly, not in clusters. $P (2$ or more events in $(t, t + Δ t)) = o (Δ t)$ as $Δ t \to 0$ .
Homogeneity or Uniformity: Events occur at a uniform rate $λ$ (in events per unit of time). $P ($ one event in $(t, t + Δ t)) = λ Δ_{t} + o (Δ t)$ as $Δ t \to 0.$

Little o

A function $g (Δ t)$ is $o (Δ t)$ as $Δ t \to 0$ if

Δ t \to 0 lim \frac{g ( Δ t )}{Δ t} = 0

Poisson Distribution

If $X_{t}$ is a Poisson counting process with a rate of $λ$ per unit. Then,

X_{t} \sim P o i sso n (λ t)

and

f_{t} (x) = \frac{e ^{- λ t} ( λ t ) ^{x}}{x !}, for x = 0, 1, 2, \dots

Examples

Suppose that trucks arrive at a receiving dock with an average arrival rate of $3$ per hour. What is the probability exactly $5$ trucks will arrive in a two-hour period?

We have $λ = 3, t = 2$ .
$P (X_{t} = 5) = \frac{e ^{- (3 \times 2)} ( 3 \times 2 ) ^{5}}{5 !}$

Combining Models

Website hits for a given website occur according to a Poisson process with a rate of $100$ hits per minute. A second is a break if there are no hits in that second.

What is the probability of a break in any given second?

$P (A) = \frac{e ^{- λ} λ ^{0}}{0 !} = e^{\frac{- 100}{60}} = 0.189$

Compute the probability of observing exactly $10$ breaks in $60$ consecutive seconds.

$p = P (A) = 0.189$ , for $X \sim B in o mia l (60, p)$
$P (X = 10 = (10 60) p^{10} (1 - p)^{60 - 10} = 0.124$
Compute the probability that one must wait for $30$ seconds to get $2$ breaks.

Let $Y$ be the seconds we wait for $2$ breaks. We want $P (Y = 30)$ .

Negative Binomial Distribution: $Y - 1 \sim NB (2, p)$
$P (Y = 30) = P (Y - 1 = 29) = (1 29) p^{2} (1 - p)^{30 - 2} = 0.00295$

Expected Value and Variance

Summarizing Data on Random Variables

Median: A value such that half the results are below it and half above it, when the results are arranged in numerical order.

Mode: Value which occurs most often.

Mean: The mean of $n$ outcomes $x_{1}, \dots, x_{n}$ for a random variable $X$ is $\sum_{i = 1}^{n} \frac{x _{i}}{n}$

Expectation of a Random Variable

Definition

Let $X$ be a discrete random variable with $r an g e (X) = A$ and probability function $f (x)$ . The expected value of $X$ is given by
$E (X) = x \in A \sum x f (x)$

Suppose $X$ denotes the outcome of one fair six sided die roll. Compute $E (X)$ .

We know $X \sim U (1, 6) :$ that is, $f (x) = \frac{1}{6} x = 1, 2, \dots, 6$ .

And so,

E (X) = 1 \times \frac{1}{6} + 2 \times \frac{1}{6} + \dots + 6 \times \frac{1}{6} = 3.5

Possible Range

Suppose $X$ is a random variable satisfying $a \leq X \leq b$ for all possible values of $X$ . We have,

a = a x \in A \sum f (x) = x \in A \sum a f (x) a \leq E (x) \leq b \leq x \in A \sum x f (x) \leq x \in A \sum b f (x) = b x \in A \sum f (x) = b

Expectation of $g (X)$

Let $X$ be a discrete random variable with $r an g e (X) = A$ and probability function $f (x)$ . The expected value of some function $g (X)$ of $X$ is given by

E [g (X)] = x \in A \sum g (x) f (x)

Proof

E [g (X)] = \sum y \in B y F_{Y} (y) = y \in B \sum y x \in D_{y} \sum f (x) = y \in B \sum x in D_{y} \sum g (x) f (x) = x \in A \sum g (x) f (x) ■

Linearity Properties of Expectation

For constants $a$ and $b$ ,

E [a g (X) + b] = a E [g (X)] + b

Proof

E [a g (X) + b] = all x \sum [a g (x) + b] f (x) = all x \sum [a g (x) f (x) + b f (x)] = a all x \sum g (x) f (x) + b all x \sum f (x) = a E [g (X)] + b ■

Thus it is also easy to show

E [a g_{1} (X) + b g_{2} (X)] = a E [g_{1} (X)] + b E [g_{2} (X)]

The expectation of a sum is the sum of the expectations.

Means and Variances of Distributions

Expected value of a Binomial random variable

Let $X \sim B in o mia l (n, p)$ . Find $E (X)$ .

E (X) Let y = x - 1 = x = 0 \sum n x (x n) p^{x} (1 - p)^{n - x} = x = 0 \sum n x \frac{n !}{x ! ( n - x )!} p^{x} (1 - p)^{n - x} = x = 1 \sum n \frac{n ( n - 1 )!}{( x - 1 )! [( n - 1 ) - ( x - 1 )]!} p p^{x - 1} (1 - p)^{(n - 1) - (x - 1)} = n p (1 - p)^{n - 1} x = 1 \sum n (x - 1 n - 1) (\frac{p}{1 - p})^{x - 1} = n p (1 - p)^{n - 1} y = 0 \sum n - 1 (y n - 1) (\frac{p}{1 - p})^{y} = n p (1 - p)^{n - 1} (1 + \frac{p}{1 - p})^{n - 1} = n p (1 - p)^{n - 1} \frac{( 1 - p + p ) ^{n - 1}}{( 1 - p ) ^{n - 1}} = n p

Example

If we toss a coin $n = 100$ times, with probability $p = 0.5$ of a head, we’d expect
$E [X] = 100 \times 0.5 = 50 heads$

Hypergeometric Distribution

If $X \sim h y p (N, r, n)$ , then $E [X] = n \frac{r}{N}$

Geometric Distribution

If $X \sim G eo (p)$ , then $E [X] = \frac{1 - p}{p}$

Negative Binomial

If $X \sim NB (k, p)$ , then $E [X] = \frac{( 1 - p )}{p} k$

Poisson Distribution

If $X \sim P o i (μ)$ , then $E [X] = μ$

Variances of Distribution

Using the expected value is one way of predicting the value of a random variable. But we might want to know “how likely will observed data be exactly (or close to) the expected value?”

One may wonder how much a random variable tends to deviate from its mean. Suppose $E [X] = μ$

Expected deviation:

E [(X - μ)] = E [X] - μ = 0

Expected absolute deviation:

E [∣ X - μ ∣] = x \in A \sum ∣ X - μ ∣ f (x)

Expected squared deviation:

E [(X - μ)^{2}] = x \in A \sum (X - μ)^{2} f (x)

Definition

Variance

The variance of a random variable $X$ is denoted $Va r (X)$ , and is defined by
$σ^{2} = Va r (X) = E [(X - E [X])^{2}]$

or sometimes,

Va r (X) = E (X^{2}) - [E (X)]^{2}

We derive this by

Va r (X) = E [(X - E [X])^{2}] = E [X^{2} - 2 E [X] X + (E [X])^{2}] = E [X^{2}] - 2 E [X] E [X] + (E [X])^{2} = E [X^{2}] - (E [X])^{2}

Suppose $X$ satisfies $P (X = 0) = \frac{1}{2} = P (X = 1)$ . What is $Va r (X)$ ?

$E [X] = 0.5 and E [X^{2}] = 0.5$ $Va r (X) = E [X^{2}] - E [X]^{2} = 0.25$

For all random variables $X$ , $Va r (X) \geq 0$ . $Va r (X) = 0$ if and only if $P (X = E [X]) = 1$ . $E [X^{2}] \geq (E [X])^{2}$

Definition

Standard Deviation

The standard deviation of a random variable $X$ is denoted $S D (X)$ , and defined by
$σ = S D (X) = Va r (X)$

Variance of a Linear Transformation

If $a$ and $b$ are constants, and $Y = a X + b$ , then

Va r (Y) = a^{2} Va r (X)

the constant $b$ does not affect anything. $S D (Y) = a S D (X)$

Variances of distributions

X \sim B in o mia l (n, p) X \sim h y p (N, r, n) X \sim G eo (p) X \sim NB (k, p) X \sim P o i (μ) X \sim U (a, b) \to Va r (X) = n p (1 - p) \to Va r (X) = n \frac{r}{N} (1 - \frac{r}{N}) (\frac{N - n}{N - 1}) \to Va r (X) = \frac{( 1 - p )}{p ^{2}} \to Va r (X) = \frac{( 1 - p )}{p ^{2}} k \to Va r (X) = μ \to Va r (X) = \frac{( b - a + 1 ) ^{2} - 1}{12}

Continuous Random Variables

General Terminology and Notation

Definition

A random variable $X$ is said to be continuous if its range is an interval $(a, b) \subset R$ . $X$ is continuous if it can take any value between $a$ and $b$ .

We can’t use $f (x) = P (X = x)$ with continuous distributions because $P (X = x) = 0$ .

Suppose $X$ is a crv with range $[0, 4]$ . We now use integrals instead of sums.

\int_{0}^{4} f (x) d x = 1

Probability Density Function

We say that a continuous random variable $X$ has probability density function $f (x)$ if

f (x) \geq 0 \forall x \in R \int_{- \infty}^{\infty} f (x) d x = 1 P (a \leq X \leq b) = \int_{a}^{b} f (x) d x

For a four number spinner, if $X$ is where the spinner stops, we can define our pdf as:

f (x) = {0.25 if 0 \leq x \leq 4 0 otherwise

We can see that this satisfies $\int_{- \infty}^{\infty} f (x) d x = 1$

\int_{- \infty}^{\infty} f (x) d x = \int_{- \infty}^{0} 0 d x + \int_{0}^{4} 0.25 d x + \int_{4}^{+ \infty} 0 d x = 4 \times 0.25 = 1

Spinner Example

$P (1 \leq X \leq 2)$

P (1 \leq X \leq 2) = \int_{1}^{2} f (x) d x = \int_{1}^{2} 0.25 d x = 0.25 x ∣_{1}^{2} = 0.25 \cdot 2 - 0.25 \cdot 1 = 0.25

Definition

The support of a $p dff (x)$ is defined as
$s u pp (f) = {x \in R : f (x) \neq = 0}$

Example

Suppose that $X$ is a continuous random variable with probability density function
$f (x) = {c x (1 - x) if 0 \leq x \leq 1 0 otherwise$
Compute $c$ so that this is a valid pdf.
$\int_{- \infty}^{\infty} f (x) d x = \int_{0}^{1} c x (1 - x) d x = 1 ⟹ c \int_{0}^{1} x - x^{2} d x = 1 ⟹ x [\frac{x ^{2}}{2} - \frac{x ^{3}}{3}]_{0}^{1} = 1 ⟹ c [\frac{1}{2} - \frac{1}{3}] = 1 ⟹ c = 6$
Compute $P (X > \frac{1}{2})$
$P (X > \frac{1}{2}) = \int_{1/2}^{1} 6 x (1 - x) d x = 6 [\frac{x ^{2}}{2} - \frac{x ^{3}}{3}]_{1/2}^{1} = 6 ([\frac{1}{2} - \frac{1}{3}] - [\frac{1/4}{2} - \frac{1/8}{3}]) = 6 [\frac{1}{6} - \frac{1}{12}] = \frac{1}{2}$

Note

P (X = a) \neq = f (x) P (a \leq X \leq b) = P (a < X < b) = P (a \leq X < b) = P (a < x \leq b)

Definition

The Cumulative Distribution Function of a random variable $X$ is
$F (x) = P (X \leq x)$

If $X$ is continuous with pdf $f (x)$ , then

F (x) = \int_{- \infty}^{x} f (u) d u

By the fundamental theorem of calculus

f (x) = \frac{d}{d x} F (x)

Properties:

$F (x)$ is defined for all real $x$
$F (x)$ is a non-decreasing function of $x$ for all real $x$
$lim_{x \to - \infty} F (x) = 0$ and $lim_{x \to \infty} F (x) = 1$
$P (a < X \leq b) = F (b) - F (a)$
Since $P (X = a) = 0$ , we have $P (a \leq X \leq b) = F (b) - F (a)$

Spinner example,
$F (x) = \int_{- \infty}^{x} f (u) d u = \int_{0}^{x} 0.25 d x = 0.25 x$
Thus,
$F (x) = ⎩ ⎨ ⎧ 0 if x < 0 0.25 x 0 \leq x < 4 1 x \geq 4$
We can then get the pdf by
$f (x) = \frac{d}{d x} F (x) = \frac{d}{d x} 0.25 x = 0.25$

General approach to find $F (x)$ from $f (x)$

Treat each piece of $f (x)$ separately
Note $F (x) = 0$ for $x <$ the minimum value in the support of $f (x)$
Note $F (x) = 1$ for $x \geq$ the maximum value in the support of $f (x)$
Find $F (x) = \int_{- \infty}^{x} f (u) d u$

General approach to find $f (x)$ from $F (x)$

Treat each piece of $F (x)$ separately
Find $f (x) = \frac{d}{d x} F (x)$

Note

P (a \leq X \leq b) = F (b) - F (a) = \int_{- \infty}^{b} f (x) d x - \int_{- \infty}^{a} f (x) d x = \int_{a}^{b} f (x) d x

Quantile

Suppose $X$ is a continuous random variable with CDF $F (x)$ . The $p^{th}$ quantile of $X$ is the value $q (p)$ such that

P (X \leq q (p)) = p

If $p = 0.5$ , then $q (0.5)$ is the median of X. We can find a given quantile by solving $F (x) = p$ for $x$ .

Change of Variables

What if we want to find the CDF or pdf of a function of $X$ ? For the spinner example, the winning is inverse of the point we spin. Hence, the winning is $Y = \frac{1}{X}$ , and we want to find

F_{Y} (y) = P (Y \leq y)

Specific example

P (Y \leq 2) = P (X^{- 1} \leq 2) = P (X \geq 0.5) = \frac{7}{8}

How can we generalize this approach to $P (Y \leq y)$

The process

Find the range of values of $Y$
Write the CDF of $Y$ as a function of $X$
Use $F_{X} (x)$ to find $F_{Y} (y)$
Differentiate $F_{Y} (y)$ if we want the pdf of $Y$ , $f_{Y} (y)$

Spinner example:

Since $X \in [0, 4]$ , we know $Y \in [\frac{1}{4}, \infty]$

Write the CDF of $Y$ as a function of $X$ . Let $y \in [\frac{1}{4}, \infty)$ .
$F_{Y} (y) = P (Y \leq y) = P (\frac{1}{X} \leq y) = P (X \geq \frac{1}{y}) = 1 - P (X < \frac{1}{y}) = 1 - F_{X} (y^{- 1})$
Then use $F_{X} (x)$ to find $F_{Y} (y)$ .

$F_{Y} (y) = 1 - F_{X} (Y^{- 1})$ , and we know
$F_{X} (x) = \frac{x}{4} x \in [0, 4] F_{X} (y^{- 1}) = \frac{y ^{- 1}}{4} = \frac{1}{4 y}$
Thus,
$F_{Y} (y) = {0 y < \frac{1}{4} 1 - \frac{1}{4 y} y \geq \frac{1}{4}$
Differentiate $F_{Y} (y)$ if we want the pdf of $Y$ , $f_{Y} (y)$ . We have $F_{Y} (y) = 1 - \frac{1}{4 y}$ for $y \geq \frac{1}{4}$ , so the pdf is
$f_{Y} (y) = \frac{d}{d y} F_{Y} (y) = \frac{1}{4 y ^{2}}, y \geq \frac{1}{4}$

Expectation, Mean, and Variance for Continuous Random Variables

If $X$ is a continuous random variable with $p dff (x)$ , and $g : R \to R$ , then

E (g (X)) = \int_{- \infty}^{\infty} g (x) f (x) d x

Thus,

E (X) = \int_{- \infty}^{\infty} x f (x) d x Va r (X) = E ([X - E (X)]^{2}) = \int_{- \infty}^{\infty} (x - E (X))^{2} f (x) d x

Note that the shortcut still holds.

Example

f (x) = {6 x (1 - x) if 0 \leq x \leq 1 0 otherwise

E (X) = \int_{- \infty}^{\infty} x f (x) d x = 0 + \int_{0}^{1} x \cdot 6 x (1 - x) d x + 0 = 6 [\frac{1}{3} - \frac{1}{4}]_{0}^{1} = 0.5

Now solve for $Va r (X)$

E (g (X)) E (X^{2}) = \int_{- \infty}^{\infty} g (x) f (x) d x = 0 + \int_{0}^{1} x^{2} 6 x (1 - x) d x + 0 = \int_{0}^{1} (6 x^{3} - 6 x^{4}) d x = 6 [\frac{x ^{4}}{4} - \frac{x ^{5}}{5}]_{0}^{1} = 6 [\frac{1}{4} - \frac{1}{5}] = 0.3

Thus,

Va r (X) = E [X^{2}] - (E [x])^{2} = 0.3 - 0.25 = 0.05

Continuous Uniform Distribution

Definition

We say that $X$ has a continuous uniform distribution on $(a, b)$ if $X$ takes values in $(a, b)$ (or $[a, b]$ ) where all subintervals of a fixed length have the same probability.

$X$ has pdf
$f (x) = {\frac{1}{b - 1} x \in (a, b) 0 otherwise$
then the CDF is
$F (x) = ⎩ ⎨ ⎧ 0 \int_{a}^{x} \frac{1}{b - a} d u = \frac{x - a}{b - a} 1 x < a a \leq x \leq b x > b$

Continuous Uniform: Mean and Variance

E (X) = \frac{( a + b )}{2}

E (X^{2}) = \int_{- \infty}^{\infty} x^{2} f (x) d x = \int_{a}^{b} x^{2} \frac{1}{b - a} d x = \frac{1}{( b - a )} (\frac{x ^{3}}{3} ∣_{a}^{b}) = \frac{b ^{3} - a ^{3}}{3 ( b - a )} = \frac{( b - a ) ( b ^{2} + ab + a ^{2} )}{3 ( b - a )} = \frac{b ^{2} + ab + a ^{2}}{3}

Va r (X) = E (X^{2}) - [E (X)]^{2} = \frac{b ^{2} + ab + a ^{2}}{3} - (\frac{b + a}{2})^{2} = \frac{4 b + 4 ab + 4 a ^{2} - 3 b ^{2} - 6 ab - 3 a ^{2}}{12} = \frac{b ^{2} - 2 ab + a ^{2}}{12} = \frac{( b - a ) ^{2}}{12}

Exponential Distribution

Consider cars arriving following a Poisson process. The number of cars arriving in $t$ minutes follows $P o i (λ t)$

Let $X =$ the time you wait before you see the first car, in minutes. This is a crv.

CDF

F (x) = P (X \leq x) = P (time to 1st event > x) = 1 - P (no event occurs in (0, x)) = 1 - P (Y_{x} = 0)

where $Y_{x} \sim P o i (λ x)$ is the number of events in $(0, x)$ .

Thus,

F (x) = 1 - \frac{e ^{- λ x} ( λ x ) ^{0}}{0 !} = 1 - e^{- λ x}

Taking the derivative gives the pdf:

f (x) = {λ e^{- λ x} x > 0 0 x \leq 0

Exponential Distribution

Let $θ = \frac{1}{λ}$ provide an alternate parameterization of the exponential distribution.

Definition

We say that $X$ has an exponential distribution with parameter $θ$ $(X \sim e x p (θ))$ if the density of $X$ is
$f (x) = {\frac{1}{θ} e^{- x / θ} x > 0 0 x \geq 0$

The CDF becomes $F (x) = 1 - e^{x / θ}$ for $x \geq 0$ .

In general, for any Poisson process with rate $λ$ , the time between events will follow an exponential distribution $e x p (θ = 1/ λ)$

Now we want to compute $E [X]$ and $Va r (X)$ . We will need to solve

E [X] = \int_{- \infty}^{\infty} x f (x) d x = \int_{0}^{\infty} x \frac{1}{θ} e^{- x / θ} d x

and

E [X^{2}] = \int_{0}^{\infty} x^{2} \frac{1}{θ} e^{- x / θ} d x

We can use the Gamma function.

Definition

The Gamma function, $Γ (α)$ is defined for all $α > 0$ as
$Γ (α) = \int_{0}^{\infty} y^{α - 1} e^{- y} d y$

Note

$Γ (α) = (α - 1) Γ (α - 1)$
If $a \in Z^{+}$ , then $Γ (α) = (α - 1)!$

The Gamma function tells us that if $α$ is a positive integer, then

\int_{0}^{\infty} y^{α - 1} e^{- y} d y = (α - 1)!

If we write $y = \frac{x}{θ}$ , then $d x = θ d y$ and

E [X] = \int_{0}^{\infty} x \frac{1}{θ} e^{- x / θ} d x = \int_{0}^{\infty} y e^{- y} θ d y = θ \int_{0}^{\infty} y^{1} e^{- y} d y

Because

Γ (α) = \int_{0}^{\infty} y^{α - 1} e^{- y} d y = (α - 1)!

E [X] = θ \int_{0}^{\infty} y^{1} e^{- y} d y = θ Γ (2) = θ (1!) = θ

We can use the same trick with Variance to see that $Va r (X) = θ^{2}$

Memoryless Property

P (X > t + x ∣ X > t) P (a \leq X - t \leq b ∣ X > t) = P (X > x) = P (a \leq X \leq b)

Computer Generated Random Numbers

Transform simulated observations from $U \sim U (0, 1)$ to obtain observations from $X$ with CDF $F$ . Find a function $h$ such that $X = h (U)$ .

F (x) = P (X \leq x) = P (h (U) \leq x) = P (U \leq h^{- 1} (x)) assuming h is strictly increasing = h^{- 1} (x) given the CDF of U \sim U (0, 1)

Assuming $F$ is continuous and strictly increasing, our result $F (\cdot) = h^{- 1} (\cdot)$ implies

h = F^{- 1}

Example

Consider the CDF of the $g eo (p)$ distribution
$F (x) = {1 - (1 - p)^{[x] + 1} x \geq 0 0 x < 0$
Find a transformation $h$ so that if $U \sim U (0, 1), X = h (u)$ has CDF $F$ .

CDF is not continuous nor strictly increasing so we have to use the generalized inverse $F^{- 1} (u) = in f {x; F (x) \geq u}$ . For a given $0 \leq u \leq 1$ :
$F (x) 1 - (1 - p)^{[x] + 1} (1 - p)^{[x] + 1} ([x] + 1) lo g (1 - p) [x] \geq u \geq u \leq 1 - u \leq lo g (1 - u) \geq \frac{lo g ( 1 - u )}{lo g ( 1 - p )} - 1$
The smallest $x$ that satisfies the equation above is
$x = ⌈ \frac{lo g ( 1 - u )}{lo g ( 1 - p )} - 1 ⌉$
Thus $h (u) = in f {x; F (x) \geq u} = ⌈ \frac{l o g ( 1 - u )}{l o g ( 1 - p )} - 1 ⌉$

Normal Distribution

Characteristics: Symmetric about a mean value, more concentrated around the mean than the tails (and unimodal).

Definition

$X$ is said to have a normal distribution (or Gaussian distribution) with mean $μ$ and variance $σ^{2}$ if the density of $X$ is
$f (x) = \frac{1}{2 π σ ^{2}} e^{\frac{- ( x - μ ) ^{2}}{2 σ ^{2}}}, x \in R$
or $X \sim N (μ, σ^{2})$ , or $X \sim G (μ, σ)$ .

Properties:

Symmetric about its mean: If $X \sim N (μ, σ^{2})$ , then $P (X \leq μ - t) = P (X \geq μ + t)$
Density of unimodal: Peak is at $μ$ . The mode, median, and mean are the same $μ$ .
Mean and Variance are the parameters: $E (X) = μ$ , and $Va r (X) = σ^{2}$

A classic problem, if $X \sim N (μ, σ^{2})$ , then what is the value of

P (a \leq X \leq b) = \int_{a}^{b} \frac{1}{2 π σ ^{2}} e^{\frac{- ( x - μ ) ^{2}}{2 σ ^{2}}} d x = ???

This integral is weird.

Definition

We say that $X$ is a standard normal random variable if $X \sim N (0, 1)$ . The probability density function is:
$φ (x) = \frac{1}{2 π} e^{\frac{- x ^{2}}{2}}$
The CDF is
$ϕ (x) = \int_{- \infty}^{x} \frac{1}{2 π} e^{\frac{- y ^{2}}{2}} d y$

Standard Normal Tables (Z-Tables). Values of the function $ϕ (x)$ are tabulated in Standard Normal Tables.

We use symmetry:

⟹ ⟹ ⟹ P (∣ Z ∣ \leq c) = 0.2 P (Z \leq - c) + P (Z \geq c) = 0.8 P (Z \geq c) = 0.4 P (Z \leq c) = 0.6

Standardization

Theorem

If $X \sim N (μ, σ^{2})$ , then defining
$Z = \frac{X - μ}{σ}$
we have $Z \sim N (0, 1)$ .

An important consequence of $X \sim N (μ, σ^{2})$ :

P (X \leq x) = P (\frac{X - μ}{σ} \leq \frac{x - μ}{σ}) = P (Z \leq \frac{x - μ}{σ})

where $Z \sim N (0, 1)$

The process to find $P (Z \leq x)$ when $X \sim N (μ, σ^{2})$ :

Compute $\frac{x - μ}{σ}$
Use standard normal tables to find $P (Z \leq \frac{x - μ}{σ})$
This equals $P (X \leq x)$

P (X > x) P (a < X < b) = P (Z > \frac{x - μ}{σ}) = P (\frac{a - μ}{σ} < Z < \frac{b - μ}{σ})

where $Z \sim N (0, 1)$ .

Let $X \sim N (μ, σ)$ . Find $a$ such that $P (X \leq a) = 0.6$ ; this is the $60$ th percentile of $Z$ .

P (\frac{X - μ}{σ} \leq \frac{a - μ}{σ}) = 0.6 ⟹ \frac{a - μ}{σ} = Φ^{- 1} (0.6)

So $a = σ Φ^{- 1} (0.6) + μ$

Multivariate Distributions

Basic Terminology and Techniques

So far we’ve only considered univariate distributions.

Suppose that $X$ and $Y$ are discrete random variables defined on the sample space. The joint probability function of $X$ and $Y$ is

f (x, y) = P ({X = x} \cap {Y = y}) x \in X (S), y \in Y (S)

a shorthand is,

f (x, y) = P (X = x, Y = y)

For a collection of $n$ discrete random variables, $X_{1}, \dots, X_{n}$ , the joint probability function is defined as

f (x_{1}, x_{2}, \dots, x_{n}) = P (X_{1} = x_{1}, X_{2} = x_{2}, \dots, X_{n} = x_{n})

For example if we rolled three dice, and let $X_{i}$ denote the result on the $i^{th}$ die, we’d have $f (x_{1}, x_{2}, x_{3}) = \frac{1}{216}$

Properties:

$0 \leq f (x, y) \leq 1$
$\sum_{x, y} f (x, y) = 1$

Computing probability from the joint probability function:

Let $A$ be a subset of $(x, y)$ values that $(X, Y)$ could take. Then

P ((X, Y) \in A) = (x, y) \in A \sum f (x, y)

Definition

Suppose that $X$ and $Y$ are discrete random variables with joint probability function $f (x, y)$ . The marginal probability function of $X$ is
$f_{X} (x) = P (X = x) = y \in Y (S) \sum f (x, y), x \in X (S)$
Similarly, the marginal distribution of $Y$ is
$f_{Y} (y) = P (Y = y) = x \in X (S) \sum f (x, y), y \in Y (S)$

Definition

Suppose that $X$ and $Y$ are discrete random variables with joint probability function $f (x, y)$ and marginal probability functions $f_{X} (x)$ and $f_{Y} (y)$ . $X$ and $Y$ are said to be independent random variables if and only if
$f (x, y) = f_{X} (x) f_{Y} (y), \forall x \in X (S), y \in Y (S)$
This is the same as saying
$P (X = x, Y = y) = P (X = x) P (Y = y), \forall x, y$

Rolling two 6-sided dice with $X =$ the outcome on the first die and $Y =$ the outcome on the second die:

f (x, y) = \frac{1}{36}, x, y \in {1, 2, 3, 4, 5, 6}

but we also know

f_{X} (x) = \frac{1}{6}, f_{Y} (y) = \frac{1}{6} x, y \in {1, 2, 3, 4, 5, 6}

and so $f (x, y) = \frac{1}{36} = \frac{1}{6} \times \frac{1}{6} = f_{X} (x) f_{Y} (y)$ and we have independence.

This abstracts to size $n$ .

Definition

The conditional probability function of $X$ given $Y = y$ is denoted $f_{X ∣ Y} (x ∣ y)$ , and is defined to be

f_{X ∣ Y} (x ∣ y) = P (X = x ∣ Y = y) = \frac{P ( X = x , Y = y )}{P ( Y = y )} = \frac{f ( x , y )}{f _{Y} ( y )}

What about functions of random variables?

Suppose that

h : R^{2} \to R

For jointly distributed random variables $X$ and $Y$ , $U = h (X, Y)$ is a random variable. If $X$ and $Y$ have joint p.f. $f (x, y)$ , then the probability function of $U$ is given by:

f_{U} = P (U = t) = (x, y) : h (x, y) = t \sum f (x, y)

If $X \sim B in o mia l (n, p)$ and $Y \sim B in o mia l (m, p)$ independently, then $X + Y \sim B in o mia l (n + m, p)$
If $X \sim P o i (μ_{1})$ and $Y \sim P o i (μ_{2})$ independently, then $X + Y \sim P o i (μ_{1} + μ_{2})$

Multinomial Distribution

Definition

Multinomial Distribution: Consider an experiment in which:

Individual trials have $k$ possible outcomes, and the probabilities of each individual outcome are denoted $p_{i}, i = 1, \dots, k$ , so that $p_{1} + p_{2} + \dots + p_{k} = 1$

Trials are independently repeated $n$ times, with $X_{i}$ denoting the number of times outcome $i$ occurred, so that $X_{1} + X_{2} + \dots X_{k} = n$

If $X_{1}, \dots, X_{k}$ have multinomial distribution with parameters $n$ and $p_{1}, \dots, p_{k}$ , then their joint probability function is
$f (x_{1}, \dots x_{k}) = \frac{n !}{x _{1} ! x _{2} ! \dots x _{k} !} p_{1}^{x_{1}} \dots p_{k}^{x_{k}},$
where $x_{1}, \dots, x_{k}$ satisfy $x_{1} + \dots + x_{k} = n, x_{i} \geq 0$ .

The terms

\frac{n !}{x _{1} ! x _{2} ! \dots x _{k} !}, for x_{1} + \dots + x_{k} = n

are called multinomial coefficients.

If $X_{1}, \dots, X_{k}$ have joint Multinomial distribution with parameters $n$ and $p_{1}, \dots, p_{k}$ , then

X_{i} \sim B in (n, p_{i})

Also

\sum X_{i} \sim B in (n, \sum p_{i})

Expectation for Multivariate Distributions: Covariance and Correlation

Recall,

E [X] = \sum x f (x) E [X] = \int x f (x) d x

Definition

Suppose $X$ and $Y$ are jointly distributed random variables with joint probability function $f (x, y)$ . Then for a function $g : R^{2} \to R$ ,
$E (g (X, Y)) = (x, y) \sum g (x, y) f (x, y)$

More generally, if $g : R^{n} \to R$ , and $X_{1}, \dots, X_{n}$ have joint p.f. $f (x_{1}, \dots, x_{n})$ , then

E (g (X_{1}, \dots, X_{n})) = (x_{1}, \dots, x_{n}) \sum g (x_{1}, \dots, x_{n}) f (x_{1}, \dots, x_{n})

Linear Properties of Expectation

E (X + Y) = x \sum x f_{X} (x) + y \sum y f_{Y} (y) = E (X) + E (Y)

and in general,

E [a \cdot g_{1} (X, Y) + b \cdot g_{2} (X, Y)] = a \cdot E (g_{1} (X, Y)) + b \cdot E (g_{2} (X, Y))

Independence gives a useful - but simplistic - way of describing relationships between variables.

Definition

If $X$ and $Y$ are jointly distributed, then $C o v (X, Y)$ denotes the covariance between $X$ and $Y$ . It is defined by
$C o v (X, Y) = E [(X - E (X)) (Y - E (Y))]$
or the shortcut,
$C o v (X, Y) = E (X Y) - E (X) E (Y)$

Properties of Covariance

Positive $C o v (X, Y) ⟹ Y$ increases as $X$ increases
Negative $C o v (X, Y) ⟹ Y$ decreases as $X$ increases
The larger the absolute value of $C o v (X, Y)$ , the stronger the relationship is
$C o v (X, X) = Va r (X) . C o v (X, X) = E [XX] - E [X] E [X] = E [X^{2}] - E [X]^{2}$
$C o v (X, c) = 0$ for any constant $c$ .
$C o v (Y, X) = C o v (X, Y) = E [(Y - μ_{Y}) (X - μ_{X})]$

Theorem

If $X$ and $Y$ are independent, then $C o v (X, Y) = 0$ .

Proof

C o v (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})] = all y \sum [all x \sum (x - μ_{X}) (y - μ_{Y}) f_{1} (x) f_{2} (y)] = all y \sum [(y - μ_{Y}) f_{2} (y) all x \sum (x - μ_{X}) f_{1} (x)] = all y \sum [(y - μ_{Y}) f_{2} (y) E (X - μ_{X})] = all y \sum 0 = 0 ■

$X$ and $Y$ are independent $⟹$ $C o v (X, Y) = 0$ .

It is important to know that the converse is false. If $C o v (X, Y) = 0$ then $X$ and $Y$ are not necessarily independent.

$C o v (X, Y) = 0 ⟺ E [X Y] = E [X] E [Y]$

$X$ and $Y$ are uncorrelated if $C o v (X, Y) = 0$ , but that does not mean they are independent.

Independence $⟹ C o v (X, Y) = 0 ⟹ E [X Y] = E [X] E [Y]$ Independence $⟹ E [g_{1} (X) g_{2} (Y)] = E [g_{1} (X)] E [g_{2} (Y)]$

Definition

The correlation of $X$ and $Y$ is denoted $corr (X, Y)$ , and is defined by
$ρ = \frac{C o v ( X , Y )}{S D ( X ) S D ( Y )}$
with $- 1 \leq ρ \leq 1$ .

Correlation measures the strength of the linear relationship between $X$ and $Y$ .

Properties:

If $p \approx + 1$ , then $X$ and $Y$ will have an approximately positive linear relationship
If $p \approx - 1$ , then $X$ and $Y$ will have an approximately negative linear relationship
If $p \approx 0$ , then $X$ and $Y$ are said to be uncorrelated.

We say that $X$ and $Y$ are uncorrelated if $C o v (X, Y) = 0$ (or $corr (X, Y) = 0$ ).

$X$ and $Y$ are independent $⟹$ $X$ and $Y$ are uncorrelated.

Once again, the converse is not true.

Mean and Variance of a Linear Combination of Random Variables

Suppose $X_{1}, \dots, X_{n}$ are jointly distributed random variables with joint probability function $f (x_{1}, \dots, x_{n})$ . A linear combination of the random variables $X_{1}, \dots, X_{n}$ is a random variable of the form

i = 0 \sum n a_{i} X_{i}

where $a_{1}, \dots, a_{n} \in R$

Examples
$S_{n} \overset{ˉ}{X} = i = 1 \sum n X_{i} a_{i} = 1, 1 \leq i \leq n = i = 1 \sum n \frac{1}{n} X_{i} a_{i} = \frac{1}{n}, 1 \leq i \leq n$

Expected Value of a Linear Combination:

E (i = 1 \sum n a_{i} X_{i}) = i = 1 \sum n a_{i} E (X_{i})

Mean of sample mean:

E [\overset{ˉ}{X}] = i = 1 \sum n \frac{1}{n} E [X_{i}] = \frac{1}{n} n μ = μ

Covariance of linear combinations:

Two useful results:

C o v (a X + bY, c U + d V) C o v (i = 1 \sum n a_{i} X_{i}, j = 1 \sum m b_{j} Y_{j}) = a c C o v (X, U) + a d C o v (X, V) + b c C o v (Y, U) + b d C o v (Y, V) = i = 1 \sum n j = 1 \sum m a_{i} b_{j} C o v (X_{i}, Y_{j})

Variance of linear combination

When $C o v (X, Y) = 0$

$Va r (X + Y) = Va r (X) + Va r (Y)$

$Va r (X - Y) = Va r (X) + Va r (Y)$

When $C o v (X, Y) > 0$

$Va r (X + Y) > Va r (X) + Va r (Y)$

$Va r (X - Y) < Va r (X) + Va r (Y)$

In the case of two random variables $X$ and $Y$ , and constants $a$ and $b$ , we have:

Va r (a X + bY) = a^{2} Va r (X) + b^{2} Va r (Y) + 2 ab C o v (X, Y)

If $a = b = 1$

Va r (X + Y) = Va r (X) + Va r (Y) + 2 C o v (X, Y)

If $a = 1, b = - 1$

Va r (X - Y) = Va r (X) + Va r (Y) - 2 C o v (X, Y)

In general, let $X_{1}, X_{2}, \dots, X_{n}$ be random variables, and write $Va r (X_{i}) = σ_{i}^{2}$ , then:

Va r (i = 1 \sum n a_{i} X_{i}) = i = 1 \sum n j = 1 \sum n a_{i} a_{j} C o v (X_{i}, X_{j}) = i = 1 \sum n a_{i}^{2} σ_{i}^{2} + 2 i = 1 \sum n j = i + 1 \sum n a_{i} a_{j} C o v (X_{i}, X_{j})

If $X_{1}, X_{2}, \dots, X_{n}$ are independent, then $C o v (X_{i}, X_{j}) = 0$ (for $i \neq = j$ ) and so

Va r (i = 1 \sum n a_{i} X_{i}) = i = 1 \sum n a_{i}^{2} Va r (X_{i}) = i = 1 \sum n a_{i}^{2} σ_{i}^{2}

Variance of sample mean

In general, we have for $X_{1}, X_{2}, \dots, X_{n}$ independent random variables all with variance $σ^{2}$

Va r (\overset{ˉ}{X}) = \frac{σ ^{2}}{n}

this holds for any set of independent random variables that have the same variance.

Linear Combinations of Independent Normal Random Variables

Linear transformation of a normal random variable:

Let $X \sim N (μ, σ^{2})$ and $Y = a X + b$ where $a$ and $b$ are constants, then

Y \sim N (a μ + b, a^{2} σ^{2})

The linear transformation of a normal random variable is still normal.

Linear combination of $2$ independent normal random variables:

Let $X \sim N (μ_{1}, σ_{1}^{2})$ and $Y \sim N (μ_{2}, σ_{2}^{2})$ independently, and let $a$ and $b$ be constants, then

a X + bY \sim N (a μ_{1} + b μ_{2}, a^{2} σ_{1}^{2} + b^{2} σ_{2}^{2})

Linear combination of independent normal random variables is still normal.

Let $X_{1}, X_{2}, \dots, X_{n}$ be independent $N (μ, σ^{2})$ random variables. Then

S_{n} \equiv i = 1 \sum n X_{i} \sim N (n μ, n σ^{2})

and

\overset{ˉ}{X} \sim N (μ, σ^{2} / n)

The IQs of UWaterloo Math students are normally distributed with mean $120$ and variance $100$ . The probability that the average IQ in a class of $25$ students is between $118$ and $123$ is:
$⟹ P (118 \leq \overset{ˉ}{X} \leq 123) X_{i} \sim N (120, 100) ⟹ \overset{ˉ}{X} \sim N (120, 100/25) = P (\frac{118 - 120}{2} \leq \frac{X ˉ - 120}{2} \leq \frac{123 - 120}{2}) = P (- 1 \leq Z \leq 1.5)$

Indicator Random Variables

If $X \sim B in o mia l (n, p)$ we can think of $X$ in the following way:

Observe the first trial, and see if it succeeds. We set $X_{1}$ to be $1$ if it succeeds and $0$ if it fails.

Observe the second trial, and see if it succeeds. We set $X_{2}$ to be $1$ if it succeeds and $0$ if it fails.

Observe the $n^{th}$ trial, and see if it succeeds. We set $X_{n}$ to be $1$ if it succeeds and $0$ if it fails.

Sum the successful events to find $X$

X = i = 1 \sum n I_{i}

$X$ is a linear combination of random variables.

Definition

Let $A$ be an event which may possibly result from an experiment. We say that $1_{A}$ is the indicator random variable of the event $A$ . $1_{A}$ is defined by:
$1_{A} = {1 if A occurs 0 otherwise$

Covariance of Indicator Random Variables:

Suppose we have two events, $A$ and $B$ , and we define

1_{A} = 1 if A occurs, and 1_{A} = 0 otherwise 1_{B} = 1 if B occurs, and 1_{B} = 0 otherwise

How do we find $C o v (1_{A}, 1_{B})$

For general, $X, Y$

C o v (X, Y) = E [X Y] - E [X] E [Y]

so for indicator variables we have

C o v (1_{A}, 1_{B}) = E [1_{A} \times 1_{B}] - E [1_{A}] E [1_{B}]

We know that $E [1_{A}] = P (A)$ and $E [1_{B}] = P (B)$ , so we just have to find

E [1_{A} \times 1_{B}] = 1 \times P (A \cap B) + 0 \times (1 - P (A \cap B)) = P (A \cap B)

If $A$ and $B$ are events, then

C o v (1_{A}, 1_{B}) = P (A \cap B) - P (A) P (B)

$1_{A}$ and $1_{B}$ are independent $⟺ 1_{A}$ and $1_{B}$ are uncorrelated.

Central Limit Theorem and Moment Generating Functions

Central Limit Theorem

If $X_{1}, X_{2}, \dots, X_{n}$ are independent random variables from the same distribution, with mean $μ$ and variance $σ^{2}$ , then as $n \to \infty$ , the shape of the probability histogram for the random variable $S_{n} = \sum_{i = 1}^{n} X_{i}$ approaches the shape of a $N (n μ, n σ^{2})$ probability density function.

The cumulative distribution function of the random variable

\frac{\sum _{i = 1}^{n} X _{i} - n μ}{σ n} = \frac{S _{n} - n μ}{σ n}

approaches the $N (0, 1)$ cumulative distribution function. Similarly, the cumulative distribution function of

\frac{X ˉ - μ}{σ / n}

approaches the $N (0, 1)$ cumulative distribution function.

In other words, if $n$ is large:

S_{n} = i = 1 \sum n X_{i}

has approximately a $N (n μ, n σ^{2})$ distribution, and

\overset{ˉ}{X} = \frac{1}{n} i = 1 \sum n X_{i}

has approximately a $N (μ, σ^{2} / n)$ distribution.

Example

Roll a $6$ -sided die $1000$ times and record each result. If the die is a fair die, estimate the probability that the sum of the die rolls is less than $3400$ .

Let $X_{i}$ be the dot for the $i -$ th roll. We want the probability $P (\sum_{i = 1}^{1000} X_{i} < 3400)$ . Without CLT this is very difficult.

Using CLT:

Seems reasonable to assume independence, each $X_{i}$ is a discrete $U (1, 6)$ random variable, and $n = 1000$ is large.

Mean and variance
$μ = \frac{a + b}{2} = 3.5 E [S_{n}] = n μ = 3.5 n σ^{2} = \frac{( b - a + 1 ) ^{2} - 1}{12} = \frac{35}{12} Va r (S_{n}) = n σ^{2} = \frac{35}{12} n$
Apply CLT (Standardization)
$\frac{S _{n} - 3.5 n}{\frac{35}{12} n} \to N (0, 1)$
Use the Z-table
$P (S_{n} < 3400) = P \frac{S _{n} - 3.5 n}{\frac{35}{12} n} < \frac{3400 - 3.5 n}{\frac{35}{12} n} \approx P (Z < - 1.852) = 0.032$

Note:

CLT doesn’t hold if $μ$ and/or $σ^{2}$ don’t exist
Accuracy depends on the size of $n$ and the actual distribution of the $X_{i}$
CLT works for any distribution of $X_{i}$

Normal approximation to binomial

Note that $S_{n} = \sum_{i = 1}^{n} X_{i} \sim B in o mia l (n, p)$ if all $X_{i} \sim B in o mia l (1, p)$ are independent.

Then if $S_{n} \sim B in o mia l (n, p)$ then for large $n$ , the random variable

Z = \frac{S _{n} - n p}{n p ( 1 - p )}

has approximately a $N (0, 1)$ .

Continuity Correction

We need to be careful with CLT when working with discrete random variables. For example we’ve computed $P (15 \leq S_{n} \leq 20)$ . But we know that $X$ can’t take non-integer values. Therefore, $P (14.5 \leq S_{n} \leq 20.5)$ gives us a better estimate. This is called continuity correction.

We should apply this when approximating discrete distributions using the CLT, and not continuous distributions.

$P (a \leq X \leq b) \to P (a - 0.5 \leq X \leq b + 0.5)$
$P (X \leq b) \to P (X \leq b + 0.5)$
$P (X < b) \to P (X < b - 0.5)$
$P (X \geq a) \to P (X \geq a - 0.5)$
$P (X > a) \to P (X > a + 0.5)$
$P (X = x) \to P (x - 0.5 \leq X \leq x + 0.5)$

If $X \sim P o i sso n (μ)$ , then the cumulative distribution function of

Z = \frac{X - μ}{μ}

approaches that of a $N (0, 1)$ random variable as $μ \to \infty$

Moment Generating Functions

This is the third type of function that uniquely determines a distribution.

Definition

The moment generating function (MGF) of a random variable $X$ is given by
$M_{X} (t) = E (e^{tX}), t \in R$

If $X$ is discrete with probability function $f (x)$ , then

M_{X} (t) = x \sum e^{t x} f (x), t \in R

If $X$ is continuous with density $f (x)$

M_{X} (t) = \int_{- \infty}^{\infty} e^{t x} f (x) d x, t \in R

Properties:

M_{X} (t) = 1 + tE (X) + \frac{t ^{2} E ( X ^{2} )}{2 !} + \frac{t ^{3} E ( X ^{3} )}{3 !} + \dots

So long as $M_{X} (t)$ is defined in a neighbourhood of $t = 0$

\frac{d}{d t ^{k}} M_{X} (0) = E (X^{k})

Suppose $X$ has a $B in o mia l (n, p)$ distribution. Then its moment generating function is

M (t) = x = 0 \sum n e^{t x} (x n) p^{x} (1 - p)^{n - x} = x = 0 \sum n (x n) (p e^{t})^{x} (1 - p)^{n - x} = (p e^{t} + 1 - p)^{n}

Therefore,

M^{'} (t) M^{''} (t) = n p e^{t} (p e^{t} + 1 - p)^{n - 1} = n p e^{t} (p e^{t} + 1 - p)^{n - 1} + n (n - 1) p^{2} e^{2 t} (p e^{t} + 1 - p)^{n - 2}

E (X) E (X^{2}) Va r (X) = M^{'} (0) = n p = M^{''} (0) = n p + n (n - 1) p^{2} = E (X^{2}) - E (X)^{2} = n p (1 - p)

What is the Moment Generating Function for $I_{A}$ ?

The distribution of $I_{A}$ is

P (I_{A} = 1) M_{I_{A}} (t) = P (A), P (I_{A} = 0) = 1 - P (A) = E [e^{t I_{A}}] = e^{t \times 0} P (I_{A} = 0) + e^{t \times 1} P (I_{A} = 1) = 1 - P (A) + e^{t} P (A)

What is the Moment Generating Function for $X \sim U (a, b)$ ?

When $t = 0$ , we have $M_{X} (t) = E [e^{tX}] = E [e^{0 \times X}] = E [1] = 1$

When $t \neq = 0$

M_{X} (t) = E [e^{tX}] = \int_{a}^{b} e^{t x} f (x) d x = \int_{a}^{b} e^{t x} \frac{1}{b - a} d x = \frac{1}{b - a} \int_{a}^{b} e^{t x} d x = \frac{1}{b - a} \frac{e ^{t x}}{t} ∣_{a}^{b} = \frac{e ^{b t} - e ^{a t}}{t ( b - a )}

Theorem (Uniqueness)

Suppose that random variables $X$ and $Y$ have MGF’s $M_{X} (t)$ and $M_{Y} (t)$ respectively. If $M_{X} (t) = M_{Y} (t)$ for all $t$ , then $X$ and $Y$ have the same distribution.

Theorem

Theorem

Suppose that $X$ and $Y$ are independent and each have moment generating functions $M_{X} (t)$ and $M_{Y} (t)$ . Then the moment generating function of $X + Y$ is
$M_{X + Y} (t) = E (e^{t (X + Y)}) = E (e^{tX}) E (e^{t Y}) = M_{X} (t) M_{Y} (t)$

We can use this to prove the following.

Suppose that $X \sim N (μ_{1}, σ_{1}^{2})$ and $Y \sim N (μ_{2}, σ_{2}^{2})$ and $X$ and $Y$ are independent.

X + Y \sim N (μ_{1} + μ_{2}, σ_{1}^{2} + σ_{2}^{2})

Multivariate Moment Generating Functions

Definition

The joint moment generating function of two random variables, $X$ and $Y$ is
$M (s, t) = E (e^{s X + t Y})$

And so if $X$ and $Y$ are independent

M (s, t) = E (e^{s X}) E (e^{t Y}) = M_{X} (s) M_{Y} (t)

Jaiden Ratti

Explorer

STAT230

Introduction to Probability

Definitions of Probability

Mathematical Probability Models

Samples Spaces and Probability

Probability and Counting Techniques

Addition and Multiplication Rules

Counting Arrangements or Permutations

Counting Subsets or Combinations

Arrangements when Symbols are Repeated

Useful Series and Sums

Probability Rules and Conditional Probability

General Methods

Rules for Unions of Events

Intersection of Events and Independence

Conditional Probability

Product Rules, Law of Total Probability and Bayes’ Theorem

Discrete Random Variables

Random Variables and Probability Functions

Discrete Uniform Distribution

Hypergeometric Distribution

Binomial Distribution

Negative Binomial Distribution

Geometric Distribution

Poisson Distribution (from Binomial)

Poisson Distribution from Poisson Process

Combining Models

Expected Value and Variance

Summarizing Data on Random Variables

Expectation of a Random Variable

Means and Variances of Distributions

Continuous Random Variables

General Terminology and Notation

Continuous Uniform Distribution

Exponential Distribution

Computer Generated Random Numbers

Normal Distribution

Multivariate Distributions

Basic Terminology and Techniques

Multinomial Distribution

Expectation for Multivariate Distributions: Covariance and Correlation

Mean and Variance of a Linear Combination of Random Variables

Linear Combinations of Independent Normal Random Variables

Indicator Random Variables

Central Limit Theorem and Moment Generating Functions

Central Limit Theorem

Moment Generating Functions

Multivariate Moment Generating Functions

Graph View

Table of Contents

Backlinks