11 Probability Theory

11.1 Sample Space

• The sample space $$\Omega$$ is the set of all outcomes
• We are interested in calculating probabilities on relevant subsets of this space, called events: $$A \subseteq \Omega$$
• Examples —
• Two coin flips: $$\Omega =$$ {HH, HT, TH, TT}
• SNP genotypes: $$\Omega =$$ {AA, AT, TT}
• Amazon product rating: $$\Omega =$$ {1 star, 2 stars, …, 5 stars}
• Political survey: $$\Omega =$$ {agree, disagree}

11.2 Measure Theoretic Probabilty

$(\Omega, \mathcal{F}, \Pr)$

• $$\Omega$$ is the sample space
• $$\mathcal{F}$$ is the $$\sigma$$-algebra of events where probability can be measured
• $$\Pr$$ is the probability measure

11.3 Mathematical Probability

A proper mathematical formulation of a probability measure should include the following properties:

1. The probability of any even $$A$$ is such that $$0 \leq \Pr(A) \leq 1$$
2. If $$\Omega$$ is the sample space then $$\Pr(\Omega)=1$$
3. Let $$A^c$$ be all outcomes from $$\Omega$$ that are not in $$A$$ (called the complement); then $$\Pr(A) + \Pr(A^c) = 1$$
4. For any $$n$$ events such that $$A_i \cap A_j = \varnothing$$ for all $$i \not= j$$, then $$\Pr\left( \cup_{i=1}^n A_i \right) = \sum_{i=1}^n \Pr(A_i)$$, where $$\varnothing$$ is the empty set

11.4 Union of Two Events

The probability of two events are calculated by the following general relationship:

$\Pr(A \cup B) = \Pr(A) + \Pr(B) - \Pr(A \cap B)$

where we note that $$\Pr(A \cap B)$$ gets counted twice in $$\Pr(A) + \Pr(B)$$.

11.5 Conditional Probability

An important calclation in probability and statistics is the conditional probability. We can consider the probability of an event $$A$$, conditional on the fact that we are restricted to be within event $$B$$. This is defined as:

$\Pr(A | B) = \frac{\Pr(A \cap B)}{\Pr(B)}$

11.6 Independence

Two events $$A$$ and $$B$$ by definition independent when:

• $$\Pr(A | B) = \Pr(A)$$
• $$\Pr(B | A) = \Pr(B)$$
• $$\Pr(A \cap B) = \Pr(A) \Pr(B)$$

All three of these are equivalent.

11.7 Bayes Theorem

A common approach in statistics is to obtain a conditional probability of two events through the opposite conditional probability and their marginal probability. This is called Bayes Theorem:

$\Pr(B | A) = \frac{\Pr(A | B)\Pr(B)}{\Pr(A)}$

This forms the basis of Bayesian Inference but has more general use in carrying out probability calculations.

11.8 Law of Total Probability

For events $$A_1, \ldots, A_n$$ such that $$A_i \cap A_j = \varnothing$$ for all $$i \not= j$$ and $$\cup_{i=1}^n A_i = \Omega$$, it follows that for any event $$B$$:

$\Pr(B) = \sum_{i=1}^n \Pr(B | A_i) \Pr(A_i).$