11 Probability Theory

11.1 Sample Space

  • The sample space \(\Omega\) is the set of all outcomes
  • We are interested in calculating probabilities on relevant subsets of this space, called events: \(A \subseteq \Omega\)
  • Examples —
    • Two coin flips: \(\Omega =\) {HH, HT, TH, TT}
    • SNP genotypes: \(\Omega =\) {AA, AT, TT}
    • Amazon product rating: \(\Omega =\) {1 star, 2 stars, …, 5 stars}
    • Political survey: \(\Omega =\) {agree, disagree}

11.2 Measure Theoretic Probabilty

\[(\Omega, \mathcal{F}, \Pr)\]

  • \(\Omega\) is the sample space
  • \(\mathcal{F}\) is the \(\sigma\)-algebra of events where probability can be measured
  • \(\Pr\) is the probability measure

11.3 Mathematical Probability

A proper mathematical formulation of a probability measure should include the following properties:

  1. The probability of any even \(A\) is such that \(0 \leq \Pr(A) \leq 1\)
  2. If \(\Omega\) is the sample space then \(\Pr(\Omega)=1\)
  3. Let \(A^c\) be all outcomes from \(\Omega\) that are not in \(A\) (called the complement); then \(\Pr(A) + \Pr(A^c) = 1\)
  4. For any \(n\) events such that \(A_i \cap A_j = \varnothing\) for all \(i \not= j\), then \(\Pr\left( \cup_{i=1}^n A_i \right) = \sum_{i=1}^n \Pr(A_i)\), where \(\varnothing\) is the empty set

11.4 Union of Two Events

The probability of two events are calculated by the following general relationship:

\[\Pr(A \cup B) = \Pr(A) + \Pr(B) - \Pr(A \cap B)\]

where we note that \(\Pr(A \cap B)\) gets counted twice in \(\Pr(A) + \Pr(B)\).

11.5 Conditional Probability

An important calclation in probability and statistics is the conditional probability. We can consider the probability of an event \(A\), conditional on the fact that we are restricted to be within event \(B\). This is defined as:

\[\Pr(A | B) = \frac{\Pr(A \cap B)}{\Pr(B)}\]

11.6 Independence

Two events \(A\) and \(B\) by definition independent when:

  • \(\Pr(A | B) = \Pr(A)\)
  • \(\Pr(B | A) = \Pr(B)\)
  • \(\Pr(A \cap B) = \Pr(A) \Pr(B)\)

All three of these are equivalent.

11.7 Bayes Theorem

A common approach in statistics is to obtain a conditional probability of two events through the opposite conditional probability and their marginal probability. This is called Bayes Theorem:

\[\Pr(B | A) = \frac{\Pr(A | B)\Pr(B)}{\Pr(A)}\]

This forms the basis of Bayesian Inference but has more general use in carrying out probability calculations.

11.8 Law of Total Probability

For events \(A_1, \ldots, A_n\) such that \(A_i \cap A_j = \varnothing\) for all \(i \not= j\) and \(\cup_{i=1}^n A_i = \Omega\), it follows that for any event \(B\):

\[\Pr(B) = \sum_{i=1}^n \Pr(B | A_i) \Pr(A_i).\]