*Almost All* of Probability

Probability

Probability is a mathematical language that describes the unobserved world, using vocabulary from Measure Theory.

graph RL
A("Probability space (Ω,𝓕,𝐏)")
O("Space Ω") --> A
F("Sigma field 𝓕") --> A
F1@{shape: braces, label: "1\\. Empty set<br>2. Closure under complementation<br>3. Closure under countable unions"} --- F
P("Probability measure 𝐏") --> A
P1@{shape: braces, label: "1\\. Non-negativity<br>2. Countable addivity<br>3. Normalization"} --- P
style P1 text-align:left
style F1 text-align:left

The three defining conditions of a probability measure are also called the probability axioms.

Basic Concepts

• Probability Space
  • Sigma Field
  • Measure
    • Caratheodory’s Extension
• Independence
• Conditional Probability
  • Law of Total Variance
  • Bayes’ theorem
• Random Variable
  • CDF, PMF, PDF
  • Mean, Variance, Moment
  • MGF, CF, PGF
  • Reproductivity
• Multiple Random Variables
  • Joint Distribution
  • Covariance
    • Cauchy-Schwarz Inequality
• Convergence of Random Variables
  • Law of Large Numbers
  • Central Limit Theorem
  • Chernoff Bound

Advanced Notes

• Borel-Cantelli Lemma
• Probability Inequalities
• Moment Method
• Order Statistics
• Abstract Integration
  • Fatou’s Lemma
  • Fubini’s Theorem
• Stochastic Asymptotic Notation
• Extreme Value Theory

Problems

• Pairwise Independence Is Not Mutual Independence
• Covariance and Independence
• Draw and Add Balls
• A Counting Problem
• A Plausible Treatment Test
• Matching Problem

Common Distributions

Distribution	Notation	Parameters	CDF	PMF/PDF	Mean	Variance	MGF	CF
Normal Distribution	$\mathcal{N}(\mu ,{\sigma }^2)$ Link to original	$\mu \in \mathbb{R},{\sigma }^2\in {\mathbb{R}}_{+}$ Link to original	/	$f(x)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}$ Link to original	$\mu$ Link to original	${\sigma }^2$ Link to original	$e^{\mu t+{\sigma }^2{t}^2/2}$ Link to original	$e^{it\mu -{\sigma }^2{t}^2/2}$ Link to original
Uniform Distribution	$\mathcal{U}(a,b)$ Link to original	$a,b\in \mathbb{R}$ Link to original	$F(x)=\{\begin{array}{ll}0,& x<a,\\ \frac{x-a}{b-a},& a\le x\le b,\\ 1,& x>b.\end{array}$ Link to original	$f(x)=\{\begin{array}{ll}\frac{1}{b-a},& a\le x\le b,\\ 0,& \text{otherwise.}\end{array}$ Link to original	$\frac{a+b}{2}$ Link to original	$\frac{(b-a{)}^2}{12}$ Link to original	$\frac{e^{tb}-e^{ta}}{(b-a)t}$ Link to original
Bernoulli Distribution	/	$p\in [0,1]$ Link to original	$F(x)=\{\begin{array}{ll}0,& x<0,\\ q=1-p,& 0\le x\le 1,\\ 1,& x>1.\end{array}$ Link to original	$p(n)=\{\begin{array}{ll}p,& n=1,,\\ q:=1-p,& n=0.\end{array}$ Link to original	$p$ Link to original	$pq$ Link to original	$q+p{e}^t$ Link to original
Binomial Distribution	$B(n,p)$ Link to original	$n\in \mathbb{N},p\in [0,1]$ Link to original	/	$p(k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$ Link to original	$np$ Link to original	$npq$ Link to original	$(q+p{e}^t{)}^n$ Link to original
Geometric Distribution	/	$p\in [0,1]$ Link to original	$F(n)=1-q^n$ Link to original	$p(n)=p{q}^{n-1}$ Link to original	$1/p$ Link to original	$q/p^2$ Link to original	$\frac{p{e}^t}{1-q{e}^t}$, $e^t<1/q$ Link to original
Poisson Distribution	/	$\lambda >0$ Link to original	/	$p(n)=e^{-\lambda }\frac{{\lambda }^n}{n!}$ Link to original	$\lambda$ Link to original	$\lambda$ Link to original	$\exp (\lambda (e^t-1))$ Link to original	$\exp (\lambda (e^{it}-1))$ Link to original
Exponential Distribution	/	$\lambda >0$ Link to original	$1-e^{-\lambda x}$ Link to original	$f(x)=\{\begin{array}{ll}\lambda e^{-\lambda x},& x\ge 0,\\ 0,& x<0\end{array}$ Link to original	$\frac{1}{\lambda }$ Link to original	$\frac{1}{{\lambda }^2}$ Link to original	$\lambda /(\lambda -t),t<\lambda$ Link to original	$\lambda /(\lambda -it)$ Link to original
Gamma Distribution	/	$\alpha ,\lambda >0$ Link to original	/	$f(x)=\{\begin{array}{ll}\frac{\lambda e^{-\lambda x}(\lambda x{)}^{\alpha -1}}{\Gamma (\alpha )},& x\ge 0,\\ 0,& x<0\end{array}$ Link to original	$\alpha /\lambda$ Link to original	$\alpha /{\lambda }^2$ Link to original	${\left(\frac{\lambda }{\lambda -t}\right)}^{\alpha },t<\lambda$ Link to original
Beta Distribution	/	$a,b>0$ Link to original	/	$f(x)=\frac{\Gamma (a+b)}{\Gamma (a)\Gamma (b)}{x}^{a-1}(1-x{)}^{b-1}$ Link to original	$\frac{a}{a+b}$ Link to original	$\frac{ab}{(a+b{)}^2(a+b+1)}$ Link to original	$\frac{\Gamma (a+b)}{\Gamma (a)\Gamma (b)}{\int }_0^1{x}^{a-1}(1-x{)}^{b-1}{e}^{tx}\mathrm{d}x$ Link to original
Dirichlet Distribution	/	$\alpha \in {\mathbb{R}}_{++}^V$ Link to original	/	$p\left(x\mid \alpha \right)=\frac{\alpha \left({\sum }_v{\alpha }_v\right)}{{\prod }_{v=1}^V\alpha \left({\alpha }_v\right)}{\prod }_{v=1}^V{x}_v^{{\alpha }_v-1}$ Link to original	$\left({\alpha }_i/{\sum }_v{\alpha }_v\right)$ Link to original	$\mathrm{Cov}(x_i,x_j)=\frac{{\delta }_{ij}{\tilde{\alpha }}_i-{\tilde{\alpha }}_i{\tilde{\alpha }}_j}{{\sum }_v{\alpha }_v+1}$, where ${\tilde{\alpha }}_i={\alpha }_i/\left({\sum }_v{\alpha }_v\right)$ Link to original
Chi-Square Distribution	${\chi }_n^2$ Link to original	$n$ Link to original	/	$f(x)=\frac{e^{-x/2}(x/2{)}^{n/2-1}}{2\Gamma (n/2)},x\ge 0$ Link to original	$n$ Link to original	$2n$ Link to original	$(1-2t{)}^{-n/2}$ Link to original
Wishart Distribution
t Distribution	/	$n\in \mathbb{N}$ Link to original	/	$\frac{\Gamma \left(\frac{n+1}{2}\right)}{\sqrt{n\pi }\Gamma \left(\frac{n}{2}\right)}{\left(1+\frac{x^2}{n}\right)}^{-\frac{n+1}{2}}$ Link to original	0 Link to original	$\frac{n}{n-2}$ Link to original	Undefined Link to original
F Distribution	/	$n,m\in \mathbb{N}$ Link to original	/	/	$m/(m-2)$ Link to original	$\frac{2m^2(m+n-2)}{n(m-2{)}^2(m-4)}$ Link to original	Undefined Link to original
Hypergeometric Distribution	/	$N\in \mathbb{N},M,n\in [N]$ Link to original	/	$p(k)=\frac{\left(\genfrac{}{}{0ex}{}{M}{k}\right)\left(\genfrac{}{}{0ex}{}{N-M}{n-k}\right)}{\left(\genfrac{}{}{0ex}{}{N}{k}\right)}$ Link to original	$\frac{nM}{N}$ Link to original	$\frac{nM(N-n)(N-M)}{N^2(N-1)}$ Link to original	/
Cauchy Distribution	/	$a$ Link to original	$\frac{1}{\pi }\left(\arctan \frac{x}{a}+\frac{\pi }{2}\right)$ Link to original	$\frac{1}{\pi }\frac{a}{x^2+a^2}$ Link to original	Undefined Link to original	Undefined Link to original	Undefined Link to original
Discrete Power Law Distribution	/	Discrete: $\alpha \in {\mathbb{R}}_{++}$ Link to original	$F(k)=1-1/(k+1{)}^{\alpha }$ Link to original	$p(k)=1/k^{\alpha }-1/(k+1{)}^{\alpha }$ Link to original	Discrete: ${\sum }_{k=1}^{\infty }1/k^{\alpha }$ Link to original	Discrete: ${\sum }_{k=1}^{\infty }2k^{1-\alpha }-k^{-\alpha }-{\left({\sum }_{k=1}^{\infty }{k}^{-\alpha }\right)}^2$ Link to original	Discrete: $1+(e^t-1){\sum }_{k=0}^{\infty }{e}^{tk}(k+1{)}^{-\alpha }$ Link to original
Continuous Power Law Distribution	/	Continuous: $\alpha ,c\in {\mathbb{R}}_{++}$, $\beta =c^{\alpha }$ Link to original	$F(x)=1-c^{\alpha }/x^{\alpha }$ for $x\ge c$ Link to original	$f(x)=\alpha c^{\alpha }/x^{\alpha +1}$ Link to original	Continuous: $\frac{\alpha }{\alpha -1}$ Link to original	Continuous: $\frac{\alpha }{(\alpha -1{)}^2(\alpha -2)}$ Link to original	/
Dirac Distribution	${\delta }_{x_0}$ Link to original	$x_0\in \mathbb{R}$ Link to original	$F(x)=1\left\{x\ge x_0\right\}$ Link to original	Transclude of dirac-distribution#^pdf	$x_0$ Link to original	$0$ Link to original	$e^{x_{0}t}$ Link to original
Laplace Distribution

References

• Textbooks
  • Dimitri P. Bertsekas and John N. Tsitsiklis, Introduction to Probability
  • Geoffrey Grimmett and David Stirzaker, Probability and Random Processes
  • Sheldon Ross, Introduction to Probability and Statistics for Engineers and Scientists
  • Gangjian Ying and Ping He, Probability Theory
• Courses
  • MIT 6.7700 w/ Prof. Philippe Rigollet, and 6.431 w/ Prof. John Tsitsiklis
  • Columbia STAT 5701, 5703
  • Fudan MATH 130009 w/ Prof. Gangjian Ying