# Technical notes on mathematical expectation For the entirety of this page, we will assume that our random variables are discrete and take their values in a finite sample space $\mathcal{S}$. Most of these results hold in the exact same form for discrete random variables with infinite state spaces and for continuous random variables, though the proofs differ in detail. >[!QUOTE] Expected value of a function of a random variable >Let $X$ be a discrete random variable and let $f(x)$ be a deterministic real-valued function. Then >$ >\mathbb{E}(X) = \sum_{x \in \mathcal{S}} f(x) \mathbb{P}(X = x). >$ *Remark.* This is often known as the [law of the unconscious statistician](https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician) and is formulated as a theorem with a [proof](https://statproofbook.github.io/P/mean-lotus). #### Expected value of functions of random variables --- >[!NOTE] Proposition >Let $X$ be a discrete random variable that takes its values in the set $\mathcal{S}$. Let $a$ and $b$ be deterministic real numbers. Then >$ >\mathbb{E}(aX + b) = a \mathbb{E}(X) + b >$ *Proof.* We use the definition of the function of a random variable in the first step. Then we distribute the factor $\mathbb{P}(X = x)$ and separate the sums. Factoring out the $a$ and $b$ in each sum, we remember the definition of expected value, $\mathbb{E}(X) = \sum_x x \mathbb{P}(X = x)$, and that a random variable's probabilities sum to 1, $\sum_x \mathbb{P}(X = x) = 1$. Mathematically: $ \begin{aligned} \mathbb{E}(a X + b) &= \sum_{x \in \mathcal{S}} (ax + b) \mathbb{P}(X = x) \\ &= \sum_{x \in \mathcal{S}} ax \mathbb{P}(X = x) + \sum_{x \in \mathcal{S}} b \mathbb{P}(X = x) \\ &= a \sum_{x \in \mathcal{S}} x \mathbb{P}(X = x) + b \sum_{x \in \mathcal{S}} \mathbb{P}(X = x) \\ &= a \mathbb{E}(X) + b. \end{aligned} $ --- Linear functions like $f(x) = ax +b$ are the exception to the rule when it comes to mathematical expectation. In general, if $f$ is nonlinear, we have that $ \mathbb{E}(f(X)) \neq f(\mathbb{E}(X)). $ In fact, when a function is concave or convex, we there is a predictable way in which these differ. >[!quote] Definition: Convex function >We say that a function $f$ is convex on an interval $\mathcal{I}$ if for every $x, y \in \mathcal{I}$, and $\alpha \in [0,1]$, we have >$ >f\big(\alpha x + (1-\alpha) y\big) \leq \alpha f(x) + (1-\alpha)f(y). >$ A convex function in this sense is sometimes called *concave up*, and if the inequality is reversed it is called *concave down.* >[!NOTE] Jensen's Inequality >Suppose that $X$ is a discrete random variable that takes its values in $\mathcal{S} \subset \mathbb{R}$. If $f(x)$ is a convex function, then ^jensen-inequality >$ >f(\mathbb{E}(X)) \leq \mathbb{E}(f(X)) >$ The [[#Proof of Jensen's Inequality|proof]] can be found below. *Remark 1.* If $f$ is concave down, then the inequality is reversed. *Remark 2.* Since $f(x)=x^2$ is convex we see that $ \mathrm{Var}(X) = \mathbb{E}(X^2) - \big(\mathbb{E}(X)\big)^2 \geq 0 $ Also when thinking about economic utility functions that are concave down, we see that $ \sqrt{\mathbb{E}(X)} \leq \mathbb{E}(\sqrt{X}) $ when $X$ is a nonnegative random variable. This helps model ideas like "the first million is worth more than the second million!" #### Expected values of sums >[!quote] Linearity of Expectation. >Suppose that $X$ and $Y$ are random variables that have finite expected values $\mu_X$ and $\mu_Y$, respectively. If $a$ and $b$ are real-valued constants, then >$ >\mathbb{E}(aX + bY) = a \mu_X + b \mu_Y. >$ >[!quote] Variance of sums of random variables >Suppose that $X$ and $Y$ are random variables that have finite expected values ($\mu_X$ and $\mu_Y$) and finite variances ($\sigma_X^2$ and $\sigma_Y^2)$. Then >$ >\mathrm{Var}(aX + bY) = a^2 \sigma_X^2 + b^2 \sigma_Y^2 + a b \,\mathrm{Cov}(X,Y) >$ >where $\mathrm{Cov}(X,Y) \coloneqq \mathbb{E}(XY) - \mu_x \mu_y$ > >In particular, if $X$ and $Y$ are independent, then > >$\displaystyle \text{Independent } X,Y \text{ :} \qquad \mathrm{Var}(aX + bY) = a^2 \sigma_X^2 + b^2 \sigma_Y^2.$ --- #### Proofs *Proof of [[#^jensen-inequality|Jensen's Inequality]] .* We proceed recursively. Let $n$ refer to the size of the sample space $\mathcal{S}.$ *Case* $n$ = 1. Here, $X$ can only take one value $x$ with probability 1. And so $ f(\mathbb{E}(X)) = f(x) = \mathbb{E}(f(X)). $ *Case* $n=2$. Though we rarely explicitly show the case $n = 2$ for inductive proofs, it is instructive to do so here. Suppose that $X$ has the distribution $ X \sim \begin{cases} x_1, & \text{w.p. } p; \\ x_2, & \text{w.p. } 1 - p. \end{cases} $ where "w.p." means "with probability." Then by the convexity of the function $f$, $ \begin{aligned} f\big(\mathbb{E}(X)\big) &= f\big(p x_1 + (1-p) x_2\big) \\ &\leq f(x_1) p + f(x_2) (1-p) \\ &\leq \mathbb{E}(f(X)) \end{aligned} $ *Case* $n+1$. Suppose that the relation holds for the case $n$. If there are $n + 1$ members of the set $\mathcal{S}$ then the probabilities $\big\{\mathbb{P}(X = x_k)\big\}_{k=1}^{n+1}$ add up to one. Let $\pi$ be the sum of the first $n$ probabilities, which means they will sum to less than one: $ \pi = \sum_{k=1}^n \mathbb{P}(X = x_k) < 1. $ To move forward we need to notice two things: for the first $n$ terms, if we renormalize their probabilities, we can apply the inductive step to that set; then, we can group the first $n$ together and the proof reduces to the case $n=2$. To this end, let $\tilde X$ have the distribution on the first $n$ values of $\mathcal{S}$ where $\mathbb{P}(\tilde X = x_k) = \mathbb{P}(X = x_k) / \pi$. You can check that, by definition, these probabilities add to 1. From the inductive step we know that $f\big(\mathbb{E}(\tilde X)\big) \leq \mathbb{E}\big(f(\tilde X)\big)$. Let $\tilde \mu = \mathbb{E}(\tilde X)$ and define $ Y = \begin{cases} \tilde \mu, & \text{w.p. } \pi; \\ x_{n+1} & \text{w.p. } \mathbb{P}(X = x_{n+1}). \end{cases} $ Then $\mathbb{E}(Y) = \mathbb{E}(X)$ (for an exercise, check this!).