Mathematical Foundations of Machine Learning

Robert Nowak

Conditional probability

Defined as the solution to the equation $p(x,y)=p(y|x)p(x$) or, rearranged, $p(y|x)=p(x,y)/p(x)$

In this way, we can think of conditional probabilities as a mathematical object rather than a more intuitive “words” explanation.

Marginal probability

To marginalize is to sum over all possible values of a variable. Thus, if the joint density in question is $p(x,y)$, the marginal probability of a random variable $X$ is $p(x)={\sum }_{y}p(x,y)={\int }_{y}p(x,y)dx$ Marginalization is often useful when you need to remove a random variable from consideration. By marginalizing above, $y$ drops out of the equation.

Expected value

The expected value of a function of a random variable is $E[f(X)]={\sum }_{x}f(x)p(x)=\int f(x)p(x)dx$ Note that this works for any function – you just evaluate the function at each potential realization and multiply that by the probability of that realization, then sum it all up. You can also use this for things like conditional expectations, where the “function” is one of the variables and the probability is the conditional probability: $E[Y|X]={\sum }_{y}yp(y|x)=\int yp(y|x)dx$

---

References

paperreadonline