Conditional Entropy and Mutual Information

tags: math, probability, Information theroy

Conditional Entropy

The formula for conditional entropy is: $H[X|Y]=-\int \int p(x,y)\ln p(y|x)$ We can think of this formula as the expected value of information contained in $y$ if we already know a value of $x$ and this is what is really is

Mutual Information

$I[X,Y]=-\int \int p(x,y)ln(\frac{p(x)p(y)}{p(x,y)})dydx$ Mutual information is seen as the reduction in uncertainty in one of the variables by the knowing the other one.