Entropy and Cross entropy

tags: math, probability, Information theroy

Entropy is simply the average amount (Expected value) of Information in a distribution, or the average amount of surprise in a distribution. $H[x]=E_{p(x)}[h(x)]=-\sum p(x)\ln p(x)$ Since it gives us the average amount of information in a distribution, it is also considered the lower bound for the number of bits for coding a data distribution, because if you created a coding that its average length is shorted and the entropy of the data distribution you are basically dropping information.

Remember: Data != Information, data is the way to convey this information

Cross-entropy

Let’s say we have data points that come from the distribution $p(x)$ and we don’t this distribution, we just got data-points from it and we try to model the unknown distribution in $q(x)$, so the cross entropy is the average amount of information in the data-points thinking they came from $q(x)$ $CE=-\int p(x)\log q(x)$

While it may seem weird that cross entropy is defined as above because how would we know $p(x)$. In practice we use the data-points them selves and cross entropy becomes $CE=-\frac{1}{N}\sum \log q(x)$