tags: math, probability. Information theroy
We need to quantify the amount of information we acquired from the happening of an event. The best way to think about this is to imagine a day that is completely windy, no sun, and maybe even it will rain, If someone told you it will rain today then he didn’t tell us much information, it is obvious that it will not rain, but someone told you the sun will come up later, here that somebody told us a lot of information because it had a low probability of occurrence.
We also want that when we observe two unrelated events occurring, the total gained information is the summation of the information gained from both events. In this way we can define a formula for info as follows: or