Date: 04/8 Chapter 2: Probability
- 2.37 Page 54
- 2.44
- FIrst order moment
- Figure 2.10
- 2.69
- 2.71, How does it perform with linear transformations, I think i still need to take care of scaling factor (but the place of the mode will remain the same in the liner). TODO: Check the parts I underlined in p.63 if I understood them well again
- 2.13
- 2.88
- 2.91
- 2.96 https://chatgpt.com/share/6892ac75-4518-800f-96e0-f82aedb99ee8 (last message)- KL Divergence and why Cross entropy instead of entropy
Cross entropy: Think of it if I have samples from p(x) but I think they came from q(x) or another distribution I modeled, and I also made a bitcode that is accustomed to q(x) to send these samples, then how many bits will I send on average, The idea is that most probably Ill send more than needed because the code I created is not corrected because the real samples are from p(x)
And this is not about the code, this about the amount of information I think q(x) has:
- Convex functions
- Jensens Inequality https://math.stackexchange.com/questions/2031062/proof-of-nonnegativity-of-kl-divergence-using-jensens-inequality
- 2.107
- Eq 2.36, Used to model deterministic variables as distributions