Chapter 3

Introduction

Humans learn the fastest when they know they are blatantly wrong, neural networks in contrast learn the slowest when they are completely wrong

This happens due to the flatness of the sigmoid function in the partial derivatives

Cross entropy cost function

$C=-\frac{1}{n}{\sum }_x\left[y\ln a+(1-y)\ln (1-a)\right]$ Why did we choose this function ? Well, the rate of change of this function with respect to a weight gives: $\frac{\partial C}{\partial w_j}=\frac{1}{n}{\sum }_x{x}_j(\sigma (z)-y)$ which means the more we are of from the correct output the more we learn

What is entropy ?

For simplicity, we can consider entropy to be the amount of surprise, the smaller the probability of occurrence the greater the element of surprise. but still that is not entropy. Entropy is the expected value of surprise

Entropy is also the amount of uncertainity in a distribution. For example If a have 3 red balls and 1 green, I am mostly certaint that i’ll pick a red ball

.Another way to look at entropy is that it is encoding information, for example if the probability that today is sunny, and I told you it is going to be cold. In this way I told you more information, maybe I am not sure

A yes or no question is a single bit.

What is cross entropy then ? Well consider we have a predicted distribution (the output of the neural network), the higher the probability the more bits we will send according to Shannon’s theory, and consider another probability distribution we have which is the correct one. If the probabilities are off by a significant amount we will be sending more bits than it is actually needed.

TLDR; Cross entropy is quantifying how much a predicted distribution function differs from the correct one.

Softmax

${\sum }_j{a}_j^L=\frac{{\sum }_j{e}^{z_j^L}}{{\sum }_k{e}^{z_k^L}}=1$ Another activation function that can be used is the softmax function. The softmax function gives the benefit of the output being a probability distribution. The exponential is used to amplify the maximum, make the function positive and ensure derrivability

Log likelihood

$C\equiv -\ln a_y^L.$

Regularization

Regularization helps in making our neural network or model generalize better. The smaller the weight the less a certain input will dominate, making the neural network much better in generalizing than learning the details of our data

Dropout regularization

In the dropout method we tend to cancel or drop some neurons and train the network with out them using a mini-batch. A great intuition to have here is that we are teaching the available neurons to not rely on other neurons so they learn more robustly and generalize more. Another way to think about it is that we make each small network over fit to the data then we average the over fitting leading to generalization

Weight Initialization

initializing the weight with a normal distribution if mean 0 and standard deviation $\frac{1}{\sqrt{n_{inputneuron}}}$ results in a much faster learning neural network because $z$ will not saturate

Tweaking hyper parameters

in early stages emphasize getting a fast response