Backpropagation

tags: deep learning

Some notations

• $w_{jk}^l$ means the weight of the $k^{th}$ neuron in $(l-1)$ connected to the $j^{th}$ neuron
• $b_j^l$ means the bias of the $j^{th}$ neuron in the $l^{th}$ layer
• $a_j^l=\sigma \left({\sum }_k{w}_{jk}^l{a}_k^{l-1}+b_j^l\right)$ to get a neuron activation
• Hadamard product is the multiplying corresponding elements in two matrices

First Equation

We will consider the rate of change of the cost function multiplied by the activation to tell us the amount of error. This equation seems quite natural ${\delta }_j^L=\frac{\partial C}{\partial a_j^L}{\sigma }^{\prime }(z_j^L)$ if $z_j$ is changed by an amount this will lead to change in the sigmoid function with a some amount, multiplying the change by the change in the the cost function by amount $\frac{\partial C}{\partial a_j^L}$, leads to the amount of error.

Second Equation

${\delta }^l=((w^{l+1}{)}^T{\delta }^{l+1})\odot {\sigma }^{\prime }(z^l)$

Backpropagation and gradient descent

• Input a set of training examples
  • For each training example x : Set the corresponding input activation ax,1, and perform the following steps:
  • Feedforward: For each l=2,3,…,L compute zx,l=wlax,l−1+bl and ax,l=σ(zx,l)- .
  • Output error δx,L : Compute the vector δx,L=∇aCx⊙σ′(zx,L)- .
  • Backpropagate the error: For each l=L−1,L−2,…,2 compute δx,l=((wl+1)Tδx,l+1)⊙σ′(zx,l)- - .
• Gradient descent: For each l=L,L−1,…,2 update the weights according to the rule wl→wl−ηm∑xδx,l(ax,l−1)T, and the biases according to the rule bl→bl−ηm∑xδx,l.

NOTE: The derivative of sum is the sum of derivatives