Hello and welcome to this video on fully connected neural networks. In this video, we will go through the calculation of the back propagation for a fully connected neural network. In the last lecture, we calculated the back propagation for a network with only two nodes in series. So, let's see how we can expand this calculation to a fully connected neural network. In a fully connected network, each layer contains several nodes and each node is connected to all of the nodes in the previous and in the next layers. The weights in the layer are identified by two indices, k and j, where k indicates the receiving node and j indicates the emitting node. 

The input sum at layer l and node k, which we will indicate with Z-l-k, is the weighted sum of the activation of layer l-1 plus the bias term of layer l. Like before, we can expand the derivative of the cost function with respect to the weight W-k-j in layer l using the chain rule and we notice that the last two terms are identical to before. The derivative of the activation, A, is just the derivative of the sigmoid and the derivative of the input sum, Z, is the activation of the previous layer. 

The only different term in this case is the derivative of the cost with respect to the activation A-l-k. Since the activation A-l-k is part of the input to each of the following nodes in the next layer, l+1, we have to apply the chain rule to each of them, and sum all the contributions together. Notice that in the last term, we can replace the derivative of the input sum with the weights of the layer, l+1. And the derivative of the cost function with the delta, like we did in the one dimensional case. So, we notice that the deltas are connected by a recursive formula. 

Which means we can obtain the deltas at layer l as a function of the deltas at layer l+1. We have already seen this in the unidimensional case when delta one was a function of delta two. So, this is why it's called back propagation and how it works with many nodes. Again, we start from the last layer, the one closest to the output, and we calculate the deltas and propagate them back through the layers using the recursive formula that is displayed here. I know your head may be spinning a little bit after all these indecisive formulas, so in the next video, we will see how to simplify the notation a little bit. 

But, first of all, I want to complement you for making it this far. In this video, we went through one of the hardest part of neural networks which was calculating the back propagation for a fully connected network. You've gone through the back propagation algorithm which is the core of how neural networks are train. So, give yourself a big applause and take a quick break. Neural networks have no more mysteries for you. Thank you for watching and see you in the next video.