Neural Networks: Computing Partial Derivatives

We will compute the partial derivative of the Cross-Entropy cost function

with respect to one weight in each connection with a sigmoid activation function for each layer.

Given

Therefore

Take aways:

• We Can compute the partial derivative of a weight by multiplying the value of the node in layer ℓ and the 𝛿 of the connected node in layer ℓ+1
• 𝛿’s of the nodes in layer ℓ can be computed from the 𝛿’s of the nodes in the layer ℓ+1