Backpropagation

Study guide: Exam MAS-II

Backpropagation is the chain rule applied backwards through a Neural Network to get the gradient of the loss with respect to every weight in one sweep. It is what makes gradient descent on a network with thousands of weights feasible.

\frac{\partial L}{\partial w_{kj}} = \frac{\partial L}{\partial A_k}\cdot\frac{\partial A_k}{\partial z_k}\cdot\frac{\partial z_k}{\partial w_{kj}}

w \leftarrow w - \eta\,\frac{\partial L}{\partial w}

• Forward pass computes the activations layer by layer and the loss; backward pass propagates \partial L/\partial z from the output back to the input, reusing each layer's result for the one below
• Cost is roughly the same as a forward pass — the saving over differentiating each weight numerically is what made deep networks practical
• \eta is the learning rate: too large and the loss oscillates or diverges, too small and training crawls
• Stochastic gradient descent applies the update on mini-batches rather than the full dataset, adding noise that helps escape poor local minima
• The loss surface is non-convex — different random starting weights give different fits, so results are not reproducible without a fixed seed
• Regularization comes as a weight-decay penalty (ridge on the weights), early stopping, or dropout

One Gradient-Descent Step A single output weight is w = 0.40. The forward pass gives a prediction of 3.0 against an actual of 5.0, under squared-error loss L = (y - \hat y)^2, and the activation feeding that weight is A = 2.0. With \eta = 0.05, find the updated weight.

Answer \frac{\partial L}{\partial \hat y} = -2(y - \hat y) = -2(5 - 3) = -4 \frac{\partial \hat y}{\partial w} = A = 2 \;\Rightarrow\; \frac{\partial L}{\partial w} = -4(2) = -8 w \leftarrow 0.40 - 0.05(-8) = 0.40 + 0.40 = 0.80 The weight rises because the prediction was too low and the activation feeding it was positive — the step size is the gradient scaled by \eta.