Follow a derivative through a composition

THE IDEA

Read the example.

Here u = 2x + 1 and y = u². The first map has derivative 2 and the second has derivative 2u. Their product gives dy/dx = 4u. This small computation graph is the central idea behind backpropagation.

dy/dx = (dy/du)(du/dx)

Use it well

Use the values from the actual forward computation when evaluating local derivatives. The chain rule combines sensitivities; it does not mean adding them at a simple composition.