Loss function

In the chain rule, the derivative of a **composite function** is equal to the product of the derivatives of inner and outer functions.

In context of Deep learning, the inner function is the Activation function while outer function is the Loss/Cost function.

In BackPropogation, you have to calculate the gradient of the cost function which inturns calculates the gradient/derivative Of ACTIVATION FUNCTION.

Vanishing and Exploding Gradients

• Gradients are used to update the weights of the model.
• Gradients are calculated as the partial derivative of the Loss Function w.r.t the activations.
• If the activations are too large or too small, the derivative of the Loss Funciton will also be large or small respectively, which will cause the Gradient to be too large or small respectively.
• In both cases, the large or small gradients will prevent the model from converging because the weights will not be updated enough to reach the minimum of the loss function.

Various techniques have been proposed to mitigate the vanishing gradient problem, including:

• Different weight initialization strategies.
• Non-Saturating activation functions (like ReLU).
• Batch Normalization.
• Architectures like LSTM (Long Short-Term Memory) for recurrent neural networks.