Four gradient-descent variants minimize the same loss surface from the same start -
click anywhere on the surface to drop them somewhere new. Drag to orbit, scroll to zoom.
Watch SGD stall in local minima while Adam's adaptive step sails past, or Momentum overshoot
the valley it was aiming for. Implemented from scratch: θ ← θ − η·∇L(θ) and friends.
The surface is a 2-parameter loss function, a toy stand-in for the millions-dimensional landscapes real networks train on. Each ball runs its update rule on the analytic gradient, a few steps per frame:
SGD follows the raw gradient. Momentum (β=0.92) accumulates velocity, faster in ravines, prone to overshooting. RMSProp divides by a running RMS of the gradient, equalizing step sizes across dimensions. Adam combines both with bias correction.
Training physics-informed neural networks is an optimization nightmare: PDE-residual and data terms can sit seven orders of magnitude apart (I've measured it), producing exactly these pathological ravines and saddles.
Picking and tuning the optimizer isn't a detail there, it's the difference between a heart model that converges and one that doesn't. This toy is how I explain that problem in one picture.