What gradient descent does
Machine-learning training often means finding parameters that minimize a loss function. Gradient descent approaches that problem through local slope information. The gradient is a vector of partial derivatives pointing toward the steepest increase in loss. Subtracting a fraction of that vector moves parameters downhill. Repeating the update turns derivatives into a sequence of candidate solutions.
The idea is simple enough to write in a few lines, yet its behavior depends on the objective geometry, starting point, learning rate, gradient noise, and optimizer state. A round convex bowl is easy. A narrow curved valley can produce slow zigzags. A non-convex surface adds saddle points and local minima. This visualization makes those differences visible rather than hiding them inside a training call.
Compare SGD, Momentum, and Adam interactively
Choose a loss surface, drag the shared start marker, and press Play. The chart advances three update rules from identical initial coordinates. Plain SGD follows the current gradient. Momentum combines the gradient with a decaying velocity. Adam estimates both the first moment of the gradient and its uncentered second moment, then uses those estimates to adapt the update in each coordinate.
The simulation runs entirely in JavaScript. Its objectives use KMath.sum from @kanaries/ml, while the optimizer equations are intentionally written in the component so learners can inspect them. This distinction matters: the lines are a transparent educational simulation, not undocumented internal traces extracted from a model.
Reading the optimizer trajectories
On the convex bowl, SGD usually takes a direct path but may move faster along the steep axis than the shallow one. Momentum builds speed where gradients agree and can cross the minimum before damping out. Adam normalizes updates using recent squared gradients, so its path may look more balanced across differently scaled directions.
The Rosenbrock objective contains a long, curved valley. Reaching the valley is not the same as following it to the minimum, which exposes oscillation and coordinate imbalance. The rippled surface adds several local basins. No first-order optimizer can promise the global minimum on every non-convex objective; initialization changes which basin the path encounters.
Learning rate is part of the algorithm
The learning rate multiplies every update. Raise it and optimization covers more distance per iteration, but overly aggressive steps overshoot or bounce between sides of a valley. Lower it and the path becomes stable but may make negligible progress within the iteration budget. Try several rates on each surface and compare loss values rather than judging motion alone.
Real training commonly changes the rate over time. Decay schedules reduce it as training approaches a solution, warmup starts cautiously, and adaptive methods scale coordinates based on gradient history. Batch size changes gradient noise as well. The clean surfaces here isolate update behavior before those production considerations are introduced.
From equations to machine-learning models
For a model with millions of parameters, the chart would be impossible to draw directly, but the update logic is the same. Automatic differentiation computes gradients, the optimizer updates tensors, and a validation metric checks whether lower training loss improves generalization. Regularization adds terms or constraints so the chosen parameters do not merely memorize training examples.
Gradient clipping in this playground caps very steep gradients so all paths remain visible. Clipping is also used in neural-network training, especially for exploding gradients, but it changes the effective update and should be monitored. Likewise, finite precision, stopping criteria, and reproducible random seeds matter when an experiment becomes production software.
JavaScript and Python implementation
The code tabs implement the same convex objective and SGD loop with @kanaries/ml in JavaScript and NumPy in Python. Momentum would add a velocity vector; Adam would add first- and second-moment vectors plus bias correction. Keeping that state explicit is a useful way to understand what an optimizer contributes beyond the raw gradient.
Continue with the logistic regression calculator to connect optimization to classification probabilities, inspect the JavaScript logistic regression API for a fitted estimator, or contrast iterative optimization with the projection geometry in the PCA visualization.
Diagnosing optimization in practice
Record training loss, validation loss, gradient norm, update norm, and learning rate together. A flat training loss with tiny updates can indicate a rate that is too low, saturated activations, poor feature scaling, or a coding error. Exploding loss and non-finite numbers point toward an excessive rate, unstable arithmetic, or gradients that need clipping. Falling training loss paired with worsening validation performance is an overfitting signal, not an optimizer victory.
Optimizer comparisons should use the same initialization, batches, preprocessing, stopping budget, and random seeds. One run is rarely enough when minibatch order or initialization is stochastic. Repeat experiments and report the distribution of the metric that matters to the application. Checkpoints should include optimizer state as well as model parameters; restoring weights without Momentum velocity or Adam moments changes the subsequent trajectory. Finally, remember that faster reduction of the training objective does not guarantee better generalization. The optimizer is one part of a system that includes the objective, data, regularization, schedule, architecture, and evaluation protocol.