The Silver Stepsize Schedule

Apr 2026 · University of Oxford

Read the full reportPDF, 27 pages, 13.3 MB

κ^0.79

convergence rate, against κ for a constant step

ESSS

new extension that works for any step budget

2ⁿ

step counts where the schedule pays off fully

Overview

Gradient descent has to choose how far to step at each iteration, and for badly conditioned problems a constant step is slow. The Silver Stepsize Schedule of Altschuler and Parrilo mixes very short and very long steps in a recursive, fractal pattern and provably beats constant steps without using momentum, converging at a rate of about κ to the power 0.79 rather than κ.

This project reviews the classical accelerated methods, explains the schedule through the polynomials that describe the error, and investigates it numerically. I developed the Extended Silver Stepsize Schedule (ESSS), which handles any number of iterations by combining complete Silver schedules following the binary expansion of the step count.

What I found

The full benefit appears only at particular step counts, the powers of two, where the largest step in each cycle has its effect. Stopping part way gives a much weaker result.
ESSS consistently beat simply truncating the schedule for step counts that are not powers of two, and matched it otherwise.
At power of two budgets, cycling a slightly shorter schedule beat the theoretically certified one in all the experiments ran.
On the Rosenbrock function, which is not quadratic, ESSS also outperformed other extensions of the schedule.

Left, the paths taken by three gradient descent methods down the curved Rosenbrock valley. Right, a chart showing the error of each method over 256 steps, with the extended Silver schedule lowest. — Left: the first 31 steps of the Silver schedule. Right: worst case error on a quadratic problem with condition number 500, computed directly from the stepsizes, compared with constant steps. Drawn for this page.