A learning-rate schedule changes the optimizer’s update size as training progresses.
dotnet add package RecursiveAI.OneModel.TrainingRequires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneModel.Training; var schedule = new WarmupLr( new CosineLr(0.1f, 4, 0.01f), 2, 0.1f); for (int step = 0; step <= 7; step++) Console.WriteLine( $"Step {step}: " + $"{schedule.LrAt(step):F4}");
The schedule uses 2 warmup steps starting at 10% of the rate, then cosine decay from 0.1 down to 0.01 over 4 steps.
Unzip the example, then dotnet run.
Step 0: 0.0100 Step 1: 0.0550 Step 2: 0.1000 Step 3: 0.0868 Step 4: 0.0550 Step 5: 0.0232 Step 6: 0.0100 Step 7: 0.0100
The full program
The snippet at the top previews the rates. The download applies those same rates to eight real SGD updates of a scalar weight, using weight squared as the loss.
using RecursiveAI.OneModel.Tensors; using RecursiveAI.OneModel.Training; using var weight = Tensor.Scalar(3f).WithRequiresGrad(); using var optimizer = new Sgd([weight], 0.1f); var schedule = new WarmupLr( new CosineLr(0.1f, 4, 0.01f), 2, 0.1f); for (int step = 0; step <= 7; step++) { optimizer.ZeroGrad(); using var loss = Ops.Mul(weight, weight); loss.Backward(); schedule.Apply(optimizer); Console.WriteLine($"Step {optimizer.StepCount}: {optimizer.LearningRate:F4}"); optimizer.Step(); } optimizer.ZeroGrad();
Step 0: 0.0100 Step 1: 0.0550 Step 2: 0.1000 Step 3: 0.0868 Step 4: 0.0550 Step 5: 0.0232 Step 6: 0.0100 Step 7: 0.0100
Schedule steps count optimizer updates, not epochs; call Apply before Step so the first update uses step index zero.
Variations
To drop the rate by half every two updates, replace the schedule declaration in the full program:
var schedule = new StepLr(0.1f, 2, 0.5f);It prints:
Step 0: 0.1000 Step 1: 0.1000 Step 2: 0.0500 Step 3: 0.0500 Step 4: 0.0250 Step 5: 0.0250 Step 6: 0.0125 Step 7: 0.0125
Related recipes
The short answer
Compose WarmupLr around CosineLr and call schedule.Apply(optimizer) before each update. Apply reads optimizer.StepCount; it does not advance a separate scheduler counter. The snippet at the top previews exactly the rate function that the downloadable training loop applies.
The example uses two warmup steps followed by four cosine intervals, with a peak rate of 0.1 and a minimum of 0.01. These small counts make the boundaries visible in the printed output. They are not training recommendations. The learning-rate reference gives the formulas and the meaning of the step index.
What does a learning rate scheduler do?
It assigns the optimizer’s learning rate from a step number. The optimizer still owns gradient clearing, parameter updates and its step count. A schedule does not compute a loss or run backward propagation. That is why the download includes a real scalar training loop as well as the rate preview.
In OneModel, LrAt(stepIndex) is a pure function. Calling it repeatedly with the same index gives the same rate without moving training forward. Apply evaluates that function at the number of updates already taken. On a fresh optimizer that count is zero, so the first call sets the rate for step zero. Applying only once before a whole training run leaves the rate unchanged during subsequent updates.
For epoch-based scheduling, decide explicitly how epochs map to optimizer updates. If each epoch contains several batches, the optimizer count advances for every Step, not once per epoch. If gradients accumulate across batches, it advances only when you actually update parameters.
What is the warmup learning rate?
Warmup starts at a fraction of the wrapped schedule’s initial rate and increases linearly toward it. In WarmupLr(after, W, f), the initial rate is after.LrAt(0) * f. While the index is below W, the fraction is f + (1 - f) * step / W. At W the wrapped schedule starts at its own index zero.
Here the starting fraction is 0.1, so the first rate is 0.01. Step one uses 0.055. Step two starts the wrapped cosine at 0.1. The warmup does not consume the cosine’s duration: it precedes it. When planning a run, include both the warmup length and the cosine duration in your update budget.
What is a cosine learning rate?
Cosine decay interpolates from a base rate to a minimum using half a cosine wave. CosineLr(baseLr, totalSteps, minLr) reaches the minimum at its own totalSteps index and stays there afterward. It does not restart or turn upward after that boundary.
In this composition, the cosine’s index zero is global step two, its midpoint is global step four, and its minimum is reached at global step six. Step seven stays at 0.01. The output therefore distinguishes a completed schedule from a new warmup or restart.
Recreate the same schedule configuration when resuming training and restore the optimizer’s step count with its state. The schedule has no separate counter to restore. If you instead create a fresh optimizer at step zero, applying the schedule restarts warmup even if the model weights came from a later checkpoint. Changing the schedule parameters also changes resumed rates, so preserve those settings with your training configuration.