---
title: "Learning rate warmup and cosine decay in C#"
url: https://recursiveai.org/onemodel/cookbook/schedule-training-learning-rate/
description: "Learning rate warmup and cosine decay in C# with OneModel. Runnable .NET example with the actual output and a downloadable project."
version: "1.3.0"
---

# Learning rate warmup and cosine decay in C#

**OneModel** is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneModel.Training) · [GitHub](https://github.com/recursive-ai-oss/onemodel-csharp) · [Docs](https://recursiveai.org/onemodel/docs/) · [Download example (.zip)](https://recursiveai.org/onemodel/cookbook/schedule-training-learning-rate/schedule-training-learning-rate.zip)

A learning-rate schedule changes the optimizer’s update size as training progresses.

```shell
dotnet add package RecursiveAI.OneModel.Training
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneModel.Training;

var schedule = new WarmupLr(
    new CosineLr(0.1f, 4, 0.01f),
    2, 0.1f);
for (int step = 0; step <= 7; step++)
    Console.WriteLine(
        $"Step {step}: " +
        $"{schedule.LrAt(step):F4}");
```

The schedule uses 2 warmup steps starting at 10% of the rate, then cosine decay from 0.1 down to 0.01 over 4 steps.

Unzip [the example](https://recursiveai.org/onemodel/cookbook/schedule-training-learning-rate/schedule-training-learning-rate.zip), then `dotnet run`.

```text
Step 0: 0.0100
Step 1: 0.0550
Step 2: 0.1000
Step 3: 0.0868
Step 4: 0.0550
Step 5: 0.0232
Step 6: 0.0100
Step 7: 0.0100
```

## The full program

The snippet at the top previews the rates. The download applies those same rates to eight real SGD updates of a scalar weight, using weight squared as the loss.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

using var weight = Tensor.Scalar(3f).WithRequiresGrad();
using var optimizer = new Sgd([weight], 0.1f);
var schedule = new WarmupLr(
    new CosineLr(0.1f, 4, 0.01f), 2, 0.1f);
for (int step = 0; step <= 7; step++)
{
    optimizer.ZeroGrad();
    using var loss = Ops.Mul(weight, weight);
    loss.Backward();
    schedule.Apply(optimizer);
    Console.WriteLine($"Step {optimizer.StepCount}: {optimizer.LearningRate:F4}");
    optimizer.Step();
}
optimizer.ZeroGrad();
```

```text
Step 0: 0.0100
Step 1: 0.0550
Step 2: 0.1000
Step 3: 0.0868
Step 4: 0.0550
Step 5: 0.0232
Step 6: 0.0100
Step 7: 0.0100
```

Schedule steps count optimizer updates, not epochs; call `Apply` before `Step` so the first update uses step index zero.

## Variations

To drop the rate by half every two updates, replace the schedule declaration in the full program:

```csharp
var schedule = new StepLr(0.1f, 2, 0.5f);
```

It prints:

```text
Step 0: 0.1000
Step 1: 0.1000
Step 2: 0.0500
Step 3: 0.0500
Step 4: 0.0250
Step 5: 0.0250
Step 6: 0.0125
Step 7: 0.0125
```

## Related recipes

- [Train a model in C#](https://recursiveai.org/onemodel/cookbook/train-a-model/)
- [Differentiate a tensor expression](https://recursiveai.org/onemodel/cookbook/differentiate-tensor-expression/)

## The short answer

Compose `WarmupLr` around `CosineLr` and call `schedule.Apply(optimizer)` before each update. `Apply` reads `optimizer.StepCount`; it does not advance a separate scheduler counter. The snippet at the top previews exactly the rate function that the downloadable training loop applies.

The example uses two warmup steps followed by four cosine intervals, with a peak rate of 0.1 and a minimum of 0.01. These small counts make the boundaries visible in the printed output. They are not training recommendations. The [learning-rate reference](https://recursiveai.org/onemodel/docs/training/#5-learning-rate-schedules) gives the formulas and the meaning of the step index.

## What does a learning rate scheduler do?

It assigns the optimizer’s learning rate from a step number. The optimizer still owns gradient clearing, parameter updates and its step count. A schedule does not compute a loss or run backward propagation. That is why the download includes a real scalar training loop as well as the rate preview.

In OneModel, `LrAt(stepIndex)` is a pure function. Calling it repeatedly with the same index gives the same rate without moving training forward. `Apply` evaluates that function at the number of updates already taken. On a fresh optimizer that count is zero, so the first call sets the rate for step zero. Applying only once before a whole training run leaves the rate unchanged during subsequent updates.

For epoch-based scheduling, decide explicitly how epochs map to optimizer updates. If each epoch contains several batches, the optimizer count advances for every `Step`, not once per epoch. If gradients accumulate across batches, it advances only when you actually update parameters.

## What is the warmup learning rate?

Warmup starts at a fraction of the wrapped schedule’s initial rate and increases linearly toward it. In `WarmupLr(after, W, f)`, the initial rate is `after.LrAt(0) * f`. While the index is below W, the fraction is `f + (1 - f) * step / W`. At W the wrapped schedule starts at its own index zero.

Here the starting fraction is 0.1, so the first rate is 0.01. Step one uses 0.055. Step two starts the wrapped cosine at 0.1. The warmup does not consume the cosine’s duration: it precedes it. When planning a run, include both the warmup length and the cosine duration in your update budget.

## What is a cosine learning rate?

Cosine decay interpolates from a base rate to a minimum using half a cosine wave. `CosineLr(baseLr, totalSteps, minLr)` reaches the minimum at its own `totalSteps` index and stays there afterward. It does not restart or turn upward after that boundary.

In this composition, the cosine’s index zero is global step two, its midpoint is global step four, and its minimum is reached at global step six. Step seven stays at 0.01. The output therefore distinguishes a completed schedule from a new warmup or restart.

Recreate the same schedule configuration when resuming training and restore the optimizer’s step count with its state. The schedule has no separate counter to restore. If you instead create a fresh optimizer at step zero, applying the schedule restarts warmup even if the model weights came from a later checkpoint. Changing the schedule parameters also changes resumed rates, so preserve those settings with your training configuration.
