---
title: "Resume training from a checkpoint in C#"
url: https://recursiveai.org/onemodel/cookbook/resume-training-from-checkpoint/
description: "Save a model, its Adam optimizer, and training progress to a checkpoint. Resume on the CPU and compare the result with uninterrupted training."
version: "1.3.0"
---

# Resume training from a checkpoint in C#

**OneModel** is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneModel.Training) · [GitHub](https://github.com/recursive-ai-oss/onemodel-csharp) · [Docs](https://recursiveai.org/onemodel/docs/) · [Download example (.zip)](https://recursiveai.org/onemodel/cookbook/resume-training-from-checkpoint/resume-training-from-checkpoint.zip)

`Checkpoint.Load` restores the weights and Adam state. The zip includes `step3.checkpoint`; without the zip, run the [full program](#full-program) first to create it.

```shell
dotnet add package RecursiveAI.OneModel.Training
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

using var model = new Linear(1, 1,
    new Generator(42));
using var adam = new Adam(model.Parameters());
Checkpoint.Load(
    "step3.checkpoint", model, adam);
using var x = Tensor.FromSpan<float>([1], 1, 1);
using var y = Tensor.FromSpan<float>([3], 1, 1);
using var scope = new ActivationScope();
Losses.Mse(model.Forward(x), y).Backward();
adam.Step();
Console.WriteLine($"Step: {adam.StepCount}");
```

Unzip [the example](https://recursiveai.org/onemodel/cookbook/resume-training-from-checkpoint/resume-training-from-checkpoint.zip), then `dotnet run`.

Takes one more Adam step and prints:

```text
Step: 4
```

## Full program

The program makes the checkpoint from scratch. It records the number of completed steps, loads into a fresh model and optimizer, then runs both branches through step six on the CPU. The input and target are the same at every step.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

// One training pair: x = 1, target = 3.
using var input = Tensor.FromSpan<float>([1], 1, 1);
using var target = Tensor.FromSpan<float>([3], 1, 1);
using var original = new Linear(1, 1, new Generator(42));
using var adam = new Adam(original.Parameters(), 0.1f);
for (int step = 0; step < 3; step++)
    TrainStep(original, adam);

const string path = "step3.checkpoint";
Checkpoint.Save(path, original, adam,
    new Dictionary<string, long> { ["completed"] = 3 });

// A different seed and starting learning rate show what Load replaces.
using var resumed = new Linear(1, 1, new Generator(7));
using var resumedAdam = new Adam(resumed.Parameters(), 0.01f);
var info = Checkpoint.Load(path, resumed, resumedAdam);
long completed = info.Extra["completed"];
Console.WriteLine($"Restored step: {completed}");

for (long step = completed; step < 6; step++)
{
    TrainStep(original, adam);
    TrainStep(resumed, resumedAdam);
}

// Compare the serialized tensors, optimizer moments and step count.
using var uninterruptedState = new MemoryStream();
using var resumedState = new MemoryStream();
Checkpoint.Save(uninterruptedState, original, adam);
Checkpoint.Save(resumedState, resumed, resumedAdam);
bool same = uninterruptedState.ToArray().AsSpan()
    .SequenceEqual(resumedState.ToArray());
if (!same)
    throw new InvalidOperationException("The training runs diverged.");
Console.WriteLine($"Final optimizer step: {resumedAdam.StepCount}");
Console.WriteLine("Final checkpoints match byte for byte.");

void TrainStep(Linear model, Adam optimizer)
{
    using var scope = new ActivationScope();
    optimizer.ZeroGrad();
    var loss = Losses.Mse(model.Forward(input), target);
    loss.Backward();
    optimizer.Step();
}
```

```text
Restored step: 3
Final optimizer step: 6
Final checkpoints match byte for byte.
```

The final comparison includes the model and Adam state, not just the loss. If the target model’s tensor layout or the optimizer’s kind or fixed settings don’t match the checkpoint, `Load` throws `CheckpointException` with kind `Mismatch` before copying tensors.

## Variations

**Load only the model.** In the snippet at the top, replace the `Load` call with this overload to keep your newly created Adam fresh. The snippet then prints `Step: 1`.

```csharp
Checkpoint.Load("step3.checkpoint", model);
```

## Related recipes

- [Copy a neural network’s state dictionary](https://recursiveai.org/onemodel/cookbook/copy-model-state-dictionary/)
- [Resume a deterministic data loader](https://recursiveai.org/onemodel/cookbook/resume-deterministic-data-loader/)

## The short answer

Use `Checkpoint.Save(path, model, optimizer, extra)` at the end of a training step. To continue, construct matching model and optimizer objects, pass both to `Checkpoint.Load`, and resume with the next batch. Read any application progress you saved from `CheckpointInfo.Extra`. For this program, `completed` means the number of steps already taken, so the first update after reading `3` is step four.

## Do you need to load the optimizer state to resume training?

Yes if you want the same continuation, not just the same starting weights. Adam uses a running first and second moment of past gradients and a step count. A new Adam sees the same weights on its first forward pass, but its first update uses fresh moments and starts the step count at one. That’s why the example compares serialized optimizer state as well as the model tensors.

Recreate the same model structure and use the same optimizer kind and fixed settings, such as Adam’s betas and epsilon. The initial seed can differ because the model tensors are overwritten. The learning rate can also start at a different value: the [checkpoint reader](https://recursiveai.org/onemodel/docs/training/#6-checkpoints-checkpoint) restores it along with the optimizer step count. A model-only file can’t restore an optimizer that wasn’t saved.

## What should you save in a training checkpoint?

Save at a step boundary, after `Step()` and before you consume the next batch. The file includes named parameters and buffers, so any running normalization statistics are part of the saved model too. Pass integer progress values such as completed epochs or batches to `extra`. Use a consistent convention: if the stored value counts completed batches, it points to the next batch in a zero-based sequence.

A checkpoint doesn’t choose the next batch for you. When batches are shuffled, track [the loader’s progress](https://recursiveai.org/onemodel/docs/data-loading/) separately so the resumed run sees the same examples in the same order. Recreate your training mode and any scheduler or other application state needed for the next update. The sample repeats a single batch at a fixed learning rate, so its only application counter is `completed`. The byte-for-byte match is for that continuation, not a guarantee for a loop that changes its batches or learning-rate schedule after a restart.
