Recursive AI

Resume training from a checkpoint in C#

OneModel is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Checkpoint.Load restores the weights and Adam state. The zip includes step3.checkpoint; without the zip, run the full program first to create it.

Terminal
dotnet add package RecursiveAI.OneModel.Training

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
using var model = new Linear(1, 1,
    new Generator(42));
using var adam = new Adam(model.Parameters());
Checkpoint.Load(
    "step3.checkpoint", model, adam);
using var x = Tensor.FromSpan<float>([1], 1, 1);
using var y = Tensor.FromSpan<float>([3], 1, 1);
using var scope = new ActivationScope();
Losses.Mse(model.Forward(x), y).Backward();
adam.Step();
Console.WriteLine($"Step: {adam.StepCount}");

Unzip the example, then dotnet run.

Takes one more Adam step and prints:

Text
Step: 4

Full program

The program makes the checkpoint from scratch. It records the number of completed steps, loads into a fresh model and optimizer, then runs both branches through step six on the CPU. The input and target are the same at every step.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
// One training pair: x = 1, target = 3.
using var input = Tensor.FromSpan<float>([1], 1, 1);
using var target = Tensor.FromSpan<float>([3], 1, 1);
using var original = new Linear(1, 1, new Generator(42));
using var adam = new Adam(original.Parameters(), 0.1f);
for (int step = 0; step < 3; step++)
    TrainStep(original, adam);
 
const string path = "step3.checkpoint";
Checkpoint.Save(path, original, adam,
    new Dictionary<string, long> { ["completed"] = 3 });
 
// A different seed and starting learning rate show what Load replaces.
using var resumed = new Linear(1, 1, new Generator(7));
using var resumedAdam = new Adam(resumed.Parameters(), 0.01f);
var info = Checkpoint.Load(path, resumed, resumedAdam);
long completed = info.Extra["completed"];
Console.WriteLine($"Restored step: {completed}");
 
for (long step = completed; step < 6; step++)
{
    TrainStep(original, adam);
    TrainStep(resumed, resumedAdam);
}
 
// Compare the serialized tensors, optimizer moments and step count.
using var uninterruptedState = new MemoryStream();
using var resumedState = new MemoryStream();
Checkpoint.Save(uninterruptedState, original, adam);
Checkpoint.Save(resumedState, resumed, resumedAdam);
bool same = uninterruptedState.ToArray().AsSpan()
    .SequenceEqual(resumedState.ToArray());
if (!same)
    throw new InvalidOperationException("The training runs diverged.");
Console.WriteLine($"Final optimizer step: {resumedAdam.StepCount}");
Console.WriteLine("Final checkpoints match byte for byte.");
 
void TrainStep(Linear model, Adam optimizer)
{
    using var scope = new ActivationScope();
    optimizer.ZeroGrad();
    var loss = Losses.Mse(model.Forward(input), target);
    loss.Backward();
    optimizer.Step();
}
Text
Restored step: 3
Final optimizer step: 6
Final checkpoints match byte for byte.

The final comparison includes the model and Adam state, not just the loss. If the target model’s tensor layout or the optimizer’s kind or fixed settings don’t match the checkpoint, Load throws CheckpointException with kind Mismatch before copying tensors.

Variations

Load only the model. In the snippet at the top, replace the Load call with this overload to keep your newly created Adam fresh. The snippet then prints Step: 1.

C#
Checkpoint.Load("step3.checkpoint", model);

The short answer

Use Checkpoint.Save(path, model, optimizer, extra) at the end of a training step. To continue, construct matching model and optimizer objects, pass both to Checkpoint.Load, and resume with the next batch. Read any application progress you saved from CheckpointInfo.Extra. For this program, completed means the number of steps already taken, so the first update after reading 3 is step four.

Do you need to load the optimizer state to resume training?

Yes if you want the same continuation, not just the same starting weights. Adam uses a running first and second moment of past gradients and a step count. A new Adam sees the same weights on its first forward pass, but its first update uses fresh moments and starts the step count at one. That’s why the example compares serialized optimizer state as well as the model tensors.

Recreate the same model structure and use the same optimizer kind and fixed settings, such as Adam’s betas and epsilon. The initial seed can differ because the model tensors are overwritten. The learning rate can also start at a different value: the checkpoint reader restores it along with the optimizer step count. A model-only file can’t restore an optimizer that wasn’t saved.

What should you save in a training checkpoint?

Save at a step boundary, after Step() and before you consume the next batch. The file includes named parameters and buffers, so any running normalization statistics are part of the saved model too. Pass integer progress values such as completed epochs or batches to extra. Use a consistent convention: if the stored value counts completed batches, it points to the next batch in a zero-based sequence.

A checkpoint doesn’t choose the next batch for you. When batches are shuffled, track the loader’s progress separately so the resumed run sees the same examples in the same order. Recreate your training mode and any scheduler or other application state needed for the next update. The sample repeats a single batch at a fixed learning rate, so its only application counter is completed. The byte-for-byte match is for that continuation, not a guarantee for a loop that changes its batches or learning-rate schedule after a restart.

All onemodel recipes