Recursive AI

Accumulate gradients across microbatches in C#

OneModel is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Call Backward() for each piece of a batch, then Step() once to apply the combined gradient.

Terminal
dotnet add package RecursiveAI.OneModel.Training

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
using var weight = Tensor.Scalar(0f)
    .WithRequiresGrad();
using var sgd = new Sgd([weight], 0.25f);
using var half = Tensor.Scalar(0.5f); // 1 / number of microbatches
float[] targets = [1f, 3f];
foreach (float value in targets)
{
    using var scope = new ActivationScope();
    using var target = Tensor.Scalar(value);
    Losses.Mse(weight, target).Backward(half);
}
sgd.Step();
Backprop.ZeroGrad(weight);
Console.WriteLine(weight.Data<float>()[0]);

Unzip the example, then dotnet run.

Text
1

The full program

The download shows what each microbatch adds to the gradient of a single trainable scalar. The targets are 1 and 3. The weight starts at 0, and SGD uses a learning rate of 0.25.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
using var weight = Tensor.Scalar(0f)
    .WithRequiresGrad();
using var sgd = new Sgd([weight], 0.25f);
using var half = Tensor.Scalar(0.5f);
foreach (float value in new[] { 1f, 3f })
{
    using var scope = new ActivationScope();
    using var target = Tensor.Scalar(value);
    Losses.Mse(weight, target).Backward(half);
    Console.WriteLine($"Accumulated gradient: {weight.Grad.Data<float>()[0]}");
}
sgd.Step();
Backprop.ZeroGrad(weight);
Console.WriteLine($"Optimizer steps: {sgd.StepCount}");
Console.WriteLine($"Weight: {weight.Data<float>()[0]}");
Console.WriteLine($"Gradient after zeroing: {weight.Grad.Data<float>()[0]}");
Text
Accumulated gradient: -1
Accumulated gradient: -4
Optimizer steps: 1
Weight: 1
Gradient after zeroing: 0

If you omit the 0.5 backward seed, the two mean-loss gradients are summed rather than averaged, so this example takes twice the intended update.

Variations

Handle microbatches of different sizes. In the snippet at the top, replace everything from using var half through the end of the foreach block with the code below. Sum the squared errors in each microbatch and use one third as the backward seed because there are three target elements overall. The final weight still prints 1:

C#
using var scale = Tensor.Scalar(1f / 3f);
float[][] batches = [[1f], [2f, 3f]];
foreach (var values in batches)
{
    using var scope = new ActivationScope();
    using var target = Tensor.FromSpan<float>(
        values, Shape.Create(values.Length));
    var loss = Losses.Mse(weight, target, mean: false);
    loss.Backward(scale);
}

Accumulate gradients for a layer. Keep the two using directives from the snippet at the top and replace the rest with this code. The Linear layer starts with a weight and bias of zero; Sgd updates both after the two backward passes.

C#
using var model = new Linear(1, 1, new Generator(42));
model.Weight.MutableData<float>()[0] = 0f;
model.Bias.MutableData<float>()[0] = 0f;
using var sgd = new Sgd(model.Parameters(), 0.25f);
using var half = Tensor.Scalar(0.5f);
(float Input, float Target)[] batches = [(1f, 1f), (2f, 3f)];
foreach (var batch in batches)
{
    using var scope = new ActivationScope();
    using var input = Tensor.FromSpan<float>(
        [batch.Input], Shape.Create(1, 1));
    using var target = Tensor.FromSpan<float>(
        [batch.Target], Shape.Create(1, 1));
    var prediction = model.Forward(input);
    Losses.Mse(prediction, target).Backward(half);
}
sgd.Step();
model.ZeroGrad();
Console.WriteLine($"Optimizer steps: {sgd.StepCount}");
float weight = model.Weight.Data<float>()[0];
float bias = model.Bias.Data<float>()[0];
Console.WriteLine(FormattableString.Invariant(
    $"Weight: {weight}, bias: {bias}"));
Text
Optimizer steps: 1
Weight: 1.75, bias: 1

The short answer

Run a fresh forward and Backward() for each microbatch without clearing the parameter gradients in between. OneModel adds each backward result into the leaf’s existing gradient. After the last microbatch, call the optimizer’s Step() once and zero or clear the gradients before the next group. The autograd reference specifies this accumulation behavior.

What is gradient accumulation?

It’s adding gradients from several forward and backward passes before updating a model’s parameters. A separate forward creates a graph for each microbatch, and the usual Backward() releases that graph’s saved records. You don’t need retainGraph: true to sum contributions into the same parameter. This lets you work through the inputs in pieces, and ActivationScope releases each microbatch’s temporary activations when its loop iteration ends. The parameter values must stay fixed until the last piece has contributed.

In this example each loss is (weight − target)². At weight zero the two unscaled derivatives are −2 and −6. Each is multiplied by 0.5 in backward, so the leaf first holds −1 and then −4. The single SGD step subtracts 0.25 × (−4), giving a new weight of 1. Zeroing the gradient after that step doesn’t change the weight.

How do you scale loss for gradient accumulation?

Choose the reduction first. Losses.Mse defaults to the mean over all squared differences. If every microbatch has the same element count, divide each mean loss’s gradient by the number of microbatches that actually contribute to the update. If counts differ, weight each microbatch mean by its share of the total element count. Equivalently, sum each microbatch’s loss and divide each sum by the same total count, as the variation does. This also handles a final partial group, provided you count only elements in that group. If your objective is a sum rather than a mean, don’t divide at all.

This matches the math of a single batch for a loss that is a sum or mean of independent items with the same predictions. Don’t assume bit-identical floating-point results, or treat a pass that computes batch statistics as independent per item. For example, OneModel batch normalization in training mode uses the current batch’s statistics, which can change when you split a batch.

When do you zero gradients during accumulation?

Start each group with no previous contribution. Clear or zero after Step() or before the first backward of the next group, not between microbatches. Backprop.ZeroGrad(weight) resets the gradient in place; Backprop.ClearGrad(weight) releases it so the next backward can create it again. For this MSE example the denominator counts elements, but the optimizer counts one step for the entire group regardless of how many backward calls contributed.

All onemodel recipes