---
title: "Accumulate gradients across microbatches in C#"
url: https://recursiveai.org/onemodel/cookbook/accumulate-gradients-across-microbatches/
description: "Accumulate neural network gradients over microbatches in C#, scale a mean loss correctly, and run one SGD update with OneModel."
version: "1.3.0"
---

# Accumulate gradients across microbatches in C#

**OneModel** is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneModel.Training) · [GitHub](https://github.com/recursive-ai-oss/onemodel-csharp) · [Docs](https://recursiveai.org/onemodel/docs/) · [Download example (.zip)](https://recursiveai.org/onemodel/cookbook/accumulate-gradients-across-microbatches/accumulate-gradients-across-microbatches.zip)

Call `Backward()` for each piece of a batch, then `Step()` once to apply the combined gradient.

```shell
dotnet add package RecursiveAI.OneModel.Training
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

using var weight = Tensor.Scalar(0f)
    .WithRequiresGrad();
using var sgd = new Sgd([weight], 0.25f);
using var half = Tensor.Scalar(0.5f); // 1 / number of microbatches
float[] targets = [1f, 3f];
foreach (float value in targets)
{
    using var scope = new ActivationScope();
    using var target = Tensor.Scalar(value);
    Losses.Mse(weight, target).Backward(half);
}
sgd.Step();
Backprop.ZeroGrad(weight);
Console.WriteLine(weight.Data<float>()[0]);
```

Unzip [the example](https://recursiveai.org/onemodel/cookbook/accumulate-gradients-across-microbatches/accumulate-gradients-across-microbatches.zip), then `dotnet run`.

```text
1
```

## The full program

The download shows what each microbatch adds to the gradient of a single trainable scalar. The targets are `1` and `3`. The weight starts at `0`, and SGD uses a learning rate of `0.25`.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

using var weight = Tensor.Scalar(0f)
    .WithRequiresGrad();
using var sgd = new Sgd([weight], 0.25f);
using var half = Tensor.Scalar(0.5f);
foreach (float value in new[] { 1f, 3f })
{
    using var scope = new ActivationScope();
    using var target = Tensor.Scalar(value);
    Losses.Mse(weight, target).Backward(half);
    Console.WriteLine($"Accumulated gradient: {weight.Grad.Data<float>()[0]}");
}
sgd.Step();
Backprop.ZeroGrad(weight);
Console.WriteLine($"Optimizer steps: {sgd.StepCount}");
Console.WriteLine($"Weight: {weight.Data<float>()[0]}");
Console.WriteLine($"Gradient after zeroing: {weight.Grad.Data<float>()[0]}");
```

```text
Accumulated gradient: -1
Accumulated gradient: -4
Optimizer steps: 1
Weight: 1
Gradient after zeroing: 0
```

If you omit the `0.5` backward seed, the two mean-loss gradients are summed rather than averaged, so this example takes twice the intended update.

## Variations

**Handle microbatches of different sizes.** In the snippet at the top, replace everything from `using var half` through the end of the `foreach` block with the code below. Sum the squared errors in each microbatch and use one third as the backward seed because there are three target elements overall. The final weight still prints `1`:

```csharp
using var scale = Tensor.Scalar(1f / 3f);
float[][] batches = [[1f], [2f, 3f]];
foreach (var values in batches)
{
    using var scope = new ActivationScope();
    using var target = Tensor.FromSpan<float>(
        values, Shape.Create(values.Length));
    var loss = Losses.Mse(weight, target, mean: false);
    loss.Backward(scale);
}
```

**Accumulate gradients for a layer.** Keep the two `using` directives from the snippet at the top and replace the rest with this code. The `Linear` layer starts with a weight and bias of zero; `Sgd` updates both after the two backward passes.

```csharp
using var model = new Linear(1, 1, new Generator(42));
model.Weight.MutableData<float>()[0] = 0f;
model.Bias.MutableData<float>()[0] = 0f;
using var sgd = new Sgd(model.Parameters(), 0.25f);
using var half = Tensor.Scalar(0.5f);
(float Input, float Target)[] batches = [(1f, 1f), (2f, 3f)];
foreach (var batch in batches)
{
    using var scope = new ActivationScope();
    using var input = Tensor.FromSpan<float>(
        [batch.Input], Shape.Create(1, 1));
    using var target = Tensor.FromSpan<float>(
        [batch.Target], Shape.Create(1, 1));
    var prediction = model.Forward(input);
    Losses.Mse(prediction, target).Backward(half);
}
sgd.Step();
model.ZeroGrad();
Console.WriteLine($"Optimizer steps: {sgd.StepCount}");
float weight = model.Weight.Data<float>()[0];
float bias = model.Bias.Data<float>()[0];
Console.WriteLine(FormattableString.Invariant(
    $"Weight: {weight}, bias: {bias}"));
```

```text
Optimizer steps: 1
Weight: 1.75, bias: 1
```

## Related recipes

- [Release training activation memory](https://recursiveai.org/onemodel/cookbook/scope-training-activation-memory/)
- [Clip gradients by norm](https://recursiveai.org/onemodel/cookbook/clip-neural-network-gradients/)

## The short answer

Run a fresh forward and `Backward()` for each microbatch without clearing the parameter gradients in between. OneModel adds each backward result into the leaf’s existing gradient. After the last microbatch, call the optimizer’s `Step()` once and zero or clear the gradients before the next group. The [autograd reference](https://recursiveai.org/onemodel/docs/autograd/#2-the-engine-tensorbackward-backprop) specifies this accumulation behavior.

## What is gradient accumulation?

It’s adding gradients from several forward and backward passes before updating a model’s parameters. A separate forward creates a graph for each microbatch, and the usual `Backward()` releases that graph’s saved records. You don’t need `retainGraph: true` to sum contributions into the same parameter. This lets you work through the inputs in pieces, and [`ActivationScope`](https://recursiveai.org/onemodel/cookbook/scope-training-activation-memory/) releases each microbatch’s temporary activations when its loop iteration ends. The parameter values must stay fixed until the last piece has contributed.

In this example each loss is `(weight − target)²`. At weight zero the two unscaled derivatives are `−2` and `−6`. Each is multiplied by `0.5` in backward, so the leaf first holds `−1` and then `−4`. The single SGD step subtracts `0.25 × (−4)`, giving a new weight of `1`. Zeroing the gradient after that step doesn’t change the weight.

## How do you scale loss for gradient accumulation?

Choose the reduction first. `Losses.Mse` defaults to the mean over all squared differences. If every microbatch has the same element count, divide each mean loss’s gradient by the number of microbatches that actually contribute to the update. If counts differ, weight each microbatch mean by its share of the total element count. Equivalently, sum each microbatch’s loss and divide each sum by the same total count, as the variation does. This also handles a final partial group, provided you count only elements in that group. If your objective is a sum rather than a mean, don’t divide at all.

This matches the math of a single batch for a loss that is a sum or mean of independent items with the same predictions. Don’t assume bit-identical floating-point results, or treat a pass that computes batch statistics as independent per item. For example, [OneModel batch normalization in training mode](https://recursiveai.org/onemodel/docs/training/#1-modules) uses the current batch’s statistics, which can change when you split a batch.

## When do you zero gradients during accumulation?

Start each group with no previous contribution. Clear or zero after `Step()` or before the first backward of the next group, not between microbatches. `Backprop.ZeroGrad(weight)` resets the gradient in place; `Backprop.ClearGrad(weight)` releases it so the next backward can create it again. For this MSE example the denominator counts elements, but the optimizer counts one step for the entire group regardless of how many backward calls contributed.
