Recursive AI

Keep the BatchNorm running mean fixed in eval mode in C#

OneModel is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Call Eval() to keep batch normalization’s running mean and variance fixed.

Terminal
dotnet add package RecursiveAI.OneModel.Training

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
using var norm = new BatchNorm2d(1);
using var batch =
    Tensor.FromSpan<float>([2, 6], 2, 1, 1, 1);
using var noGrad = GradMode.NoGrad();
norm.Forward(batch).Dispose();
Console.WriteLine(
    norm.RunningMean.Item<float>());
norm.Eval();
norm.Forward(batch).Dispose();
Console.WriteLine(
    norm.RunningMean.Item<float>());

Unzip the example, then dotnet run.

A new layer starts in training mode, so the first pass moves the running mean from 0 to 0.4 (default momentum 0.1 times batch mean 4); after Eval(), it stays at 0.4.

Text
0.4
0.4

The full program

The example trains on a single batch with values 2 and 6, both with target 0. The batch has shape [2, 1, 1, 1]: two samples, one channel, one row, one column. After one optimizer step, it switches to evaluation mode and feeds in 8 as a single sample. StateDict() lets you read the running mean and variance before and after both passes.

C#
using System.Globalization;
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
using var norm = new BatchNorm2d(1, momentum: 0.5f);
using var batch = Tensor.FromSpan<float>([2, 6], 2, 1, 1, 1);
PrintStats("Initial");
 
norm.Train(true);
using (var activations = new ActivationScope())
using (var optimizer = new Sgd(norm.Parameters(), learningRate: 0.1f))
{
    using var target = Tensor.FromSpan<float>([0, 0], 2, 1, 1, 1);
    var loss = Losses.Mse(norm.Forward(batch), target);
    loss.Backward();
    optimizer.Step();
}
PrintStats("After training");
 
norm.Eval();
using (GradMode.NoGrad())
{
    using var probe = Tensor.FromSpan<float>([8], 1, 1, 1, 1);
    using var output = norm.Forward(probe);
    Console.WriteLine($"Eval output: {output.Item<float>():F3}");
}
PrintStats("After eval");
 
void PrintStats(string label)
{
    var state = norm.StateDict();
    var mean = state.Single(entry => entry.Name == "running_mean").Tensor;
    var variance = state.Single(entry => entry.Name == "running_var").Tensor;
    Console.WriteLine($"{label}: mean={mean.Item<float>()}, var={variance.Item<float>()}");
}
Text
Initial: mean=0, var=1
After training: mean=2, var=4.5
Eval output: 2.263
After eval: mean=2, var=4.5

If you forget Eval(), a forward pass in training mode uses the new batch’s statistics and updates both buffers, even when gradient recording is off.

Variations

Use evaluation mode for validation, then resume training. Replace the lines from norm.Eval() onward in the snippet at the top:

C#
norm.Eval();
using (GradMode.NoGrad())
{
    using var validation = norm.Forward(batch);
    Console.WriteLine(norm.IsTraining);
}
norm.Train(true);
Console.WriteLine(norm.IsTraining);
Text
0.4
False
True

The short answer

After training, call Eval() on the layer or its parent model. BatchNorm2d then uses running_mean and running_var without updating either. Read them from StateDict() to check what training collected. Use GradMode.NoGrad() separately if you don’t need a backward pass.

What does batch normalization normalize?

For an input shaped [N, C, H, W], BatchNorm2d normalizes each channel separately. In training mode it measures that channel across the samples and spatial positions in the current batch, subtracts the mean, and divides by the square root of the variance plus a small epsilon. It then applies the channel’s learned scale (weight) and shift (bias).

The layer also blends the new statistics into its running buffers. momentum is the weight of the new statistic in that blend. In the full program the batch mean is 4, so with momentum 0.5 the running mean moves from 0 to 2. The training forward pass uses the batch variance, 4, to normalize. The running buffer instead uses the unbiased estimate, 8 for these two values. Blending it halfway with the initial 1 gives 4.5. See OneModel’s batch normalization implementation.

What does evaluation mode do to BatchNorm?

Eval() makes the layer use the statistics collected from training, so the sample’s normalization doesn’t depend on what else is in the prediction batch. It doesn’t recompute the running statistics from validation data. The layer still applies the learned scale and shift, which is why the printed prediction includes the effect of the optimizer step.

The running mean and variance are buffers, not optimizer parameters. StateDict() includes them alongside weight and bias; reading them as numbers at each stage lets you compare the values. Eval() also sets the mode on child modules, so with a model made of several layers you normally call it once on the parent. If you’re validating between training epochs, switch back with Train(true) before the next training pass. See the OneModel training guide.

All onemodel recipes