Recursive AI

Clip gradients by norm in C#

OneModel is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

Gradient clipping scales down large gradients before the optimizer updates weights.

Terminal
dotnet add package RecursiveAI.OneModel.Training

Requires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
using var weight = Tensor.Scalar(3f)
    .WithRequiresGrad();
using var loss = Ops.Mul(weight, weight);
loss.Backward();
float norm = GradClip.ClipNorm([weight], 1f);
Console.WriteLine($"Before: {norm:F3}");
Console.WriteLine(
    $"After: {weight.Grad.Data<float>()[0]:F3}");

Unzip the example, then dotnet run.

Text
Before: 6.000
After: 1.000

The full program

The download also takes one SGD step. Its scalar loss is weight squared, so the gradient starts at six; no model or dataset download is needed.

C#
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;
 
using var weight = Tensor.Scalar(3f).WithRequiresGrad();
using var optimizer = new Sgd([weight], 0.1f);
using var loss = Ops.Mul(weight, weight);
loss.Backward();
float before = GradClip.ClipNorm([weight], 1f);
Console.WriteLine($"Before: {before:F3}");
Console.WriteLine($"After: {weight.Grad.Data<float>()[0]:F3}");
optimizer.Step();
Console.WriteLine($"Weight: {weight.Data<float>()[0]:F3}");
optimizer.ZeroGrad();
Text
Before: 6.000
After: 1.000
Weight: 2.900

A non-finite norm is returned without scaling anything; check it before calling the optimizer if training can produce NaN or infinity.

Variations

Replace the clipping line in the full program to use the largest absolute gradient instead of the L2 norm:

C#
float before = GradClip.ClipNorm(
    [weight], 1f, infinityNorm: true);

For this scalar, the full program’s output is unchanged. To clamp each gradient element separately instead, replace that line with:

C#
float before = weight.Grad.Data<float>()[0];
GradClip.ClipValue([weight], 0.5f);

The full program then prints:

Text
Before: 6.000
After: 0.500
Weight: 2.950

The short answer

Call loss.Backward(), then GradClip.ClipNorm(parameters, maxNorm), then the optimizer’s Step(). The method changes gradients in place and returns their norm before clipping. It does not update weights itself. The training package supplies both clipping and optimizers; the tensor operations come through its core dependency.

Use one collection containing all parameters whose joint gradient you want to constrain. Clipping each tensor separately is a different operation: it gives each tensor its own allowance rather than sharing a single norm budget. The training reference defines the reduction across every element of every gradient.

How does gradient norm clipping work?

The default norm is L2: square each gradient element, add those squares, then take the square root. OneModel accumulates squares in double precision in parameter and element order. It calculates the scale as maxNorm / (norm + 1e-6), rounds it to float and multiplies gradients only when that scale is less than one. Small gradients are therefore not amplified.

In the example, differentiating weight squared at weight three gives gradient six. With a maximum norm of one, clipping scales that gradient down to approximately one. The printed three-decimal output hides the small difference introduced by the denominator’s epsilon. SGD at rate 0.1 then changes the weight to approximately 2.9. The before-and-after values are gradient measurements, while the final line is a parameter value.

Frozen parameters are skipped. Row-sparse gradients contribute their stored values once per row; you do not need to convert an embedding gradient to dense storage before clipping it. These behaviors are specified in the same training reference.

What is the difference between gradient clipping by norm and value?

Norm clipping multiplies the whole gradient collection by one coefficient when needed. For finite gradients, this preserves their relative proportions. Value clipping instead clamps individual components to a fixed interval, which can change those proportions. ClipValue(parameters, 0.5f) uses the interval from −0.5 to 0.5; it is not a global norm bound.

The infinity-norm option uses the largest absolute gradient component as the norm, but still scales the whole collection. That is not the same operation as independently clamping each component. The scalar example cannot reveal this distinction because it has only one component; on a vector, the unbounded components behave differently.

What gradient clipping value should I use?

The threshold in this recipe is an illustration, not a recommended setting for all networks. Record the returned pre-clipping norm during your own training and decide how often you want clipping to intervene. Keep the loss reduction and parameter collection consistent when comparing those measurements: changing either changes the norm being measured.

Clipping is not a repair for invalid arithmetic. A non-finite norm is returned without modifying gradients, and value clipping leaves NaN values as NaN. Stop or skip the update when the norm is not finite rather than passing invalid values to the optimizer. Clear gradients between updates, and clip after the final backward pass when accumulating several contributions for one optimizer step.

All onemodel recipes