---
title: "Train sparse embeddings in C#"
url: https://recursiveai.org/onemodel/cookbook/train-sparse-embedding-table/
description: "Train sparse embeddings in C# with OneModel. Runnable .NET example with the actual output and a downloadable project."
version: "1.3.0"
---

# Train sparse embeddings in C#

**OneModel** is a free, open-source neural-network library for .NET from Recursive AI. MIT or Apache-2.0, no native dependencies.

[NuGet](https://www.nuget.org/packages/RecursiveAI.OneModel.Training) · [GitHub](https://github.com/recursive-ai-oss/onemodel-csharp) · [Docs](https://recursiveai.org/onemodel/docs/) · [Download example (.zip)](https://recursiveai.org/onemodel/cookbook/train-sparse-embedding-table/train-sparse-embedding-table.zip)

SparseAdam trains an embedding table, which maps integer IDs to vectors, using row-sparse gradients that store only the rows used by a batch.

```shell
dotnet add package RecursiveAI.OneModel.Training
```

Requires [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0) (net10.0); .NET 9 and earlier aren’t supported.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

using var table = Tensor.FromSpan<float>(
    [1, 2, 3, 4], 2, 2).WithRequiresGrad();
using var ids = Tensor.FromSpan<long>([1], 1);
using var optimizer = new SparseAdam(
    [table], 0.1f);
using var selected = Ops.Embedding(
    table, ids, -1, sparse: true);
using var loss = Ops.Sum(selected);
loss.Backward();
optimizer.Step();
Console.WriteLine(
    table.Data<float>()[2].ToString("F3"));
```

The lookup’s `-1` means no padding ID; the optimizer’s `0.1f` is the learning rate.

Unzip [the example](https://recursiveai.org/onemodel/cookbook/train-sparse-embedding-table/train-sparse-embedding-table.zip), then `dotnet run`.

```text
2.900
```

## The full program

The download uses four rows of two values each and IDs `[1, 1, 3]`. Its loss sums the selected vectors. It prints which gradient rows exist, then the first value of every row before and after one update.

```csharp
using RecursiveAI.OneModel.Tensors;
using RecursiveAI.OneModel.Training;

using var table = Tensor.FromSpan<float>(
    [1, 2, 3, 4, 5, 6, 7, 8], 4, 2).WithRequiresGrad();
using var ids = Tensor.FromSpan<long>([1, 1, 3], 3);
using var optimizer = new SparseAdam([table], 0.1f);
float[] before = table.Data<float>().ToArray();
using var selected = Ops.Embedding(table, ids, -1, sparse: true);
using var loss = Ops.Sum(selected);
loss.Backward();
Console.WriteLine($"Gradient rows: {string.Join(", ", table.SparseGrad!.Rows.ToArray())}");
optimizer.Step();
for (int row = 0; row < 4; row++)
{
    float value = table.Data<float>()[row * 2];
    Console.WriteLine($"Row {row}: {before[row * 2]:F3} -> {value:F3}");
}
optimizer.ZeroGrad();
```

```text
Gradient rows: 1, 3
Row 0: 1.000 -> 1.000
Row 1: 3.000 -> 2.900
Row 2: 5.000 -> 5.000
Row 3: 7.000 -> 6.900
```

`SparseAdam` throws on a dense gradient; keep the embedding table a leaf tensor and request `sparse: true` for its lookups.

## Variations

To skip ID 1 as padding, replace the lookup line in the full program:

```csharp
using var selected = Ops.Embedding(
    table, ids, 1, sparse: true);
```

It prints:

```text
Gradient rows: 3
Row 0: 1.000 -> 1.000
Row 1: 3.000 -> 3.000
Row 2: 5.000 -> 5.000
Row 3: 7.000 -> 6.900
```

## Related recipes

- [Train a model in C#](https://recursiveai.org/onemodel/cookbook/train-a-model/)
- [Differentiate a tensor expression](https://recursiveai.org/onemodel/cookbook/differentiate-tensor-expression/)

## The short answer

Create a trainable rank-two table, request row-sparse backward with `Ops.Embedding(..., sparse: true)` and give the table to `SparseAdam`. After backward, the table’s gradient is in `SparseGrad`, not a dense `Grad` tensor. The optimizer changes only touched rows and leaves the other rows and their optimizer moments unchanged.

The short example uses the primitive lookup operation so every initial weight and ID is visible. For an initialized module, `Embedding(count, width, generator, paddingIndex, sparse: true)` provides the same row-sparse option. Both paths are documented in the [training reference](https://recursiveai.org/onemodel/docs/training/#31-row-sparse-embedding-gradients-sparseadam).

## How do you train an embedding table with sparse gradients?

Keep the table as a leaf requiring gradients, select rows by integer ID, compute a scalar loss from the resulting vectors and call backward. Route that table to a sparse-compatible optimizer. The downloadable example uses a sum as its loss to make the update easy to inspect; it is not a language-model objective or a recommendation for learning useful embeddings.

IDs may repeat. In the download, row one appears twice, but `SparseGrad.Rows` lists it only once. Its gradient values include both contributions, accumulated in lookup-position order. The stored row numbers are ascending and unique. The padding row, when specified, is absent even if its ID appears in the input.

Clear gradients between updates with the optimizer’s `ZeroGrad`. For row-sparse gradients, zeroing releases the sparse gradient so there are no stored rows. Without clearing, later backward contributions accumulate rather than replacing the previous batch. A second sparse lookup merges touched rows; a dense contribution changes the table’s gradient to dense storage.

## What does SparseAdam update?

It updates the first moment, second moment and weights for touched elements, using the global optimizer step for bias correction. Untouched moments do not decay. This makes the choice more than a different storage format for ordinary dense Adam: the behavior of untouched rows is part of the algorithm.

The weights themselves are still a dense table. SparseAdam also keeps full-size moment tensors; the sparse saving is in gradient storage and which rows an update visits, not in eliminating the whole table or optimizer state. Budget memory for those arrays even when each batch contains few distinct IDs.

The printed unchanged rows are intentional. Rows zero and two were not looked up, so their first values stay exactly at their initial values after the update. Printing one value per row makes the example readable; the optimizer processes the entire width of each touched row.

## Which optimizer supports sparse embedding gradients?

OneModel supports `SparseAdam` and plain `Sgd` for row-sparse gradients. Sparse SGD must have no momentum or weight decay; either option causes `NotSupportedException`. `Adam` and `AdamW` reject sparse gradients with `InvalidOperationException`. SparseAdam requires rank-two parameters and has no weight decay or AMSGrad option.

For a network with a sparse embedding and dense layers, use separate optimizers: the table belongs to SparseAdam, and dense parameters can belong to AdamW. Clear both optimizers’ gradients before backward, then step both after backward. Do not place the same table in both collections.

If your objective requires weight decay over every table row, use dense gradients and a dense optimizer instead. There is no sparse AdamW in this release. Converting the table’s gradient to dense also means SparseAdam is no longer the matching optimizer. Choose the gradient representation and optimizer together rather than changing just one side of the training loop.
