SparseAdam trains an embedding table, which maps integer IDs to vectors, using row-sparse gradients that store only the rows used by a batch.
dotnet add package RecursiveAI.OneModel.TrainingRequires .NET 10 (net10.0); .NET 9 and earlier aren’t supported.
using RecursiveAI.OneModel.Tensors; using RecursiveAI.OneModel.Training; using var table = Tensor.FromSpan<float>( [1, 2, 3, 4], 2, 2).WithRequiresGrad(); using var ids = Tensor.FromSpan<long>([1], 1); using var optimizer = new SparseAdam( [table], 0.1f); using var selected = Ops.Embedding( table, ids, -1, sparse: true); using var loss = Ops.Sum(selected); loss.Backward(); optimizer.Step(); Console.WriteLine( table.Data<float>()[2].ToString("F3"));
The lookup’s -1 means no padding ID; the optimizer’s 0.1f is the learning rate.
Unzip the example, then dotnet run.
2.900The full program
The download uses four rows of two values each and IDs [1, 1, 3]. Its loss sums the selected vectors. It prints which gradient rows exist, then the first value of every row before and after one update.
using RecursiveAI.OneModel.Tensors; using RecursiveAI.OneModel.Training; using var table = Tensor.FromSpan<float>( [1, 2, 3, 4, 5, 6, 7, 8], 4, 2).WithRequiresGrad(); using var ids = Tensor.FromSpan<long>([1, 1, 3], 3); using var optimizer = new SparseAdam([table], 0.1f); float[] before = table.Data<float>().ToArray(); using var selected = Ops.Embedding(table, ids, -1, sparse: true); using var loss = Ops.Sum(selected); loss.Backward(); Console.WriteLine($"Gradient rows: {string.Join(", ", table.SparseGrad!.Rows.ToArray())}"); optimizer.Step(); for (int row = 0; row < 4; row++) { float value = table.Data<float>()[row * 2]; Console.WriteLine($"Row {row}: {before[row * 2]:F3} -> {value:F3}"); } optimizer.ZeroGrad();
Gradient rows: 1, 3 Row 0: 1.000 -> 1.000 Row 1: 3.000 -> 2.900 Row 2: 5.000 -> 5.000 Row 3: 7.000 -> 6.900
SparseAdam throws on a dense gradient; keep the embedding table a leaf tensor and request sparse: true for its lookups.
Variations
To skip ID 1 as padding, replace the lookup line in the full program:
using var selected = Ops.Embedding( table, ids, 1, sparse: true);
It prints:
Gradient rows: 3 Row 0: 1.000 -> 1.000 Row 1: 3.000 -> 3.000 Row 2: 5.000 -> 5.000 Row 3: 7.000 -> 6.900
Related recipes
The short answer
Create a trainable rank-two table, request row-sparse backward with Ops.Embedding(..., sparse: true) and give the table to SparseAdam. After backward, the table’s gradient is in SparseGrad, not a dense Grad tensor. The optimizer changes only touched rows and leaves the other rows and their optimizer moments unchanged.
The short example uses the primitive lookup operation so every initial weight and ID is visible. For an initialized module, Embedding(count, width, generator, paddingIndex, sparse: true) provides the same row-sparse option. Both paths are documented in the training reference.
How do you train an embedding table with sparse gradients?
Keep the table as a leaf requiring gradients, select rows by integer ID, compute a scalar loss from the resulting vectors and call backward. Route that table to a sparse-compatible optimizer. The downloadable example uses a sum as its loss to make the update easy to inspect; it is not a language-model objective or a recommendation for learning useful embeddings.
IDs may repeat. In the download, row one appears twice, but SparseGrad.Rows lists it only once. Its gradient values include both contributions, accumulated in lookup-position order. The stored row numbers are ascending and unique. The padding row, when specified, is absent even if its ID appears in the input.
Clear gradients between updates with the optimizer’s ZeroGrad. For row-sparse gradients, zeroing releases the sparse gradient so there are no stored rows. Without clearing, later backward contributions accumulate rather than replacing the previous batch. A second sparse lookup merges touched rows; a dense contribution changes the table’s gradient to dense storage.
What does SparseAdam update?
It updates the first moment, second moment and weights for touched elements, using the global optimizer step for bias correction. Untouched moments do not decay. This makes the choice more than a different storage format for ordinary dense Adam: the behavior of untouched rows is part of the algorithm.
The weights themselves are still a dense table. SparseAdam also keeps full-size moment tensors; the sparse saving is in gradient storage and which rows an update visits, not in eliminating the whole table or optimizer state. Budget memory for those arrays even when each batch contains few distinct IDs.
The printed unchanged rows are intentional. Rows zero and two were not looked up, so their first values stay exactly at their initial values after the update. Printing one value per row makes the example readable; the optimizer processes the entire width of each touched row.
Which optimizer supports sparse embedding gradients?
OneModel supports SparseAdam and plain Sgd for row-sparse gradients. Sparse SGD must have no momentum or weight decay; either option causes NotSupportedException. Adam and AdamW reject sparse gradients with InvalidOperationException. SparseAdam requires rank-two parameters and has no weight decay or AMSGrad option.
For a network with a sparse embedding and dense layers, use separate optimizers: the table belongs to SparseAdam, and dense parameters can belong to AdamW. Clear both optimizers’ gradients before backward, then step both after backward. Do not place the same table in both collections.
If your objective requires weight decay over every table row, use dense gradients and a dense optimizer instead. There is no sparse AdamW in this release. Converting the table’s gradient to dense also means SparseAdam is no longer the matching optimizer. Choose the gradient representation and optimizer together rather than changing just one side of the training loop.