Timing tests

Constant-time evidence for secret-handling code, in the style of Reparaz, Balasch and Verbauwhede, "Dude, is my code constant time?" (DATE 2017). Project: tests/Timing. Runner: tools/timing.sh.

Method

• Two input classes: a fixed secret and a fresh random secret per call (key-secret targets: see "Key-secret targets"). The class sequence is drawn from the CSPRNG before the run, so calls interleave at random.
• Each call is timed with Stopwatch.GetTimestamp. Inputs are prepared in batches outside the timed region; the timed region allocates nothing. A warm-up of at least half a second runs first so tiered compilation has settled; the runner also sets DOTNET_TieredCompilation=0, DOTNET_TieredPGO=0 and DOTNET_ReadyToRun=0 for full-opt JIT.
• The analysis runs Welch's t-test on the raw data and on 100 percentile crops (the k-th keeps the fastest 1 − 0.5^(10(k+1)/100) of the calls), and reports the largest |t| and which crop gave it.
• Verdict: max |t| < 4.5 passes; anything else is reported as a leak.
• Presets: short (200 000 calls) for gating merges, full (5 000 000) for milestones, --n to choose.
• Timing runs are meaningless on a loaded machine. Publishable runs are exclusive benchmark jobs (PERF_CLASS=bench); day-to-day runs are indicative heavy jobs. tools/perf-run.sh also records CPU, cores and load before, during and after.

Writing a target

Add a file under tests/Timing/ with a static method returning your targets, and add it to the list in Targets.cs. For one secret byte string going into one call:

TimingTarget.FromBytes("X25519", "scalar-multiply", 32, secret => s_sink ^= Run(secret));

For anything with more structure, subclass TimingTarget: Prepare builds the inputs for a batch from the class sequence (fixed secret or fresh random one), Run(i) is the timed call and must not allocate. Keep the result in a field so the JIT cannot remove the call.

Self-check

timing selfcheck runs an early-exit byte compare (must be flagged) and CryptographicOperations.FixedTimeEquals (must pass). It is the proof that the harness sees leaks on the machine it runs on.

Results and publishable runs

tools/timing.sh <outdir> … writes timing.json, machine.txt, load.log and publishable.txt under <outdir> (relative to the repository root). A run is marked publishable: NO when it was not class bench, or when the 1-minute load average before or during it exceeds 2.0 (Linux) or 1.5 (macOS); set PERF_MAX_LOAD to change the limit. Only publishable runs count as evidence.

Key-secret targets

When the secret is a key, the cost of building the key object (schedule, tables, pinned buffers) is not what is under test, but the object's memory is what the operation reads. A first design kept a pool of 128 objects holding the fixed key and 128 holding distinct random keys, and drew from them. It could not be made reliable: the A/A control flagged on x64, and on Apple M2 the Camellia and AES-KW key kinds flagged at |t| 8 to 15 (mean difference about 0.1 percent) while their controls passed. Pool objects of one class differ from the other in more than the key (identical schedules in 128 places against 128 different ones, placement, cache footprint), so a flag could not be attributed to the key.

The current design keeps everything but the key value equal:

• One fresh key object per call, built in an untimed hook. TimingTarget.Before(i) runs right before each timed Run(i). A key target disposes the previous object, fills one reusable key buffer, builds a new object from it (and, for the AEADs and AES-KW, the message under that key), and Run only decrypts, encrypts or unwraps. Both classes therefore allocate the same objects in the same pattern, run the same constructor, and time a call on an object that was built a moment ago.
• The class selects the key bytes with a mask, not a branch. FillKey draws fresh random bytes for every call and then blends the fixed key over them with an all-ones or all-zero byte mask. Both classes execute the same instructions and branch history before the timed call. An earlier version copied the fixed key with an if for the fixed class only; with that, Twofish-128 decrypt-key on Apple M2 flagged in three of four runs (|t| 3.2 to 6.3; the sign of the mean difference changed with each run's fixed key) while its control stayed at or below 2.9, and the flags were gone once the branch became a mask.
• An A/A control per key target. The control uses the same code with the mask forced to zero, so both classes are fresh random keys. A key kind counts only if its control passes; a flagging control means the host, not the code, and the run is repeated.
• ChaCha20-Poly1305 uses the same hook and mask (one reusable key buffer). The curves take a fresh random scalar or seed per call in the random class and have no object to rebuild, so they have no control.

tools/timing-keys.sh <outdir> [N] runs the key-secret targets. TIMING_ONLY=key keeps only the key-related operations; TIMING_PRIMITIVES chooses primitives, or Primitive/operation entries. Blowfish builds its key object in 0.5 to 1.3 ms, so each of its targets takes 15 to 25 minutes at N = 1,000,000 and runs as its own job.

Results (indicative)

N = 1,000,000 calls per target, full-opt JIT, run as shared-host jobs (not exclusive), so every number is indicative. Values are the maximum |t| over the uncropped data and 100 crops; the pass threshold is 4.5. x64: AMD Ryzen 9 7950X3D, Linux, .NET 10.0.402 (the host was busy, load 10 to 20, during part of the runs). arm64: Apple M2, macOS, .NET 10.0.403. "Setup" is the key schedule itself, timed, with a fixed or a random key. Every key kind and control below passes on both machines.

The Blowfish key-setup and the curve rows on x64 come from builds that differ from the final one only in the key-kind hook, which those targets do not use. Data-secret targets are untouched by this design and keep their last results (all pass).

Before and after on the signals that started this work (maximum |t|, key kind with its control in brackets):

What the old signals were

None of the Camellia and AES-KW signals reproduces with a fresh object per call and a masked key select. No source file under src/ changed for these results, so the earlier flags came from how the harness built its two classes. The pool design was not taken apart further: which of its differences (identical schedules, placement, footprint) caused the Camellia and AES-KW flags is not isolated. The class-dependent copy branch was isolated for Twofish-128 (above). AES-KW runs on the platform's AES and passes.

Argon2id

Argon2i is the data-independent variant and has a timing target. Argon2id has no constant-time claim as a whole: only the first two slices of its first pass are addressed from public values, and the rest picks reference blocks from block contents (RFC 9106 §3.4), so password-dependent timing is expected. Measured once on Apple M2 (one pass, 64 KiB): |t| 12.05. The public API cannot time the data-independent part alone, so there is no Argon2id target. Argon2d and scrypt are data-dependent by design and have none either.