---
title: "Normalize like a skeptic"
url: https://recursiveai.org/research/normalize-like-a-skeptic/
description: "What it takes to measure AI-assisted development honestly — score scope, never lines; classify diffs, never messages; decompose, don’t zero; and drop eras with missing data from headline claims."
date: 2026-08-27
author: Kurtis Welch
---

# Normalize like a skeptic

August 27, 2026 · Kurtis Welch

Measuring AI-assisted development honestly took more design than expected. The naive metrics all flatter you, and the flattery is exactly why they can’t be published. What survived eight months of building our own measurement instrument:

**Score scope, never lines.** One week in our corpus shows +199k lines — vendored code, worth zero. Line counts reward copying and punish consolidation, which is the opposite of what a velocity metric should do. Every commit is sized by the scope of the work its diff contains (XS through XL), and vendored code and merges score zero.

**Classify diffs, never messages.** Hundreds of commits in the corpus are literally titled `wip`. The message is what the author felt like typing; the diff is what happened. The classifier reads the diff, splits it into distinct work items, and caches every judgment per SHA with its rationale — re-runnable and auditable.

**Decompose, don’t zero.** The tempting move is to exclude documentation and upkeep as “not real work” — but they are real work, and excluding what’s inconvenient is how metrics die. Instead, every point lands in exactly one of capability, rework, or support, and the three sum to the total. Nothing is smoothed and nothing is dropped; the reader can weight the categories themselves.

**Normalize to active weeks when the timeline has gaps.** The pre-AI baseline developer wasn’t coding continuously across 2020–23, so a whole-span average would make the baseline artificially low and flatter every multiplier built on it. The baseline uses contiguous multi-week active stretches only — the strictest reading available.

**Publish coverage next to every number, and mark floors as floors.** A points total computed from 80% classification coverage is a floor, not a total, and the difference matters. Every chart the instrument draws carries its coverage inline.

**Drop eras with known-missing data from headline claims entirely.** Our agentic-coding era has a known measurement gap (concurrent client work wasn’t captured), so it supports no claim in either direction — it appears only as a conservative comparison line that favors the era it represents.

The general principle: a comparison that only survives friendly normalization isn’t evidence. If your measurement can’t be handed to the most hostile reader you can imagine, it isn’t finished.