This one comes from our image-model research program — an agent whose job is to design and run experiments against generative image models and write the findings into a learnings library. One of those findings is worth publishing on its own.
We built a metric stack to validate generated sprites: edge density, contrast distributions, silhouette consistency, palette checks. It worked — for geometry, scale, and cross-frame consistency. Then a character passed three experiments and a cross-experiment comparison with no face.
The failure is instructive. A missing face isn’t a low-contrast region or a statistical anomaly — it’s a smoothly-shaded oval missing three semantic marks. Every pixel statistic in the stack saw a perfectly plausible head. Metrics built specifically to catch the missing face afterward didn’t work either: the difference between a face and no-face lives in meaning, not in any distribution we could compute.
The general form: mechanical metrics validate geometry, scale, and consistency well, and validate semantics not at all. The two failure classes look identical in a dashboard until something ships without a face.
What we do now:
- Budget for a semantic check — a human or a vision model actually looking at the artifact — on any axis where meaning can silently vanish. It’s a real cost; pay it on the axes that matter.
- Keep generative content and mechanical composition separate. Crops, stitches, and measurements are code; the model is never asked to do geometry. That keeps the surface where semantics can fail as small as possible.
- Promote a rule to a law only after it survives a model change. Most “laws” of prompting turn out to be one model’s personality. Our learnings library keeps the distinction explicit — and later experiments have refuted six claims of an earlier synthesis, with the record keeping both versions.
Failures are findings. The verbatim prompt language of failed calls is preserved in every experiment record, because the delta between the failed prompt and the working one is usually the lesson.