Screenshot tests for a game

Capture frames of a C# game headless, compare them with reference images at a tolerance in a scenario, update the references when a change is intended, and capture a sequence of frames.

Capture a frame

onegame capture --out shots/main.png --size 1080p --after 300 --json

{"ok":true,"result":{"path":"/tmp/games/orchard/shots/main.png","width":1920,"height":1080,"scale":1,"sha256":"599056f9db1627201e3b1d49594b23751129a2496a99ae4fd32353404b3c5d3a","renderer":"NVIDIA GeForce RTX 4070","tick":300,"hash":"d0045829889cf126"}}

--after 300 runs 300 ticks first. Sizes are 360p 720p 1080p 1440p 4k deck phone tablet or WxH, and deck is 1280×800. renderer names the GPU that drew the frame, or cpu when the software renderer did. Without --out the picture goes to captures/frame.png.

Open the PNG and look at it. A matching state hash proves the run repeats, not that the picture is right.

Compare with a reference in a scenario

name: looks
description: The main screen after the first apples grow, compared with a reference picture.
seed: 7
window: 640x360
steps:
  - step: 300
expect:
  - capture: { reference: refs/main.png, size: 640x360, tolerance: 2, max-diff: 0.001 }

tolerance: 2 lets each color channel of a pixel differ by up to 2. max-diff: 0.001 lets up to 0.1% of the pixels differ by more than that. The reference path is relative to the scenario file, so this one is scenarios/refs/main.png.

The first run has nothing to compare with:

FAIL looks (324 ms, tick 300, hash d5ef04c0af6a538b)
  expect capture (line 8): No reference image at /tmp/games/orchard/scenarios/refs/main.png. Run with --update-references (or ONEGAME_UPDATE_REFERENCES=1) to create it from the current frame, then look at it.
    expected: a reference image
    actual:   none
0 passed, 1 failed

onegame test --scenario scenarios/looks.yaml --update-references

PASS looks (310 ms, tick 300, hash d5ef04c0af6a538b)
created reference /tmp/games/orchard/scenarios/refs/main.png
1 passed, 0 failed

Look at the new reference before you commit it. From then on onegame test compares against it. After the button’s color changes from green to red, the state hash is the same but the picture is not:

FAIL looks (348 ms, tick 300, hash d5ef04c0af6a538b)
  expect capture (line 8): The frame differs from refs/main.png: 6788 px (2.946 %) differ by more than 2, max channel delta 91; allowed 0.100 %. Wrote /tmp/games/orchard/artifacts/looks.0.actual.png and /tmp/games/orchard/artifacts/looks.0.diff.png.
    expected: within 0.001 of reference
    actual:   6788 px differ (max delta 91)
0 passed, 1 failed

looks.0.actual.png is the new frame. In looks.0.diff.png the pixels that differ are bright red over a darkened copy of the frame, here the whole button. When the change is intended, run with --update-references again. With --json the result lists the files under references: {created, updated}.

Capture a sequence

onegame capture --sequence grow --count 4 --every 30 --size 360p --json

{"ok":true,"result":{"directory":"/tmp/games/orchard/captures/grow","manifest":"/tmp/games/orchard/captures/grow/manifest.json","frames":4,"width":640,"height":360,"fps":2}}

That is four frames 30 ticks apart, frame-0000.png to frame-0003.png, and a manifest.json with each frame’s tick, state hash and SHA-256. The sequence folder goes under captures/, while --out for a single frame is relative to the current folder.

Good to know

• Captures render offscreen at any size, so no window or display is needed, on a GPU when there is one and otherwise on the CPU.
• GPU drivers differ along edges, which is what tolerance and max-diff are for. Make references on the machine that runs the tests, or loosen them for another rasterizer.
• On a phone and in a browser a capture compares against name-mobile.png beside name.png when that file exists.
• From C#, TestClient.Capture renders the current state; see Test a game from C#.