The Chrome accessibility-tree differential

OneHtml.Differential ax compares our accessibility tree with the one Chrome returns from Accessibility.getFullAXTree, node by node, for pages captured by tools/capture. Chrome is a black box here: only its protocol output is read. The command is in tools/OneHtml.Differential; nothing in src/ depends on it.

OneHtml.Differential ax <capture-dir> [--corpus <dir>] [--out <dir>] [--jobs n] [--limit n] [--timeout s] [--top n] [--facts on|off|both] [--dump]

Exit status is 0 whatever the rates are; the numbers are the result.

What is compared

For each capture the tool reads dom.json, snapshot.json, ax.json and meta.json:

1. Chrome's serialized DOM (doctype plus outerHTML) is parsed with OneHtml. The cascade gets the capture's viewport (width, height, deviceScaleFactor) as media features, the page URL as base URL, and the stylesheet resolver above. Scripts never ran; that is how the capture was taken.
2. The snapshot is mapped onto the document (DomSnapshotAdapter.Map). A snapshot of a different DOM is refused; the capture is counted as refused with the refusal reason and not compared, because nodes cannot be matched without the mapping.
3. Our tree is built (AccessibilityTree.Build, with AccessibilityOptions.BaseUrl the capture's final URL) and written in the Chrome-shaped export (onehtml-ax/1). Chrome's main-frame nodes and the export are read by the same code into a common shape, so both sides pass through the same normalization.
4. Nodes are matched, then compared.

Only the main frame's nodes are compared; frames of other documents, and the child ids that point into them, are ignored.

Matching

1. The root web area pairs with our document node.
2. A Chrome node with a backendDOMNodeId pairs with our node for the same DOM node. The id is translated through the snapshot's backendNodeId and the mapping's node accounting (NodeFacts.BackendNodeId).
3. Anonymous nodes. A Chrome node that has no OneHtml node (no backend id, or one the mapping does not carry: generated text, layout boxes, user-agent shadow content) pairs with the first still unpaired child of the counterpart of its nearest paired ancestor that is of the same kind (text or not), has the same role and the same ignored flag, and has the same name. Anything else stays unpaired. The number paired this way is in the summary.
4. Chrome's InlineTextBox nodes are skipped on read. They are layout boxes of a StaticText and carry no information the static text does not.
5. Unpaired nodes are counted per side. An unpaired node that is ignored is counted separately (chromeOnlyIgnored, oursOnlyIgnored): an ignored node is not part of what assistive technology walks, so it does not change the exposed tree. Chrome omits visibility: hidden and aria-hidden elements and noscript; we keep them as ignored nodes.

Fields

Each paired node is compared on the fields below. A field is compared when it applies; compared in the report is the number of times it was.

Normalization

Everything that makes the two sides spelled differently without a difference in meaning is in AxNormalizer (tools/OneHtml.Differential/AxModel.cs), nowhere else. It is:

• White space. Runs of ASCII white space (space, tab, line feed, form feed, carriage return) in names, descriptions and valuetext collapse to one space and the ends are trimmed. A no-break space stays.
• Flags. false and absent are the same for focusable, focused, disabled, required, readonly, modal, multiselectable, multiline and busy; invalid=false is absent; an empty valuetext, roledescription, keyshortcuts, autocomplete or relevant is absent. Chrome reports these flags as false where the role supports them and we leave them out. expanded, selected, checked and pressed are not touched: absent means the role does not have the state, false means it is off.
• Numbers. Property numbers are compared in their shortest round-trip spelling (1 and 1.0). The value of a meter, progressbar, slider, spinbutton or scrollbar is compared as a 32-bit float, because Chrome stores these as single-precision floats and reports them widened (0.6 arrives as 0.6000000238418579). Other values compare as written.
• Value and flags added by the last round. An empty value is absent; atomic=false is absent; busy as 1 is true.
• Parents. parent and exposedParent are compared through the nearest ancestor that has a counterpart on the other side, so an unpaired ignored node is not counted twice.
• Role of ignored nodes. Both sides call an ignored node none; its role is not compared.
• Name source of static text is text on both sides.
• Names and descriptions of ignored nodes are not compared (neither side has them).

Nothing else is normalized: no role aliases, no case folding, no Unicode normalization.

Rendered facts

--facts selects whether our tree is built with the capture's rendered facts (AccessibilityOptions.Facts). The facts are always used for the mapping in step 2, in both modes.

• off: one run without facts.
• on: one run with facts; the report is that run.
• both (default): each page is compared twice, with and without facts, over the same Chrome nodes. The per-field table and the classes are the run with facts; the section "With and without rendered facts" prints both rates side by side (with the number of mismatching nodes in brackets).

A capture whose snapshot does not map onto its DOM is refused in every mode.

Output

Written to --out; nothing in it should be copied into the repository except reduced, hand-written cases.

Rates. occurrences counts mismatching (capture, node) pairs; nodes counts distinct (page, node) pairs, so a page captured at two viewports counts its nodes once; pages is the number of pages with at least one. rate is occurrences over compared. For properties and relations, compared counts the pairs where either side has the property, so a rare property that always disagrees reads 100%.

Classes. A class is a field plus a pattern: for role, the two roles (LayoutTable -> table); for name, description and value, the role and the shape of the difference (chrome-empty, ours-empty, ours-prefix, chrome-prefix, case-only, differs); for properties, the role and both values; for ignored, which side is ignored and the roles and reasons; for the structure fields, the roles of both parents; for the presence fields, the role and, when ignored, the reasons. Output is deterministic: captures are processed in sha256 and viewport order whatever --jobs is.

Worker run

The command takes the worker's shared-job lock like the other corpus commands and exits with status 3 at once if it is held:

bash tools/OneHtml.Differential/run-worker.sh ax ~/onehtml/capture-out/<version> ~/onehtml/out/ax --corpus ~/onehtml/corpus/<version> --facts both --jobs 8

--limit N takes the first N captures by hash and --pages <sha256 prefix,...> selects pages; --dump writes a per-node listing of the selected pages.

Results with and without rendered facts

Chrome for Testing 156.0.8078.47 (protocol 1.3), headless, 1280x800, page script off; .NET 10 on the AMD Ryzen 9 7950X3D worker (32 threads), --facts both --jobs 2 --limit 2100, about 125 seconds. The sample is the first 2,100 captures by hash: 2,095 compared (1,840,086 Chrome nodes, 1,973,233 of ours, 1,779,203 matched, 284,249 of them by the anonymous-node rule) and 5 refused (a custom element whose shadow root the snapshot omits, and two whose text differs in line breaks). Rates are mismatching nodes over the times the field was compared, with the number of nodes in brackets. "Before" is the same pages with the tree as it stood at the start of this round (facts on). Fields that never differ are left out; the run's summary.md has all of them.

Without facts the tree cannot know what Chrome did not draw, so ignored, exposedParent, chromeOnly and oursOnly are high; with facts they are the figures above. No field is worse with facts than without by more than a few nodes, except prop.editable (1 node without facts, 61 with: the inner editors of text fields, see the classes). The probe pages (testdata/rendered/pages/*probe*.html, with Chrome's captures beside them) pin the rules that came out of this round: ChromeProbeTests compares the exposed structure of the generated-content probe, the inner nodes of text fields and the name of each of 170 links with Chrome's.

Classes

A class is a field plus a pattern (see "Output"). Dispositions of every class above 0.1% of its field's compared count (for the small property fields, above one node). Fixed means the class shrank to the figures given, with a hand-written test and a profile entry; open means it was not traced to a rule and is not documented as a deliberate difference.

The open classes above are not traced to a rule. The largest are name link differs (0.3% of the field's compared nodes), the generic and StaticText differences in chromeOnly and oursOnly, and the three small ignored residues.

Property fields, second pass

The first run of the full corpus (12,788 captures, facts both) left these property fields above 1%: posinset and setsize 100%, invalid 27.3%, readonly 21.5%, disabled 15.8%, valuetext 7.9%, value 4.9%, roledescription 2.3%, selected 1.7%, valuemax 1.2%, autocomplete 2.8%, hasPopup 1.0%, editable 1.1%, keyshortcuts 1.0%, and chromeOnly 1.126%. Each had one or two systematic causes. They were found by writing a probe page for each suspected behaviour (testdata/rendered/pages/axprops.html, axprops2.html, axprops3.html), capturing it in the pinned Chrome (captures beside it in testdata/rendered/captures), and fixing the tree or export to match; the rule is a profile entry of the name given. ChromePropertyTests has one case per cause; ChromePropertyProbeTests compares every element with an id on the first two probe pages property by property and lists the three differences left; ChromeProbeTests compares the inner-node structure of the third.

Causes

Before and after

Same machine and run as above (Chrome for Testing 156.0.8078.47, --facts both --jobs 2, one process; AMD Ryzen 9 7950X3D, 32 threads, load average 4 to 5 from other work during the runs, so these are counts, not timings). Counts are mismatching nodes with facts. The subset is the 171 captures whose first mismatch examples were in the classes above (every class had at least one page); the sample is 399 captures from the middle of the corpus order (--skip 3000 --limit 400, one refused), as a check that no other field gets worse.

The final code was run on both sets. Every field not named is equal; none rose. The final rates come from the next full-corpus run.

What is left, per field

Behaviour not copied

• The Chrome value of an editing host with block content (x\n\nbs for a div holding a p, a button and a span) depends on layout line breaks; ours has none (ChromePropertyProbeTests).
• aria-selected=false on a row: Chrome states nothing, we state false (as for the grid cell and option, which state it).
• Step mismatch counts only from a min (the step base is the min, else the value itself).

Findings on the fixture pages

The fixture pages (testdata/rendered/pages) were captured with Chrome 156 (headless, script off, 1280×800). After the fixes listed in the profile, what is left is below, each with a verdict. "Chrome" means Chrome's choice where the specifications leave room or are silent; "ours" means we do not yet do what Chrome does and no specification says otherwise. Evidence is Chrome's output on the page named.

Profile entries added or confirmed from this run are in accessibility-chromium-profile.md (evidence "observed in Chrome output"). Unobserved and worth a capture: an empty div and span; u, s, small and other inline formatting elements; a nested list; <label> with each control type and position; the other date and time input types; nested tables and role=presentation tables with th.